[00:38:54] 06Traffic: Local coding agents should be allowed to fetch wikipedia / wikidata - https://phabricator.wikimedia.org/T409871#11364966 (10ssingh) Hi @Monneyboi: Thanks for reporting! Setting a user-agent will fix the error as your own example above shows (thanks for trying that!) And additionally, like you mentione... [08:46:46] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Cloudcephosd: migrate to single network uplink - https://phabricator.wikimedia.org/T399180#11365310 (10fgiunchedi) [11:17:36] 10netops, 06Infrastructure-Foundations, 06SRE: Row C traffic outage Nov 11 2025 - https://phabricator.wikimedia.org/T409800#11365890 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=e41d36ab-ea9e-437e-a0db-341d018dedf6) set by cmooney@cumin1003 for 2:00:00 on 2 host(s) and their services w... [12:41:08] 06Traffic, 10bot-traffic-requests: FY 25/26 WE 5.4.6 Classify the top 30 spiders by traffic as known bots - https://phabricator.wikimedia.org/T408061#11366261 (10SLyngshede-WMF) Working with the suggestion from @Fabfur and mixing in a few suggestions from @CDanis I did a script that will query Druid for the "h... [12:48:02] 10netops, 06Infrastructure-Foundations, 06SRE: Row C traffic outage Nov 11 2025 - https://phabricator.wikimedia.org/T409800#11366307 (10BTullis) [13:35:16] 06Traffic, 06MediaWiki-Platform-Team (Radar), 07SecTeam-Processed, 07Security: SUL Integration for eventyay (Wikimania virtual event platform) - https://phabricator.wikimedia.org/T378157#11366536 (10bachkhois) For the further support of Wikimedia logins and increased security and feature extensions we... [13:53:21] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Cloudcephosd: migrate to single network uplink - https://phabricator.wikimedia.org/T399180#11366599 (10Andrew) OSD nodes up through 1034 are scheduled for decom in 2026. Unless there's an urgent port shortage, we should only retcon 1035 and... [14:22:26] 06Traffic, 06Security-Team, 10WMF-General-or-Unknown, 07ContentSecurityPolicy, 13Patch-For-Review: Add restrictive CSP to upload.wikimedia.org - https://phabricator.wikimedia.org/T117618#11366779 (10sbassett) >>! In T117618#11360663, @ssingh wrote: > Hi @sbassett. Yes that sounds good, let's set up a tim... [14:30:11] 07HTTPS, 06Traffic, 10MediaWiki-Action-API, 10MediaWiki-REST-API, and 4 others: Proposal: fail explicitly and revoke relevant API keys over plain-text HTTP connection for all Wikimedia APIs - https://phabricator.wikimedia.org/T368344#11366801 (10HCoplin-WMF) Question about: > When authenticating with cooki... [14:44:36] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 10Toolforge: Create new VRF and networks for Toolforge-on-Metal - https://phabricator.wikimedia.org/T409309#11366850 (10cmooney) So a few things emerged after the call today: * We need a device that will provide NAT (IPv4) and firewalling (IPv... [15:32:40] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11367204 (10ssingh) [15:45:50] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11367275 (10MoritzMuehlenhoff) Looks good! [15:48:26] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11367296 (10MoritzMuehlenhoff) One thing to consider, when we actually apply the role on DCs enab... [15:49:35] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11367302 (10MoritzMuehlenhoff) For eqiad best to use B and D and for codfw best to use C and D [15:49:41] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11367303 (10ssingh) >>! In T409860#11367296, @MoritzMuehlenhoff wrote: > One thing to consider, w... [15:57:27] 07HTTPS, 06Traffic, 10MediaWiki-Action-API, 10MediaWiki-REST-API, and 4 others: Proposal: fail explicitly and revoke relevant API keys over plain-text HTTP connection for all Wikimedia APIs - https://phabricator.wikimedia.org/T368344#11367331 (10BBlack) >>! In T368344#11366800, @HCoplin-WMF wrote: > Questi... [16:18:28] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11367420 (10ssingh) `sudo cookbook sre.ganeti.makekevm --vcpus 2 --memory 2 --disk 20 --network p... [16:42:54] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11367541 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin10... [16:45:55] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11367558 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin10... [16:52:06] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11367610 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin10... [16:56:04] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11367653 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin10... [17:12:02] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11367725 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 f... [17:22:13] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11367758 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 f... [17:29:08] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11367789 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 f... [17:34:55] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11367845 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 f... [17:50:10] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11367901 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin10... [17:51:42] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11367922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin10... [18:39:56] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11368129 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin10... [18:48:33] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11368156 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 f... [18:58:40] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11368174 (10RobH) Day 3 (Monday) : No Migrations, catch up day for other tasks for both Rob and John. Tuesday Holiday doesn't count Day 4 Update (Wednesday):... [19:00:32] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11368177 (10RobH) [19:00:36] 10netops, 06Infrastructure-Foundations, 06SRE: Row C traffic outage Nov 11 2025 - https://phabricator.wikimedia.org/T409800#11368178 (10RobH) [19:01:23] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11368182 (10RobH) [19:02:15] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Traffic host migrations - https://phabricator.wikimedia.org/T405623#11368192 (10ssingh) >>! In T405623#11298598, @RobH wrote: > Please note this migration has shifted from Oct 15th start date to Nov 1 start date. Hi @RobH. Asking for planning: when is... [19:03:55] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Traffic host migrations - https://phabricator.wikimedia.org/T405623#11368198 (10RobH) All cp hosts in rows C/D have been migrated as of today (last ones done) and all that is left in #traffic realm for migration is dns1006 and lvs1020 via T405602. We... [19:04:35] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Traffic host migrations - https://phabricator.wikimedia.org/T405623#11368200 (10RobH) For dns1006, since the downtime of the host is around 5-12 seconds (missing about 5-12 seq numbers via ping) I'm not sure it even has to be fully depooled as long as... [19:09:11] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11368206 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 f... [19:10:26] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Traffic host migrations - https://phabricator.wikimedia.org/T405623#11368208 (10ssingh) >>! In T405623#11368198, @RobH wrote: > All cp hosts in rows C/D have been migrated as of today (last ones done) and all that is left in #traffic realm for migratio... [19:19:25] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11368237 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin10... [19:30:28] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11368261 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 f... [19:39:07] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11368278 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin10... [20:14:17] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: eqiad row C/D Traffic host migrations - https://phabricator.wikimedia.org/T405623#11368426 (10RobH) IRC Update: Chatted with Sukhbir about this just updating the task for reference: Current plan is for John and I to knock out the remainder of all hosts that we can... [20:30:41] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11368465 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 f... [20:36:48] 06Traffic, 06Infrastructure-Foundations, 06SRE, 10vm-requests: eqiad/codfw/esams/ulsfo/eqsin/drmrs/magru: 2 VM request for hCaptcha proxy (bird/anycast), total of 14 - https://phabricator.wikimedia.org/T409860#11368489 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 f... [21:04:12] 06Traffic, 07Essential-Work, 06Experimentation Lab (Experiment Platform Sprint 15), 13Patch-For-Review: Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11368570 (10JVanderhoop-WMF) [23:16:43] FIRING: [6x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [23:21:43] FIRING: [15x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [23:26:43] FIRING: [15x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [23:31:43] RESOLVED: [15x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages