[00:07:03] PROBLEM - Host db2201 is DOWN: PING CRITICAL - Packet loss = 100% [00:10:29] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 619.96 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:11:07] PROBLEM - MariaDB Replica Lag: s3 on db1240 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 659.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:44:29] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:55:33] RECOVERY - Host db2201 is UP: PING OK - Packet loss = 0%, RTA = 30.35 ms [00:57:31] PROBLEM - mysqld processes on db2201 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [00:57:31] PROBLEM - MariaDB Replica IO: x1 on db2201 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:57:31] PROBLEM - MariaDB Replica IO: s5 on db2201 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:57:31] PROBLEM - MariaDB Replica SQL: s5 on db2201 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:57:31] PROBLEM - MariaDB Replica SQL: x1 on db2201 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:57:45] PROBLEM - MariaDB read only s5 on db2201 is CRITICAL: Could not connect to localhost:3315 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [00:57:45] PROBLEM - MariaDB read only x1 on db2201 is CRITICAL: Could not connect to localhost:3320 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [01:04:31] PROBLEM - MariaDB Replica Lag: x1 on db2201 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [01:04:31] PROBLEM - MariaDB Replica Lag: s5 on db2201 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [01:07:50] (03CR) 10Jasmine: "Thanks @dzahn@wikimedia.org - & makes sense @bcornwall@wikimedia.org & I were just discussing these over IRC too." [dns] - 10https://gerrit.wikimedia.org/r/1351524 (https://phabricator.wikimedia.org/T436917) (owner: 10Jasmine) [01:09:03] (03PS1) 10TrainBranchBot: Branch commit for wmf/1.47.0-wmf.23 [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1351554 (https://phabricator.wikimedia.org/T438219) [01:09:06] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/1.47.0-wmf.23 [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1351554 (https://phabricator.wikimedia.org/T438219) (owner: 10TrainBranchBot) [01:09:17] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1351555 [01:09:17] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1351555 (owner: 10TrainBranchBot) [01:13:53] (03PS1) 10Jasmine: trafficserver::backend: remove parsoid-rt-tests > testreduce mapping [puppet] - 10https://gerrit.wikimedia.org/r/1351556 (https://phabricator.wikimedia.org/T436917) [01:19:18] (03Merged) 10jenkins-bot: Branch commit for wmf/1.47.0-wmf.23 [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1351554 (https://phabricator.wikimedia.org/T438219) (owner: 10TrainBranchBot) [01:20:17] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1351555 (owner: 10TrainBranchBot) [01:24:25] (03PS1) 10Jasmine: role::parsoid: remove reference to testreduce discovery address [puppet] - 10https://gerrit.wikimedia.org/r/1351562 (https://phabricator.wikimedia.org/T436917) [01:31:17] (03CR) 10Scott French: "Thanks, Simon! I think this is the right call, but I'm wondering whether it makes sense to drop the logging prefix entirely. Would be inte" [cookbooks] - 10https://gerrit.wikimedia.org/r/1351336 (https://phabricator.wikimedia.org/T436846) (owner: 10Slyngshede) [01:48:04] (03CR) 10Lerickson: [C:03+1] wdqs: set request timeout to 60s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351488 (https://phabricator.wikimedia.org/T439942) (owner: 10Gmodena) [01:49:24] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:00:05] Deploy window Automatic branching of MediaWiki, extensions, skins, and vendor – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T0200) [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T0200) [02:00:41] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:07:07] RECOVERY - MariaDB Replica Lag: s3 on db1240 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:08:19] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:23:58] FIRING: KafkaMirrorMakerConsumerMaxLag: Kafka MirrorMaker main-eqiad-to-main-codfw max lag in last 10 minutes - https://wikitech.wikimedia.org/wiki/Kafka/Administration#MirrorMaker - https://grafana.wikimedia.org/d/000000521/kafka-mirrormaker?var-mirror_name=main-eqiad-to-main-codfw - https://alerts.wikimedia.org/?q=alertname%3DKafkaMirrorMakerConsumerMaxLag [02:51:31] PROBLEM - very high load average likely xfs on ms-be1074 is CRITICAL: LOAD CRITICAL - total load average: 140.76, 151.52, 90.93 https://wikitech.wikimedia.org/wiki/Swift [02:55:31] RECOVERY - very high load average likely xfs on ms-be1074 is OK: LOAD OK - total load average: 11.21, 72.99, 72.35 https://wikitech.wikimedia.org/wiki/Swift [03:00:04] Deploy window Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T0300) [03:01:48] (03PS1) 10TrainBranchBot: testwikis to 1.47.0-wmf.23 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351606 (https://phabricator.wikimedia.org/T438219) [03:01:50] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by mwpresync@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351606 (https://phabricator.wikimedia.org/T438219) (owner: 10TrainBranchBot) [03:02:40] (03Merged) 10jenkins-bot: testwikis to 1.47.0-wmf.23 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351606 (https://phabricator.wikimedia.org/T438219) (owner: 10TrainBranchBot) [03:03:01] !log mwpresync@deploy1003 Started scap sync-world: testwikis to 1.47.0-wmf.23 refs T438219 [03:03:04] T438219: 1.47.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T438219 [03:28:02] !log mwpresync@deploy1003 Rolling back deployment [03:30:18] !log mwpresync@deploy1003 Finished scap sync-world: testwikis to 1.47.0-wmf.23 refs T438219 (duration: 27m 17s) [03:30:21] T438219: 1.47.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T438219 [03:32:19] PROBLEM - OSPF status on cr1-magru is CRITICAL: OSPFv2: 4/5 UP : OSPFv3: 4/5 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [03:32:37] PROBLEM - OSPF status on cr2-eqiad is CRITICAL: OSPFv2: 8/9 UP : OSPFv3: 8/9 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [03:34:10] FIRING: BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [03:34:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr1-magru and Telxius (2001:1498:1:966:1::251) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [03:34:39] FIRING: CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [04:00:05] Deploy window Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T0400) [04:01:06] !log mwpresync@deploy1003 Pruned MediaWiki: 1.47.0-wmf.20 (duration: 01m 00s) [05:19:34] !log marostegui@cumin1004 START - Cookbook sre.mysql.depool depool db2169: moving to s7 [05:21:41] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving to s7 [05:26:00] (03PS1) 10Marostegui: mariadb: Move db2169 to s7 [puppet] - 10https://gerrit.wikimedia.org/r/1351783 (https://phabricator.wikimedia.org/T439653) [05:26:01] !log marostegui@cumin1004 START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet [05:26:01] !log marostegui@cumin1004 END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet [05:26:51] PROBLEM - MariaDB Event Scheduler s6 on db2169 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [05:26:51] PROBLEM - MariaDB read only s6 on db2169 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [05:27:14] PROBLEM - MariaDB Replica IO: s6 #page on db2169 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [05:27:27] PROBLEM - MariaDB Replica SQL: s6 #page on db2169 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [05:27:29] PROBLEM - MariaDB Events s6 on db2169 is CRITICAL: CRITICAL - Failed to query events: ERROR 2002 (HY000): Cant connect to local server through socket /run/mysqld/mysqld.sock (2) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [05:27:44] great [05:28:06] just downtimed [05:28:08] good morning, let me know if I can help [05:28:08] !incidents [05:28:09] 8611 (UNACKED) db2169 (paged)/MariaDB Replica IO: s6 (paged) [05:28:09] 8612 (UNACKED) db2169 (paged)/MariaDB Replica SQL: s6 (paged) [05:28:09] 8610 (RESOLVED) Manual (paged) for SRE managers by LSobanski (lsobanski@wikimedia.org): Test page [05:28:12] <_joe_> lmao [05:28:12] !ack [05:28:13] 8611 (ACKED) db2169 (paged)/MariaDB Replica IO: s6 (paged) [05:28:13] 8612 (ACKED) db2169 (paged)/MariaDB Replica SQL: s6 (paged) [05:28:15] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2168-2169].codfw.wmnet with reason: cloning [05:28:15] !resolve [05:28:16] 8611 (RESOLVED) db2169 (paged)/MariaDB Replica IO: s6 (paged) [05:28:16] 8612 (RESOLVED) db2169 (paged)/MariaDB Replica SQL: s6 (paged) [05:28:30] <_joe_> /msg nickserv rename marostegui ChaosMonkey [05:28:40] Cookbook failing to do their job [05:31:12] !log marostegui@cumin1004 START - Cookbook sre.mysql.depool depool db2168: moving to s7 [05:31:37] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: moving to s7 [05:36:42] !log marostegui@cumin1004 START - Cookbook sre.mysql.clone of db2168.codfw.wmnet onto db2169.codfw.wmnet [05:36:42] !log marostegui@cumin1004 END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2168.codfw.wmnet onto db2169.codfw.wmnet [05:49:24] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:54:06] (03CR) 10Marostegui: [C:03+2] mariadb: Move db2169 to s7 [puppet] - 10https://gerrit.wikimedia.org/r/1351783 (https://phabricator.wikimedia.org/T439653) (owner: 10Marostegui) [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T0600) [06:00:05] marostegui, cezmunsta, and federico3: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for Primary database switchover deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T0600). [06:09:07] (03CR) 10Trueg: [C:03+1] "nice catch!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351482 (https://phabricator.wikimedia.org/T438488) (owner: 10Gmodena) [06:23:58] FIRING: KafkaMirrorMakerConsumerMaxLag: Kafka MirrorMaker main-eqiad-to-main-codfw max lag in last 10 minutes - https://wikitech.wikimedia.org/wiki/Kafka/Administration#MirrorMaker - https://grafana.wikimedia.org/d/000000521/kafka-mirrormaker?var-mirror_name=main-eqiad-to-main-codfw - https://alerts.wikimedia.org/?q=alertname%3DKafkaMirrorMakerConsumerMaxLag [06:27:56] (03PS1) 10Muehlenhoff: Remove cuminunpriv and kerberos-ssh Cumin aliases [puppet] - 10https://gerrit.wikimedia.org/r/1351808 (https://phabricator.wikimedia.org/T439522) [06:31:48] (03CR) 10Muehlenhoff: [C:03+2] Remove cuminunpriv and kerberos-ssh Cumin aliases [puppet] - 10https://gerrit.wikimedia.org/r/1351808 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [06:38:58] RESOLVED: KafkaMirrorMakerConsumerMaxLag: Kafka MirrorMaker main-eqiad-to-main-codfw max lag in last 10 minutes - https://wikitech.wikimedia.org/wiki/Kafka/Administration#MirrorMaker - https://grafana.wikimedia.org/d/000000521/kafka-mirrormaker?var-mirror_name=main-eqiad-to-main-codfw - https://alerts.wikimedia.org/?q=alertname%3DKafkaMirrorMakerConsumerMaxLag [06:44:44] (03CR) 10Filippo Giunchedi: [C:03+1] "LGTM, very nice!" [alerts] - 10https://gerrit.wikimedia.org/r/1351431 (https://phabricator.wikimedia.org/T395448) (owner: 10Hnowlan) [07:00:05] Amir1, urbanecm, and awight: #bothumor My software never has bugs. It just develops random features. Rise for UTC morning backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T0700). [07:00:05] seanleong-wmde, dcausse, and matthiasmullie: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:00:25] o/ [07:00:26] o/ [07:00:30] I can deploy seanleon-wmde's config patch now [07:00:49] awight: sure, please go ahead [07:01:55] (03CR) 10TrainBranchBot: [C:03+2] "Approved by awight@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343917 (https://phabricator.wikimedia.org/T437577) (owner: 10Seanleong-wmde) [07:03:03] (03Merged) 10jenkins-bot: Config change for launch of stopping sending LL notifications into group1. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343917 (https://phabricator.wikimedia.org/T437577) (owner: 10Seanleong-wmde) [07:03:42] !log awight@deploy1003 Started scap sync-world: Backport for [[gerrit:1343917|Config change for launch of stopping sending LL notifications into group1. (T437577)]] [07:03:46] T437577: 🧹Deployment strategy for language links being removed from RecentChanges / Watchlists - https://phabricator.wikimedia.org/T437577 [07:05:11] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12399002 (10fgiunchedi) [07:07:19] (03PS2) 10Muehlenhoff: Only install debuerreotype on the server if base images are built [puppet] - 10https://gerrit.wikimedia.org/r/1344257 (https://phabricator.wikimedia.org/T438866) [07:08:11] !log awight@deploy1003 awight, seanleong-wmde: Backport for [[gerrit:1343917|Config change for launch of stopping sending LL notifications into group1. (T437577)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:08:18] (03PS2) 10DCausse: Do not suggest duplicates when completing special pages [extensions/CirrusSearch] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351826 (https://phabricator.wikimedia.org/T440147) [07:08:38] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 06 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [extensions/CirrusSearch] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351826 (https://phabricator.wikimedia.org/T440147) (owner: 10DCausse) [07:08:53] !log awight@deploy1003 awight, seanleong-wmde: Continuing with deployment [07:15:00] !log marostegui@cumin1004 START - Cookbook sre.mysql.pool pool db2168: Pooling back [07:15:58] FIRING: KafkaMirrorMakerConsumerMaxLag: Kafka MirrorMaker main-eqiad-to-main-codfw max lag in last 10 minutes - https://wikitech.wikimedia.org/wiki/Kafka/Administration#MirrorMaker - https://grafana.wikimedia.org/d/000000521/kafka-mirrormaker?var-mirror_name=main-eqiad-to-main-codfw - https://alerts.wikimedia.org/?q=alertname%3DKafkaMirrorMakerConsumerMaxLag [07:16:43] (03PS7) 10Muehlenhoff: ferm: Absent the NRPE check when migrating from ferm to nftables [puppet] - 10https://gerrit.wikimedia.org/r/1283620 [07:20:58] RESOLVED: KafkaMirrorMakerConsumerMaxLag: Kafka MirrorMaker main-eqiad-to-main-codfw max lag in last 10 minutes - https://wikitech.wikimedia.org/wiki/Kafka/Administration#MirrorMaker - https://grafana.wikimedia.org/d/000000521/kafka-mirrormaker?var-mirror_name=main-eqiad-to-main-codfw - https://alerts.wikimedia.org/?q=alertname%3DKafkaMirrorMakerConsumerMaxLag [07:21:13] !log awight@deploy1003 Finished scap sync-world: Backport for [[gerrit:1343917|Config change for launch of stopping sending LL notifications into group1. (T437577)]] (duration: 17m 31s) [07:21:16] T437577: 🧹Deployment strategy for language links being removed from RecentChanges / Watchlists - https://phabricator.wikimedia.org/T437577 [07:21:38] dcausse: take the wheel :-D [07:21:44] awight: thanks! [07:22:39] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy1003 using scap backport" [extensions/CirrusSearch] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351826 (https://phabricator.wikimedia.org/T440147) (owner: 10DCausse) [07:22:39] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy1003 using scap backport" [core] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351316 (https://phabricator.wikimedia.org/T440151) (owner: 10DCausse) [07:24:11] (03Merged) 10jenkins-bot: Do not suggest duplicates when completing special pages [extensions/CirrusSearch] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351826 (https://phabricator.wikimedia.org/T440147) (owner: 10DCausse) [07:25:10] !log marostegui@cumin1004 START - Cookbook sre.mysql.pool pool db2169: Pooling in s7 [07:30:05] !log dkertesz@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [07:30:08] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [07:31:25] (03Merged) 10jenkins-bot: Find canonical pages on special page completion search [core] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351316 (https://phabricator.wikimedia.org/T440151) (owner: 10DCausse) [07:31:50] !log dcausse@deploy1003 Started scap sync-world: Backport for [[gerrit:1351826|Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316|Find canonical pages on special page completion search (T440151)]] [07:31:54] T440147: Typeahead returns every Special:-page result twice - https://phabricator.wikimedia.org/T440147 [07:31:55] T440151: Canonical special page names no longer redirect to localized names in search typeahead - https://phabricator.wikimedia.org/T440151 [07:32:41] (03PS1) 10Muehlenhoff: Remove the url-downloader legacy CNAMEs [dns] - 10https://gerrit.wikimedia.org/r/1351840 (https://phabricator.wikimedia.org/T439713) [07:33:00] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db1901: test [07:33:05] (03PS2) 10Muehlenhoff: Remove the url-downloader legacy CNAMEs [dns] - 10https://gerrit.wikimedia.org/r/1351840 (https://phabricator.wikimedia.org/T439713) [07:33:35] (03CR) 10Muehlenhoff: [C:04-1] "Self -1 until https://phabricator.wikimedia.org/T439732 is resolved" [dns] - 10https://gerrit.wikimedia.org/r/1351840 (https://phabricator.wikimedia.org/T439713) (owner: 10Muehlenhoff) [07:34:02] !log fceratto@cumin1004 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test [07:34:10] FIRING: BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [07:34:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr1-magru and Telxius (2001:1498:1:966:1::251) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [07:34:39] FIRING: CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [07:37:58] !log dcausse@deploy1003 dcausse: Backport for [[gerrit:1351826|Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316|Find canonical pages on special page completion search (T440151)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:38:03] T440147: Typeahead returns every Special:-page result twice - https://phabricator.wikimedia.org/T440147 [07:38:03] T440151: Canonical special page names no longer redirect to localized names in search typeahead - https://phabricator.wikimedia.org/T440151 [07:38:31] !log dcausse@deploy1003 dcausse: Continuing with deployment [07:41:23] (03PS4) 10Muehlenhoff: Remove buster from the debian module [puppet] - 10https://gerrit.wikimedia.org/r/1344306 [07:43:56] (03CR) 10CI reject: [V:04-1] Remove buster from the debian module [puppet] - 10https://gerrit.wikimedia.org/r/1344306 (owner: 10Muehlenhoff) [07:45:02] (03PS1) 10Elukey: admin_ng: move cfssl-issuer's settings to pki.discovery.wmnet [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351901 (https://phabricator.wikimedia.org/T436809) [07:45:43] !log dcausse@deploy1003 Finished scap sync-world: Backport for [[gerrit:1351826|Do not suggest duplicates when completing special pages (T440147)]], [[gerrit:1351316|Find canonical pages on special page completion search (T440151)]] (duration: 13m 53s) [07:45:47] T440147: Typeahead returns every Special:-page result twice - https://phabricator.wikimedia.org/T440147 [07:45:48] T440151: Canonical special page names no longer redirect to localized names in search typeahead - https://phabricator.wikimedia.org/T440151 [07:45:53] matthiasmullie: can you self-deploy or do you need a deployer? [07:46:06] I can deploy, thanks! [07:46:23] alright, the floor is yours [07:46:48] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351378 (https://phabricator.wikimedia.org/T439007) (owner: 10Matthias Mullie) [07:46:48] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351379 (https://phabricator.wikimedia.org/T439943) (owner: 10Matthias Mullie) [07:46:48] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/ReaderExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351518 (https://phabricator.wikimedia.org/T439948) (owner: 10Kimberly Sarabia) [07:46:52] (03PS1) 10Giuseppe Lavagetto: [DNM] k8s cookbook to enable virtualization [cookbooks] - 10https://gerrit.wikimedia.org/r/1351902 (https://phabricator.wikimedia.org/T438287) [07:47:00] (03CR) 10Elukey: [C:03+1] Only install debuerreotype on the server if base images are built [puppet] - 10https://gerrit.wikimedia.org/r/1344257 (https://phabricator.wikimedia.org/T438866) (owner: 10Muehlenhoff) [07:47:36] (03CR) 10Elukey: [C:03+1] Remove profile::base::cuminunpriv from apt servers [puppet] - 10https://gerrit.wikimedia.org/r/1351375 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [07:47:44] (03CR) 10Elukey: [C:03+1] Remove test_krb role [puppet] - 10https://gerrit.wikimedia.org/r/1351374 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [07:48:24] (03CR) 10Daniel Kertesz: [C:03+1] liberica: Sync with liberica 0.25 idletcpconnectioncheck config [puppet] - 10https://gerrit.wikimedia.org/r/1351400 (https://phabricator.wikimedia.org/T439840) (owner: 10Vgutierrez) [07:49:08] (03Merged) 10jenkins-bot: Properly render carousel preview dialog sizing in Webkit [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351378 (https://phabricator.wikimedia.org/T439007) (owner: 10Matthias Mullie) [07:49:39] (03Merged) 10jenkins-bot: Change lead section infobox exclusion [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351379 (https://phabricator.wikimedia.org/T439943) (owner: 10Matthias Mullie) [07:49:45] (03CR) 10CI reject: [V:04-1] [DNM] k8s cookbook to enable virtualization [cookbooks] - 10https://gerrit.wikimedia.org/r/1351902 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [07:49:49] (03Merged) 10jenkins-bot: [Image Browsing] Prevent layout shift when loading carousel experiment [extensions/ReaderExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1351518 (https://phabricator.wikimedia.org/T439948) (owner: 10Kimberly Sarabia) [07:50:10] !log mlitn@deploy1003 Started scap sync-world: Backport for [[gerrit:1351378|Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379|Change lead section infobox exclusion (T439943)]], [[gerrit:1351518|[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] [07:50:17] T439007: [Image Browsing] too small preview dialog - https://phabricator.wikimedia.org/T439007 [07:50:17] T439943: [Image Browsing] Regression: lead section infobox images appear in carousel again - https://phabricator.wikimedia.org/T439943 [07:50:18] T439948: [Image Browsing] FOUC when loading carousel through experiment - https://phabricator.wikimedia.org/T439948 [07:54:26] !log mlitn@deploy1003 mlitn, ksarabia: Backport for [[gerrit:1351378|Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379|Change lead section infobox exclusion (T439943)]], [[gerrit:1351518|[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:54:55] (03CR) 10Muehlenhoff: [C:03+2] Only install debuerreotype on the server if base images are built [puppet] - 10https://gerrit.wikimedia.org/r/1344257 (https://phabricator.wikimedia.org/T438866) (owner: 10Muehlenhoff) [07:55:11] !log mlitn@deploy1003 mlitn, ksarabia: Continuing with deployment [08:00:05] andre and brennen: MediaWiki train - Utc-0+Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T0800). Please do the needful. [08:00:07] o/ No hurry. [08:00:29] !log mlitn@deploy1003 Finished scap sync-world: Backport for [[gerrit:1351378|Properly render carousel preview dialog sizing in Webkit (T439007)]], [[gerrit:1351379|Change lead section infobox exclusion (T439943)]], [[gerrit:1351518|[Image Browsing] Prevent layout shift when loading carousel experiment (T439948)]] (duration: 10m 18s) [08:00:35] T439007: [Image Browsing] too small preview dialog - https://phabricator.wikimedia.org/T439007 [08:00:36] T439943: [Image Browsing] Regression: lead section infobox images appear in carousel again - https://phabricator.wikimedia.org/T439943 [08:00:36] T439948: [Image Browsing] FOUC when loading carousel through experiment - https://phabricator.wikimedia.org/T439948 [08:00:55] !log UTC morning backport window done [08:00:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:01:14] Yay. Train time soon. [08:03:14] (03CR) 10Elukey: [C:03+1] netbox: add an option to connect to netbox-next [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [08:03:49] (03CR) 10Trueg: [C:03+1] "lgtm" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351488 (https://phabricator.wikimedia.org/T439942) (owner: 10Gmodena) [08:04:23] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling back [08:06:25] (03CR) 10Ayounsi: [C:03+2] netbox: add an option to connect to netbox-next [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [08:07:50] (03PS1) 10Fabfur: haproxykafka: support json logging configuration [puppet] - 10https://gerrit.wikimedia.org/r/1351916 (https://phabricator.wikimedia.org/T440268) [08:07:52] (03PS1) 10Fabfur: profile::cache: introduce json logging in haproxy [puppet] - 10https://gerrit.wikimedia.org/r/1351917 (https://phabricator.wikimedia.org/T440268) [08:09:16] (03CR) 10Fabfur: [C:03+1] "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/1351400 (https://phabricator.wikimedia.org/T439840) (owner: 10Vgutierrez) [08:09:49] (03Merged) 10jenkins-bot: netbox: add an option to connect to netbox-next [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [08:10:24] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling in s7 [08:10:25] (03CR) 10CI reject: [V:04-1] haproxykafka: support json logging configuration [puppet] - 10https://gerrit.wikimedia.org/r/1351916 (https://phabricator.wikimedia.org/T440268) (owner: 10Fabfur) [08:11:02] (03CR) 10CI reject: [V:04-1] profile::cache: introduce json logging in haproxy [puppet] - 10https://gerrit.wikimedia.org/r/1351917 (https://phabricator.wikimedia.org/T440268) (owner: 10Fabfur) [08:11:14] 10SRE-Access-Requests, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Backup SSH key for Guillaume Lederrey (gehel) - https://phabricator.wikimedia.org/T440278 (10Gehel) 03NEW [08:11:31] 10SRE-Access-Requests, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Backup SSH key for Guillaume Lederrey (gehel) - https://phabricator.wikimedia.org/T440278#12399345 (10Gehel) [08:11:39] (03CR) 10Daniel Kertesz: [C:03+1] "looks good to me, but better wait for other people's review :)" [puppet] - 10https://gerrit.wikimedia.org/r/1347624 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [08:11:59] (03CR) 10Slyngshede: [C:03+1] "Looks good to me." [puppet] - 10https://gerrit.wikimedia.org/r/1347624 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [08:13:33] (03PS1) 10Gehel: admin: backup SSH key for Guillaume (gehel) [puppet] - 10https://gerrit.wikimedia.org/r/1351924 (https://phabricator.wikimedia.org/T440278) [08:13:56] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [08:14:06] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox: sync [08:14:09] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [08:14:10] (03PS2) 10Btullis: apereo_cas: Add a grant_types parameter for OIDC services [puppet] - 10https://gerrit.wikimedia.org/r/1351438 (https://phabricator.wikimedia.org/T435602) [08:14:12] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [08:14:20] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [08:14:33] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db1901: test [08:14:35] !log fceratto@cumin1004 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test [08:14:43] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db1901: test [08:14:45] !log fceratto@cumin1004 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test [08:15:01] (03CR) 10Muehlenhoff: [C:03+1] "Looks good and verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1351924 (https://phabricator.wikimedia.org/T440278) (owner: 10Gehel) [08:15:27] RECOVERY - Check unit status of kube-reload-certs on deploy1003 is OK: OK: Status of the systemd unit kube-reload-certs https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [08:16:11] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1351438 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [08:16:47] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db1901: test [08:16:50] !log fceratto@cumin1004 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test [08:16:56] (03PS1) 10TrainBranchBot: group0 to 1.47.0-wmf.23 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351925 (https://phabricator.wikimedia.org/T438219) [08:16:59] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by aklapper@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351925 (https://phabricator.wikimedia.org/T438219) (owner: 10TrainBranchBot) [08:17:57] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [08:18:39] (03Merged) 10jenkins-bot: group0 to 1.47.0-wmf.23 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351925 (https://phabricator.wikimedia.org/T438219) (owner: 10TrainBranchBot) [08:22:24] (03PS1) 10Elukey: profile::k8s::deployment_server: fix kube-reload-certs.sh [puppet] - 10https://gerrit.wikimedia.org/r/1351927 (https://phabricator.wikimedia.org/T439409) [08:24:25] (03PS3) 10Btullis: apereo_cas: Add a grant_types parameter for OIDC services [puppet] - 10https://gerrit.wikimedia.org/r/1351438 (https://phabricator.wikimedia.org/T435602) [08:26:52] (03CR) 10Gehel: [C:03+2] admin: backup SSH key for Guillaume (gehel) [puppet] - 10https://gerrit.wikimedia.org/r/1351924 (https://phabricator.wikimedia.org/T440278) (owner: 10Gehel) [08:27:03] (03CR) 10JMeybohm: [C:03+1] admin_ng: move cfssl-issuer's settings to pki.discovery.wmnet [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351901 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [08:28:10] !log aklapper@deploy1003 rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs T438219 [08:28:14] T438219: 1.47.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T438219 [08:28:49] (03PS1) 10Slyngshede: geo-maps: Update with Meta changes for October 2026 [dns] - 10https://gerrit.wikimedia.org/r/1351928 [08:28:53] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db1901: test [08:28:57] !log fceratto@cumin1004 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test [08:29:28] !log marostegui@cumin1004 START - Cookbook sre.mysql.depool depool db2169: moving cloning [08:29:43] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2169: moving cloning [08:29:53] !log marostegui@cumin1004 START - Cookbook sre.mysql.depool depool db2228: moving cloning [08:30:10] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2228: moving cloning [08:30:39] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db[2169,2228].codfw.wmnet with reason: cloning [08:31:43] (03PS1) 10Marostegui: mariadb: Move db2228 to s7 [puppet] - 10https://gerrit.wikimedia.org/r/1351929 (https://phabricator.wikimedia.org/T439653) [08:35:32] (03PS1) 10Daniel Kertesz: varnish: fix race condition in flaky test [puppet] - 10https://gerrit.wikimedia.org/r/1351932 [08:35:33] (03CR) 10Marostegui: [C:03+2] mariadb: Move db2228 to s7 [puppet] - 10https://gerrit.wikimedia.org/r/1351929 (https://phabricator.wikimedia.org/T439653) (owner: 10Marostegui) [08:37:02] (03CR) 10Daniel Kertesz: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1351932 (owner: 10Daniel Kertesz) [08:43:27] (03CR) 10Daniel Kertesz: "`" [puppet] - 10https://gerrit.wikimedia.org/r/1351932 (owner: 10Daniel Kertesz) [08:43:42] (03PS1) 10Dpogorzelski: liftwing-studio: export LiteLLM metrics [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351938 (https://phabricator.wikimedia.org/T440198) [08:47:28] (03PS1) 10JMeybohm: conftool: Add remaining k8s nodes to conftool-data [puppet] - 10https://gerrit.wikimedia.org/r/1351940 (https://phabricator.wikimedia.org/T420729) [08:54:39] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [08:56:39] (03CR) 10Muehlenhoff: [C:03+2] Remove test_krb role [puppet] - 10https://gerrit.wikimedia.org/r/1351374 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [08:56:46] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [08:58:18] (03CR) 10Klausman: [C:03+1] liftwing-studio: export LiteLLM metrics [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351938 (https://phabricator.wikimedia.org/T440198) (owner: 10Dpogorzelski) [08:59:11] (03PS1) 10Jelto: cache: tls_terminator: add provenance net=gitlab-runner-digitalocean [puppet] - 10https://gerrit.wikimedia.org/r/1351947 (https://phabricator.wikimedia.org/T425441) [09:00:00] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: export LiteLLM metrics [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351938 (https://phabricator.wikimedia.org/T440198) (owner: 10Dpogorzelski) [09:00:40] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [09:00:43] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [09:01:10] (03PS1) 10JMeybohm: k8s: Remove LVS L2 adjacency handling [cookbooks] - 10https://gerrit.wikimedia.org/r/1351949 (https://phabricator.wikimedia.org/T420729) [09:02:25] (03PS2) 10Jgiannelos: mobileapps: Disable debugging on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346017 [09:02:29] (03Merged) 10jenkins-bot: liftwing-studio: export LiteLLM metrics [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351938 (https://phabricator.wikimedia.org/T440198) (owner: 10Dpogorzelski) [09:02:32] !log dkertesz@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-esams and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [09:02:35] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [09:02:52] (03CR) 10Giuseppe Lavagetto: [C:03+1] profile::k8s::deployment_server: fix kube-reload-certs.sh [puppet] - 10https://gerrit.wikimedia.org/r/1351927 (https://phabricator.wikimedia.org/T439409) (owner: 10Elukey) [09:03:24] (03CR) 10Elukey: [C:03+2] profile::k8s::deployment_server: fix kube-reload-certs.sh [puppet] - 10https://gerrit.wikimedia.org/r/1351927 (https://phabricator.wikimedia.org/T439409) (owner: 10Elukey) [09:03:57] (03PS2) 10Giuseppe Lavagetto: [DNM] k8s cookbook to enable virtualization [cookbooks] - 10https://gerrit.wikimedia.org/r/1351902 (https://phabricator.wikimedia.org/T438287) [09:05:25] (03CR) 10Isabelle Hurbain-Palatin: [C:03+1] mobileapps: Disable debugging on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346017 (owner: 10Jgiannelos) [09:05:52] (03PS3) 10Giuseppe Lavagetto: [DNM] k8s cookbook to enable virtualization [cookbooks] - 10https://gerrit.wikimedia.org/r/1351902 (https://phabricator.wikimedia.org/T438287) [09:05:59] (03PS2) 10Blake: mcrouter: switch updateStrategy to OnDelete [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348640 (https://phabricator.wikimedia.org/T420223) [09:07:39] PROBLEM - Host wikikube-worker1016 is DOWN: PING CRITICAL - Packet loss = 100% [09:08:19] RECOVERY - Host wikikube-worker1016 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [09:08:19] (03CR) 10CI reject: [V:04-1] [DNM] k8s cookbook to enable virtualization [cookbooks] - 10https://gerrit.wikimedia.org/r/1351902 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [09:08:29] jouncebot: nowandnext [09:08:29] For the next 0 hour(s) and 51 minute(s): MediaWiki train - Utc-0+Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T0800) [09:08:29] In 0 hour(s) and 51 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1000) [09:08:42] choo choo. [09:08:44] (03CR) 10Giuseppe Lavagetto: [C:04-2] "This does not work on dells. If we don't reboot dells the change to bios will remain pending and block the redfish api" [cookbooks] - 10https://gerrit.wikimedia.org/r/1346499 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [09:09:01] (03CR) 10Arnaudb: [C:03+1] "looks good to me, thanks" [puppet] - 10https://gerrit.wikimedia.org/r/1351947 (https://phabricator.wikimedia.org/T425441) (owner: 10Jelto) [09:11:48] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [09:12:09] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db1901: test [09:12:12] (03CR) 10Arnaudb: [C:03+1] "looks good to me! thanks" [puppet] - 10https://gerrit.wikimedia.org/r/1351534 (https://phabricator.wikimedia.org/T393763) (owner: 10Dzahn) [09:12:48] !log marostegui@cumin1004 START - Cookbook sre.mysql.depool depool db2168: testing [09:13:01] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2168: testing [09:13:29] !log marostegui@cumin1004 START - Cookbook sre.mysql.pool pool db2168: Pooling in s7 [09:13:38] <_joe_> jouncebot: now [09:13:38] For the next 0 hour(s) and 46 minute(s): MediaWiki train - Utc-0+Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T0800) [09:13:46] !log oblivian@cumin1004 START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2011-2018].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [09:13:50] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2011.codfw.wmnet [09:14:24] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2011.codfw.wmnet [09:14:47] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2011.codfw.wmnet [09:14:49] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2011.codfw.wmnet [09:15:03] !log oblivian@cumin1004 END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D{wikikube-worker[2011-2018].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [09:15:36] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2012.codfw.wmnet [09:15:39] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2012.codfw.wmnet [09:16:11] (03CR) 10Elukey: [C:03+2] kubernetes::deployment_server: check and reload jaeger for cert changes [puppet] - 10https://gerrit.wikimedia.org/r/1347669 (https://phabricator.wikimedia.org/T439246) (owner: 10Giuseppe Lavagetto) [09:16:31] (03PS2) 10Daimona Eaytoy: tables-catalog: Drop ce_worklist_articles [puppet] - 10https://gerrit.wikimedia.org/r/1324325 (https://phabricator.wikimedia.org/T426102) [09:16:54] (03CR) 10Jelto: [C:03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1351940 (https://phabricator.wikimedia.org/T420729) (owner: 10JMeybohm) [09:17:03] (03CR) 10Ladsgroup: [V:03+2 C:03+2] tables-catalog: Drop ce_worklist_articles [puppet] - 10https://gerrit.wikimedia.org/r/1324325 (https://phabricator.wikimedia.org/T426102) (owner: 10Daimona Eaytoy) [09:19:05] 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, and 2 others: Ensure cloudvirt capacity is more evenly spread out among racks - https://phabricator.wikimedia.org/T424658#12399593 (10fgiunchedi) @Jclark-ctr @VRiley-WMF we'll be moving four cloudvirt out of D5 and into C8/E4/F4 as per task description, would you... [09:24:11] (03PS1) 10Trueg: WDQS: Set the default list of SPARQL prefixes to be injected. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351954 (https://phabricator.wikimedia.org/T438476) [09:24:11] (03CR) 10Trueg: "This CR is supposed to be used instead of https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1343616 to use the newly introdu" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351954 (https://phabricator.wikimedia.org/T438476) (owner: 10Trueg) [09:26:28] PROBLEM - Check unit status of kube-reload-certs on deploy1003 is CRITICAL: CRITICAL: Status of the systemd unit kube-reload-certs https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [09:28:59] (03CR) 10Vgutierrez: [C:03+1] varnish: fix race condition in flaky test [puppet] - 10https://gerrit.wikimedia.org/r/1351932 (owner: 10Daniel Kertesz) [09:29:47] (03CR) 10Fabfur: [C:03+1] varnish: fix race condition in flaky test [puppet] - 10https://gerrit.wikimedia.org/r/1351932 (owner: 10Daniel Kertesz) [09:32:00] (03CR) 10Gkyziridis: [C:03+2] ml-services: deploy latest reference-quality image in revision-models [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351300 (https://phabricator.wikimedia.org/T435491) (owner: 10Gkyziridis) [09:32:17] (03PS2) 10Blake: mcrouter: add a toggle for updateStrategy. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351955 (https://phabricator.wikimedia.org/T420223) [09:32:28] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [09:32:33] (03Abandoned) 10Blake: mcrouter: switch updateStrategy to OnDelete [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348640 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [09:33:57] (03PS2) 10Blake: mcrouter: test connection reset/keepalive options [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348641 (https://phabricator.wikimedia.org/T420223) [09:34:22] (03Merged) 10jenkins-bot: ml-services: deploy latest reference-quality image in revision-models [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351300 (https://phabricator.wikimedia.org/T435491) (owner: 10Gkyziridis) [09:36:25] (03PS1) 10Btullis: idp_test: Add a dummy client secret for wmf_s3_login [labs/private] - 10https://gerrit.wikimedia.org/r/1351958 (https://phabricator.wikimedia.org/T435602) [09:37:03] 10ops-eqiad, 06Data-Persistence, 06DC-Ops, 13Patch-For-Review: ms-be1074: /dev/sdn issue - https://phabricator.wikimedia.org/T440281#12399645 (10Ladsgroup) [09:37:21] (03CR) 10Btullis: [V:03+2 C:03+2] idp_test: Add a dummy client secret for wmf_s3_login [labs/private] - 10https://gerrit.wikimedia.org/r/1351958 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [09:41:17] (03CR) 10JMeybohm: [C:03+1] mcrouter: add a toggle for updateStrategy. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351955 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [09:43:03] (03PS2) 10Fabfur: haproxykafka: support json logging configuration [puppet] - 10https://gerrit.wikimedia.org/r/1351916 (https://phabricator.wikimedia.org/T440268) [09:43:03] (03PS2) 10Fabfur: profile::cache: introduce json logging in haproxy [puppet] - 10https://gerrit.wikimedia.org/r/1351917 (https://phabricator.wikimedia.org/T440268) [09:43:17] (03CR) 10Blake: [C:03+2] mcrouter: add a toggle for updateStrategy. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351955 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [09:45:04] (03Merged) 10jenkins-bot: mcrouter: add a toggle for updateStrategy. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351955 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [09:45:22] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply [09:45:36] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply [09:45:38] !log blake@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply [09:45:42] !log blake@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply [09:45:54] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 3 others: wikikube-worker1152.eqiad.wmnet networking issue - https://phabricator.wikimedia.org/T438819#12399695 (10Jelto) >>! In T438819#12363816, @Jclark-ctr wrote: > I have installed a new NIC, and it has link. It is also showing up in iDRAC. However,... [09:46:03] (03CR) 10CI reject: [V:04-1] haproxykafka: support json logging configuration [puppet] - 10https://gerrit.wikimedia.org/r/1351916 (https://phabricator.wikimedia.org/T440268) (owner: 10Fabfur) [09:46:10] (03CR) 10CI reject: [V:04-1] profile::cache: introduce json logging in haproxy [puppet] - 10https://gerrit.wikimedia.org/r/1351917 (https://phabricator.wikimedia.org/T440268) (owner: 10Fabfur) [09:46:16] (03CR) 10Filippo Giunchedi: [C:03+2] install_server: new script detect-os-device.sh [puppet] - 10https://gerrit.wikimedia.org/r/1351276 (https://phabricator.wikimedia.org/T439831) (owner: 10Filippo Giunchedi) [09:46:23] jouncebot: nowandnext [09:46:23] For the next 0 hour(s) and 13 minute(s): MediaWiki train - Utc-0+Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T0800) [09:46:23] In 0 hour(s) and 13 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1000) [09:46:25] (03CR) 10Filippo Giunchedi: [C:03+2] install_server: use raid1-jbod-efi for new cloudcephosd hosts [puppet] - 10https://gerrit.wikimedia.org/r/1351277 (https://phabricator.wikimedia.org/T439831) (owner: 10Filippo Giunchedi) [09:47:32] (03CR) 10CWilliams: mysql: Fix fleetwide RO/RW setting (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1351376 (https://phabricator.wikimedia.org/T438831) (owner: 10Federico Ceratto) [09:49:24] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:51:01] (03PS2) 10Arthur taylor: Enable mobile editing for all Wikidata editors [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347373 (https://phabricator.wikimedia.org/T437054) (owner: 10Sadiya.mohammed13) [09:51:20] 10ops-codfw, 10database-backups, 06DBA, 06DC-Ops: db2201 (backup source s5 and x1) crashed - https://phabricator.wikimedia.org/T440283 (10Marostegui) 03NEW [09:51:33] 10ops-codfw, 10database-backups, 06DBA, 06DC-Ops: db2201 (backup source s5 and x1) crashed - https://phabricator.wikimedia.org/T440283#12399727 (10Marostegui) p:05Triage→03Medium [09:52:17] !log marostegui@cumin1004 START - Cookbook sre.mysql.pool pool db2169: Pooling back [09:52:32] RECOVERY - mysqld processes on db2201 is OK: PROCS OK: 2 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [09:52:32] RECOVERY - MariaDB Replica SQL: x1 on db2201 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [09:52:32] RECOVERY - MariaDB Replica SQL: s5 on db2201 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [09:52:32] RECOVERY - MariaDB Replica IO: x1 on db2201 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [09:52:32] RECOVERY - MariaDB Replica IO: s5 on db2201 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [09:52:46] RECOVERY - MariaDB read only s5 on db2201 is OK: Version 10.11.18-MariaDB-log, Uptime 69s, read_only: True, event_scheduler: True, 6119.38 QPS, connection latency: 0.025860s, query latency: 0.000537s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [09:52:46] RECOVERY - MariaDB read only x1 on db2201 is OK: Version 10.11.18-MariaDB-log, Uptime 48s, read_only: True, event_scheduler: True, 5360.39 QPS, connection latency: 0.030491s, query latency: 0.000703s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [09:53:00] !log marostegui@cumin1004 START - Cookbook sre.mysql.pool pool db2228: pooling in s7 [09:54:20] (03PS1) 10Elukey: kube-reload-certs: allow different labels then "release" [puppet] - 10https://gerrit.wikimedia.org/r/1351965 (https://phabricator.wikimedia.org/T439409) [09:54:49] (03PS2) 10Blake: mcrouter: switch updateStrategy to OnDelete. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351964 (https://phabricator.wikimedia.org/T420223) [09:55:38] !log filippo@cumin1004 START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm [09:55:43] (03PS1) 10Marostegui: db2207,db2209,db1218: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1351967 [09:56:22] bjensen: o/ lemme know when the keep alive change is deployed, I am curious! [09:56:56] (03CR) 10CI reject: [V:04-1] kube-reload-certs: allow different labels then "release" [puppet] - 10https://gerrit.wikimedia.org/r/1351965 (https://phabricator.wikimedia.org/T439409) (owner: 10Elukey) [09:57:01] elukey: will do! still in pre-work at the moment, trying to ensure the change doesn't get everywhere all at once :) [09:57:36] !log elukey@deploy1003 Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297 [09:57:40] T437297: Test the Docker Registry's GC with Releng/MediaWiki images - https://phabricator.wikimedia.org/T437297 [09:57:56] !log elukey@deploy1003 Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297 (duration: 00m 19s) [09:58:06] (03CR) 10Marostegui: [C:03+2] db2207,db2209,db1218: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1351967 (owner: 10Marostegui) [09:58:12] !log elukey@deploy1003 Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297 [09:58:36] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2168: Pooling in s7 [09:58:38] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . [09:58:47] !log gkyziridis@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1000) [10:00:05] elukey: A patch you scheduled for MediaWiki infrastructure (UTC mid-day) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [10:00:32] RECOVERY - MariaDB Replica Lag: s5 on db2201 is OK: OK slave_sql_lag Replication lag: 0.35 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [10:02:57] !log `elukey@registry1004:~$ time sudo -u docker-registry /usr/bin/docker-registry garbage-collect /etc/docker/registry/config-restricted.yml -m | tee gc_run_06102026` - T437297 [10:02:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:03:00] T437297: Test the Docker Registry's GC with Releng/MediaWiki images - https://phabricator.wikimedia.org/T437297 [10:03:45] (03CR) 10JMeybohm: [C:03+1] mcrouter: switch updateStrategy to OnDelete. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351964 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [10:03:48] bjensen: you remove all the fun! :D [10:05:21] (03CR) 10Gkyziridis: [C:03+2] ml-services: deploy latest liftwing-openapi-server image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351302 (https://phabricator.wikimedia.org/T435491) (owner: 10Gkyziridis) [10:05:30] i'm something of an excitement minimalist XD [10:05:54] (03CR) 10Blake: [C:03+2] mcrouter: switch updateStrategy to OnDelete. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351964 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [10:08:04] (03Merged) 10jenkins-bot: ml-services: deploy latest liftwing-openapi-server image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351302 (https://phabricator.wikimedia.org/T435491) (owner: 10Gkyziridis) [10:08:26] (03Merged) 10jenkins-bot: mcrouter: switch updateStrategy to OnDelete. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351964 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [10:08:39] !log marostegui@cumin1004 START - Cookbook sre.mysql.depool depool pc1021: testing [10:08:39] !log marostegui@cumin1004 START - Cookbook sre.mysql.parsercache [10:08:40] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [10:08:41] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: testing [10:09:20] !log installing curl security updates [10:09:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:09:26] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . [10:09:34] !log Upgrade pc1 to mariadb 10.11.19 T436082 [10:09:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:09:37] T436082: Compile and package MariaDB 10.11.19 - https://phabricator.wikimedia.org/T436082 [10:09:42] !log gkyziridis@deploy1003 helmfile [ml-serve-codfw] 'sync' command on namespace 'liftwing-openapi-server' for release 'main' . [10:10:10] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply [10:10:35] (03CR) 10JMeybohm: [C:03+1] various: replace url-downloader cname. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348555 (https://phabricator.wikimedia.org/T439713) (owner: 10Blake) [10:10:47] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on pc2021.codfw.wmnet,pc1021.eqiad.wmnet with reason: upgrading [10:10:48] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply [10:10:49] !log blake@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply [10:10:56] !log blake@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply [10:12:13] !log filippo@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage [10:12:43] !log filippo@cumin1004 START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm [10:13:10] !log marostegui@cumin1004 START - Cookbook sre.mysql.pool pool pc1021: Pooling back pc1 [10:13:10] !log marostegui@cumin1004 START - Cookbook sre.mysql.parsercache [10:13:19] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [10:13:19] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Pooling back pc1 [10:14:50] !log mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 5 visualeditor-editor (T406724) [10:14:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:14:54] T406724: Clean up watchlist and user properties of users if they don't log in for certain time - https://phabricator.wikimedia.org/T406724 [10:18:33] !log filippo@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1055.eqiad.wmnet with reason: host reimage [10:18:59] !log filippo@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm [10:19:04] (03PS4) 10Gmodena: wdqs: add wikidata prefixes config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343616 (https://phabricator.wikimedia.org/T438476) [10:19:42] (03CR) 10JMeybohm: [C:03+1] mcrouter: test connection reset/keepalive options [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348641 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [10:19:50] !log filippo@cumin1004 START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm [10:22:40] (03PS3) 10Fabfur: haproxykafka: support json logging configuration [puppet] - 10https://gerrit.wikimedia.org/r/1351916 (https://phabricator.wikimedia.org/T440268) [10:22:40] (03PS3) 10Fabfur: profile::cache: introduce json logging in haproxy [puppet] - 10https://gerrit.wikimedia.org/r/1351917 (https://phabricator.wikimedia.org/T440268) [10:23:16] (03PS3) 10Blake: mcrouter: test connection reset/keepalive options [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348641 (https://phabricator.wikimedia.org/T420223) [10:24:08] (03CR) 10Jelto: "two comments in-line" [cookbooks] - 10https://gerrit.wikimedia.org/r/1351949 (https://phabricator.wikimedia.org/T420729) (owner: 10JMeybohm) [10:24:40] (03CR) 10Gmodena: [C:03+2] wdqs: enforce quarkus rest client connection pool [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351482 (https://phabricator.wikimedia.org/T438488) (owner: 10Gmodena) [10:24:57] (03CR) 10Gmodena: [C:03+2] wdqs: set request timeout to 60s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351488 (https://phabricator.wikimedia.org/T439942) (owner: 10Gmodena) [10:25:10] (03PS2) 10Daniel Kertesz: varnish: fix race condition in flaky test [puppet] - 10https://gerrit.wikimedia.org/r/1351932 [10:27:47] (03Merged) 10jenkins-bot: wdqs: set request timeout to 60s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351488 (https://phabricator.wikimedia.org/T439942) (owner: 10Gmodena) [10:30:32] (03PS3) 10Daniel Kertesz: varnish: fix race condition in flaky test [puppet] - 10https://gerrit.wikimedia.org/r/1351932 [10:30:58] (03CR) 10JMeybohm: [C:03+1] mcrouter: test connection reset/keepalive options [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348641 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [10:31:28] (03CR) 10CI reject: [V:04-1] varnish: fix race condition in flaky test [puppet] - 10https://gerrit.wikimedia.org/r/1351932 (owner: 10Daniel Kertesz) [10:32:09] (03CR) 10Hnowlan: [C:03+2] cdn: create non-paging, ticketing ATS backend error rate alert [alerts] - 10https://gerrit.wikimedia.org/r/1337633 (https://phabricator.wikimedia.org/T436049) (owner: 10Hnowlan) [10:33:23] (03PS4) 10Daniel Kertesz: varnish: fix race condition in flaky test [puppet] - 10https://gerrit.wikimedia.org/r/1351932 [10:37:07] !log filippo@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" [10:37:12] (03Merged) 10jenkins-bot: cdn: create non-paging, ticketing ATS backend error rate alert [alerts] - 10https://gerrit.wikimedia.org/r/1337633 (https://phabricator.wikimedia.org/T436049) (owner: 10Hnowlan) [10:37:23] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2169: Pooling back [10:38:06] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2228: pooling in s7 [10:38:08] !log filippo@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" [10:38:08] !log filippo@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1055.eqiad.wmnet with OS bookworm [10:38:12] !log filippo@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm [10:40:07] !log filippo@cumin1004 START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm [10:40:11] (03PS5) 10Trueg: wdqs: add wikidata prefixes config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343616 (https://phabricator.wikimedia.org/T438476) (owner: 10Gmodena) [10:41:43] (03CR) 10Daniel Kertesz: "0 tests failed, 0 tests skipped, 45 tests passed" [puppet] - 10https://gerrit.wikimedia.org/r/1351932 (owner: 10Daniel Kertesz) [10:42:56] (03PS6) 10Trueg: wdqs: add wikidata prefixes config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343616 (https://phabricator.wikimedia.org/T438476) (owner: 10Gmodena) [10:43:49] (03PS3) 10Trueg: wdqs: proxy: version bump. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351340 (https://phabricator.wikimedia.org/T438488) (owner: 10Gmodena) [10:43:57] (03CR) 10Trueg: [C:03+2] wdqs: proxy: version bump. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351340 (https://phabricator.wikimedia.org/T438488) (owner: 10Gmodena) [10:43:59] (03CR) 10Trueg: [C:03+2] wdqs: add wikidata prefixes config (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343616 (https://phabricator.wikimedia.org/T438476) (owner: 10Gmodena) [10:45:03] (03CR) 10Vgutierrez: [C:03+1] ":) no need for the `Co-Authored-By`, looks good 😄" [puppet] - 10https://gerrit.wikimedia.org/r/1351932 (owner: 10Daniel Kertesz) [10:45:49] (03CR) 10Daniel Kertesz: [C:03+2] varnish: fix race condition in flaky test [puppet] - 10https://gerrit.wikimedia.org/r/1351932 (owner: 10Daniel Kertesz) [10:46:45] (03Merged) 10jenkins-bot: wdqs: proxy: version bump. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351340 (https://phabricator.wikimedia.org/T438488) (owner: 10Gmodena) [10:46:47] (03Merged) 10jenkins-bot: wdqs: add wikidata prefixes config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343616 (https://phabricator.wikimedia.org/T438476) (owner: 10Gmodena) [10:55:38] (03CR) 10Gmodena: [V:03+2 C:03+2] wdqs: enforce quarkus rest client connection pool [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351482 (https://phabricator.wikimedia.org/T438488) (owner: 10Gmodena) [10:56:27] !log filippo@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage [10:58:15] (03PS1) 10Btullis: idp_test: Add an OIDC client for the wmf-s3-login tool [puppet] - 10https://gerrit.wikimedia.org/r/1352001 (https://phabricator.wikimedia.org/T435602) [10:59:04] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.6 point update - https://phabricator.wikimedia.org/T434866#12399965 (10MoritzMuehlenhoff) [11:01:49] !log filippo@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1056.eqiad.wmnet with reason: host reimage [11:02:38] (03PS4) 10Gmodena: wdqs: enforce quarkus rest client connection pool [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351482 (https://phabricator.wikimedia.org/T438488) [11:02:48] !log jmm@cumin2003 START - Cookbook sre.dns.netbox [11:04:05] (03CR) 10Blake: [C:03+2] mcrouter: test connection reset/keepalive options [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348641 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [11:04:47] (03CR) 10Trueg: [C:03+2] wdqs: enforce quarkus rest client connection pool [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351482 (https://phabricator.wikimedia.org/T438488) (owner: 10Gmodena) [11:04:52] (03CR) 10Btullis: "This use of idp-test is only intended to be short-term." [puppet] - 10https://gerrit.wikimedia.org/r/1352001 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [11:05:18] !log jmm@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:05:18] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1351438 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [11:05:29] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352001 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [11:05:31] RECOVERY - MariaDB Replica Lag: x1 on db2201 is OK: OK slave_sql_lag Replication lag: 0.12 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [11:06:37] (03Merged) 10jenkins-bot: mcrouter: test connection reset/keepalive options [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348641 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [11:07:20] (03Merged) 10jenkins-bot: wdqs: enforce quarkus rest client connection pool [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351482 (https://phabricator.wikimedia.org/T438488) (owner: 10Gmodena) [11:08:40] !log blake@deploy1003 helmfile [staging] START helmfile.d/services/mw-mcrouter: apply [11:09:19] !log blake@deploy1003 helmfile [staging] DONE helmfile.d/services/mw-mcrouter: apply [11:14:18] (03CR) 10Muehlenhoff: [C:03+2] Add service catalogue entry for ldap-ro-test [puppet] - 10https://gerrit.wikimedia.org/r/1347624 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [11:21:03] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352011 [11:21:09] !log filippo@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" [11:21:25] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352012 [11:21:33] !log filippo@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - filippo@cumin1004" [11:21:33] !log filippo@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1056.eqiad.wmnet with OS bookworm [11:29:25] (03CR) 10Muehlenhoff: [C:03+2] Configure both LDAP replicas to be manageable by ldap-ro-test [puppet] - 10https://gerrit.wikimedia.org/r/1347630 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [11:29:43] (03PS1) 10Dillon: InitialiseSettings: Enable ModeratorToolkit on four group2 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352018 (https://phabricator.wikimedia.org/T437617) [11:29:59] (03PS1) 10Dillon: InitialiseSettings: Enable ModeratorToolkit on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352019 (https://phabricator.wikimedia.org/T437617) [11:30:19] 10ops-eqiad, 06SRE, 06DC-Ops: Power Supply - Status - issue on wikikube-worker1080:9290 - https://phabricator.wikimedia.org/T440202#12400050 (10Jclark-ctr) This server is out of warranty, which expired on August 9, 2026. The PSU has failed, Replaced psu from decom server cloudcontrol1008-dev [11:30:28] 10ops-eqiad, 06SRE, 06DC-Ops: Power Supply - Status - issue on wikikube-worker1080:9290 - https://phabricator.wikimedia.org/T440202#12400051 (10Jclark-ctr) 05Open→03Resolved a:03Jclark-ctr [11:31:31] (03PS1) 10Ozge: ml-services: Add llm-qwen38-27b-experiments and reduce staging resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352020 (https://phabricator.wikimedia.org/T440266) [11:33:21] !log installing chromium security updates [11:33:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:34:10] FIRING: BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:34:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr1-magru and Telxius (2001:1498:1:966:1::251) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [11:34:39] FIRING: CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [11:36:07] (03PS2) 10Ozge: ml-services: Add llm-qwen38-27b-experiments and reduce staging resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352020 (https://phabricator.wikimedia.org/T440266) [11:36:20] (03PS3) 10Ozge: ml-services: Add llm-qwen36-27b-experiments and reduce staging resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352020 (https://phabricator.wikimedia.org/T440266) [11:37:07] (03PS1) 10Muehlenhoff: thumbor-plugins: Rebuild against latest package versions in Trixie [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1352024 [11:37:14] (03CR) 10Ilias Sarantopoulos: ml-services: Add llm-qwen36-27b-experiments and reduce staging resources (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352020 (https://phabricator.wikimedia.org/T440266) (owner: 10Ozge) [11:39:10] (03PS4) 10Ozge: ml-services: Add llm-qwen36-27b-experiments and reduce staging resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352020 (https://phabricator.wikimedia.org/T440266) [11:39:15] (03PS1) 10Tryvix1509: ltwiki: Enable extended confirmed user group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352025 (https://phabricator.wikimedia.org/T440216) [11:39:19] (03PS5) 10Ozge: ml-services: Add qwen36-27b-experiments and reduce staging resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352020 (https://phabricator.wikimedia.org/T440266) [11:40:00] (03CR) 10Ozge: ml-services: Add qwen36-27b-experiments and reduce staging resources (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352020 (https://phabricator.wikimedia.org/T440266) (owner: 10Ozge) [11:41:05] !log blake@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply [11:41:19] !log blake@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply [11:45:09] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T439299#12400114 (10BTullis) OK, so it's the same drive that dropped out as last time. {F106623066,width=60%} Let's see if I had actually upgraded the RAID contro... [11:46:45] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T439299#12400116 (10BTullis) Yes, OK. I had upgraded the RAID controller firmware. {F106623353,width=50%} OK, let's replace the disk then. [11:50:53] elukey: wikikube-worker1071 has been blessed by new command line arguments [11:52:00] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [11:52:25] (03CR) 10Muehlenhoff: [C:03+2] thumbor-plugins: Rebuild against latest package versions in Trixie [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1352024 (owner: 10Muehlenhoff) [11:56:17] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [11:56:45] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [11:58:25] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12400125 (10fgiunchedi) With T439831 in place I was able to reimage 1055 and 1056 for their initial puppet runs. All disks are exposed as JBOD for these an... [12:00:04] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1000) [12:00:05] elukey: A patch you scheduled for MediaWiki infrastructure (UTC mid-day) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [12:00:05] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1200) [12:00:43] RECOVERY - OSPF status on cr2-eqiad is OK: OSPFv2: 9/9 UP : OSPFv3: 9/9 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [12:01:29] (03PS1) 10Dpogorzelski: liftwing-studio: allow unauthenticated LiteLLM metrics [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352028 (https://phabricator.wikimedia.org/T440198) [12:01:29] RECOVERY - OSPF status on cr1-magru is OK: OSPFv2: 5/5 UP : OSPFv3: 5/5 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [12:02:11] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: allow unauthenticated LiteLLM metrics [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352028 (https://phabricator.wikimedia.org/T440198) (owner: 10Dpogorzelski) [12:02:13] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] liftwing-studio: allow unauthenticated LiteLLM metrics [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352028 (https://phabricator.wikimedia.org/T440198) (owner: 10Dpogorzelski) [12:02:30] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [12:03:50] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [12:04:10] RESOLVED: BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [12:04:39] RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr1-magru and Telxius (2001:1498:1:966:1::251) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [12:04:39] RESOLVED: CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=cr1-magru:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [12:06:18] (03PS1) 10Michael Große: refactor(AccountSetup): Move post-signup onboarding checks to a shared service [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352031 [12:07:47] (03PS1) 10Santiago Faci: Deploy GrowthBook to staging based on a specific commit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352033 (https://phabricator.wikimedia.org/T439990) [12:08:20] (03PS1) 10Michael Große: fix(AccountSetup): Send onboarding exposure from the post-signup redirect [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352035 [12:08:27] (03PS2) 10Santiago Faci: Deploy GrowthBook to staging based on a specific commit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352033 (https://phabricator.wikimedia.org/T439990) [12:08:41] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 06 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352031 (owner: 10Michael Große) [12:09:14] (03CR) 10Slyngshede: [C:03+1] "LGTM." [puppet] - 10https://gerrit.wikimedia.org/r/1351438 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [12:09:14] bjensen: nice! What is your plan for the testing? I will try to match 1071's pod name with the TKO metrics in grafana, but if more pods can be updated we could take say the first top 5 TKO talkers in grafana and restart them [12:09:20] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 06 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352035 (owner: 10Michael Große) [12:09:56] elukey: that's precisely what i'm up to :) happy to update a few other pods, but i'm not seeing a dramatic change in tkos post update for this first pod [12:10:17] https://grafana.wikimedia.org/goto/s7x5r4?orgId=default [12:10:43] (i'm posting stream-of-consciousness in T420223) [12:10:43] T420223: Increase in number of memcached errors in eqiad - https://phabricator.wikimedia.org/T420223 [12:11:15] (03PS1) 10Michael Große: refactor(AccountSetup): Move post-signup onboarding checks to a shared service [extensions/GrowthExperiments] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352036 [12:11:18] (03CR) 10Ladsgroup: "I'm back now! Let me know if you want me to deploy it." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1352024 (owner: 10Muehlenhoff) [12:11:30] (03PS1) 10Michael Große: fix(AccountSetup): Send onboarding exposure from the post-signup redirect [extensions/GrowthExperiments] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352037 [12:11:40] (03CR) 10Slyngshede: [C:03+1] "Looks good, let me know how it works out. This seems potentially rather useful for other applications as well." [puppet] - 10https://gerrit.wikimedia.org/r/1352001 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [12:11:47] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 06 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352036 (owner: 10Michael Große) [12:12:02] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 06 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352037 (owner: 10Michael Große) [12:13:43] bjensen: ack, what is the pod name? [12:15:48] 1071's new pod is mcrouter-main-l6b5t, its pre-update pod was mcrouter-main-qj28t [12:15:48] (03PS1) 10Muehlenhoff: Setup LVS for ldap-ro-test/ldap-ro-test-ssl [puppet] - 10https://gerrit.wikimedia.org/r/1352041 (https://phabricator.wikimedia.org/T439730) [12:17:04] (03CR) 10Btullis: [C:03+2] apereo_cas: Add a grant_types parameter for OIDC services [puppet] - 10https://gerrit.wikimedia.org/r/1351438 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [12:17:12] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T439299#12400242 (10Jclark-ctr) Replaced Failed Drive [12:17:59] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T439299#12400245 (10Jclark-ctr) 05Open→03Resolved [12:18:17] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T440203#12400247 (10Jclark-ctr) 05Open→03Resolved a:03Jclark-ctr [12:19:12] (03CR) 10Btullis: [C:03+2] "Thanks very much. Will do." [puppet] - 10https://gerrit.wikimedia.org/r/1352001 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [12:19:14] bjensen: super thanks - may be completely random with so few data but the tko spikes seem less heavy in impact [12:19:16] elukey: 4 more nodes and their pod-pairs updated, details at https://phabricator.wikimedia.org/T420223#12400251 [12:19:28] smaller peaks, for sure! [12:20:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [12:20:24] heh [12:21:25] jouncebot: nowandnext [12:21:25] For the next 0 hour(s) and 38 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1000) [12:21:25] For the next 0 hour(s) and 38 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1200) [12:21:25] In 0 hour(s) and 38 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1300) [12:21:38] i'm going to deploy mobileapps if there's no objection [12:21:39] FIRING: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [12:22:06] (03CR) 10Isabelle Hurbain-Palatin: [C:03+2] mobile-apps: add mathjax csp rules [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347615 (https://phabricator.wikimedia.org/T431693) (owner: 10Isabelle Hurbain-Palatin) [12:22:24] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops, 13Patch-For-Review: ms-be1074: /dev/sdn issue - https://phabricator.wikimedia.org/T440281#12400264 (10CWilliams-WMF) Added a 7d silence for the Puppet alert: b2a236cc-681e-4112-b5b5-49ba4a3354e7 [12:23:17] bjensen: my reasoning is the following - there seem to be some jitter in row A/C IIRC, and we are going to pay its price with tkos no matter what (unless we raise the cross-wan timeout again). So maybe a different keep alive strategy helps reducing the impact [12:23:20] or not :D [12:23:40] (03Merged) 10jenkins-bot: mobile-apps: add mathjax csp rules [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347615 (https://phabricator.wikimedia.org/T431693) (owner: 10Isabelle Hurbain-Palatin) [12:24:13] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [12:25:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [12:25:49] elukey: do i understand correctly that the theory is that the jitter is causing the timeouts? [12:26:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [12:27:33] bjensen: the theory == my suspicion, I don't recall what was decided in the task. But IIRC I added some measurements with traceroute and those links clearly showed weird latency peaks. I don't recall what is the current wan timeout for mcrouter, but at the time it was very tight (and Effie may have increased it) [12:27:53] (03PS1) 10Jelto: deployment_server::deploy_service::service_catalog: add miscweb [puppet] - 10https://gerrit.wikimedia.org/r/1352048 (https://phabricator.wikimedia.org/T412941) [12:28:12] https://phabricator.wikimedia.org/T420223#11752904 [12:28:17] ahhhh, okay - i'm not sure what the wan timeout is here, is that a limit on how long mcrouter takes to talk to a memcached backend, or something else? [12:28:55] yeah lemme check [12:29:47] bjensen: we run with these --cross-region-timeout-ms=300 --cross-cluster-timeout-ms=100 [12:30:15] cross region is when mcrouter in eqiad has to update shards in codfw (mw-wan keys etc.. only a small subset but still) [12:30:27] gotcha, makes sense [12:30:34] (03CR) 10Scott French: [C:03+1] "Thanks, Jelto! Looks good." [puppet] - 10https://gerrit.wikimedia.org/r/1352048 (https://phabricator.wikimedia.org/T412941) (owner: 10Jelto) [12:30:58] at the network level those rows have Nokia hardware, that should be faster, but it is not clear what causes that jitter [12:31:46] again my suspicion - the jitter causes mcrouter to hit those timeouts, a shard is considered TKOed after x tries etc... [12:32:23] and we have --timeouts-until-tko=10 (so 10 timeouts for a shard before returning the tko) [12:32:31] what actually happens when we tko? we try the gutterpool? [12:32:45] correct [12:32:59] but there is some impact for the requests [12:33:04] (03PS1) 10Ladsgroup: thumbor: Bump to the most recent image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352058 [12:33:45] PROBLEM - Confd vcl based reload on cp6010 is CRITICAL: reload-vcl failed to run since 0h, 1 minutes. https://wikitech.wikimedia.org/wiki/Varnish [12:35:26] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12400316 (10fgiunchedi) With respect to 1053 and 1054 I would like to get those in their final position, which brings up to the rack power question. In {T... [12:35:44] (03CR) 10Ssingh: "@ffurnari@wikimedia.org: Please review on behalf of Traffic. Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1351947 (https://phabricator.wikimedia.org/T425441) (owner: 10Jelto) [12:38:02] !log sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db1*-bin.* -maxdepth 1 -type f -mtime +365 -delete' [12:38:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:38:19] !log ihurbain@deploy1003 helmfile [staging] START helmfile.d/services/mobileapps: apply [12:38:29] !log sudo cumin --force --no-color "A:db-all" 'find /srv/sqldata*/db2*-bin.* -maxdepth 1 -type f -mtime +365 -delete' [12:38:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:38:36] (03PS1) 10Muehlenhoff: proton: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352064 [12:38:49] marostegui cezmunsta: FYI ^ deleted a bunch more older than a year binlogs [12:39:03] that's a quadruple-check cumin command Amir1 [12:39:10] !log ihurbain@deploy1003 helmfile [staging] DONE helmfile.d/services/mobileapps: apply [12:39:16] -mtime +3 65 [12:39:22] !log ihurbain@deploy1003 helmfile [codfw] START helmfile.d/services/mobileapps: apply [12:39:35] (03CR) 10Ssingh: [C:03+1] geo-maps: Update with Meta changes for October 2026 [dns] - 10https://gerrit.wikimedia.org/r/1351928 (owner: 10Slyngshede) [12:39:38] I think the clone automation leaves them behind [12:39:57] godog: haha, yeah. I ran them a couple times before. See https://phabricator.wikimedia.org/T391344 [12:40:21] at this point, it's lawn mowing [12:40:36] !log ihurbain@deploy1003 helmfile [codfw] DONE helmfile.d/services/mobileapps: apply [12:40:45] !log ihurbain@deploy1003 helmfile [eqiad] START helmfile.d/services/mobileapps: apply [12:40:46] neat! yeah must feel good [12:41:32] !log ihurbain@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [12:42:42] (03CR) 10Jelto: [V:03+1] "PCC SUCCESS (CORE_DIFF 3): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9512/co" [puppet] - 10https://gerrit.wikimedia.org/r/1352048 (https://phabricator.wikimedia.org/T412941) (owner: 10Jelto) [12:43:25] (03CR) 10Muehlenhoff: [C:03+2] proton: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352064 (owner: 10Muehlenhoff) [12:44:03] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.7 point update - https://phabricator.wikimedia.org/T437715#12400349 (10MoritzMuehlenhoff) [12:44:29] (03CR) 10Jelto: [V:03+1 C:03+2] deployment_server::deploy_service::service_catalog: add miscweb [puppet] - 10https://gerrit.wikimedia.org/r/1352048 (https://phabricator.wikimedia.org/T412941) (owner: 10Jelto) [12:44:34] !log jmm@deploy1003 helmfile [staging] START helmfile.d/services/proton: apply [12:45:12] !log jmm@deploy1003 helmfile [staging] DONE helmfile.d/services/proton: apply [12:45:52] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 06 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352025 (https://phabricator.wikimedia.org/T440216) (owner: 10Tryvix1509) [12:46:07] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352058 (owner: 10Ladsgroup) [12:51:00] elukey: if i'm reading these graphs right, it looks like we occasionally have cross-dc requests to the gutterpool, and those seem to be the slowest. they also are too slow to ever return within 300ms. do we know what the impact would be of loosening the cross-region timeout? [12:51:08] (03PS1) 10Dpogorzelski: logstash: map Hummingbird logs to ECS [puppet] - 10https://gerrit.wikimedia.org/r/1352072 (https://phabricator.wikimedia.org/T440198) [12:51:13] (03PS1) 10Dpogorzelski: liftwing-studio: log JSON for ECS [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352073 (https://phabricator.wikimedia.org/T440198) [12:51:55] (03CR) 10Ladsgroup: [C:03+2] thumbor: Bump to the most recent image (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352058 (owner: 10Ladsgroup) [12:53:31] (03Merged) 10jenkins-bot: thumbor: Bump to the most recent image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352058 (owner: 10Ladsgroup) [12:54:09] (03PS1) 10Isabelle Hurbain-Palatin: mobile-apps: add more mathjax csp rules [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352074 (https://phabricator.wikimedia.org/T431693) [12:56:40] bjensen: in theory we may impact overall mw request serve latency, because php calls mcrouter that in turns can take more to return a result (this is my speculation) [12:57:05] but it is true that we should discuss real jitter vs upper layer expectations [12:57:15] the two things are, imho, not aligned right now [12:57:40] yeah, i'd like to understand this end-to-end a little better [12:58:03] !log jmm@deploy1003 helmfile [codfw] START helmfile.d/services/proton: apply [12:59:59] !log jmm@deploy1003 helmfile [codfw] DONE helmfile.d/services/proton: apply [13:00:04] Lucas_WMDE, urbanecm, and TheresNoTime: #bothumor My software never has bugs. It just develops random features. Rise for UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1300). [13:00:04] MichaelG_WMF and Hide_on_rosie: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:16] I’m in a meeting, might be able to deploy later [13:00:28] * MichaelG_WMF is here [13:01:03] O/ [13:01:19] hey folks! [13:01:48] I posted on #engineering-all on slack, I am holding the scap lock atm for a maintenance task for the Docker Registry that is taking more than anticipated [13:02:18] so if it is possible to delay the backport window I'd be grateful, really sorry for the impact [13:02:25] lemme know if you have anything really urgent to deploy [13:02:28] !log jmm@deploy1003 helmfile [eqiad] START helmfile.d/services/proton: apply [13:03:39] elukey: urgent in the sense that it should happen in this window, but it is fine if it is later in the window [13:03:43] !log jmm@deploy1003 helmfile [eqiad] DONE helmfile.d/services/proton: apply [13:04:13] RESOLVED: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:04:33] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [13:04:44] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [13:06:06] (03CR) 10Andrew Bogott: [C:03+2] wmcs-backup: Cope when a volume is deleted mid-flight [puppet] - 10https://gerrit.wikimedia.org/r/1345351 (owner: 10Andrew Bogott) [13:09:59] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops, 13Patch-For-Review: ms-be1074: /dev/sdn issue - https://phabricator.wikimedia.org/T440281#12400463 (10CWilliams-WMF) These "Other Error Count" values are increasing: `sh cwilliams@ms-be1074 ~ % ls -la pd_*txt -rw-r--r-- 1 cwilliams wikidev 29981 Oct 6 10... [13:11:11] jouncebot: nowandnext [13:11:11] For the next 0 hour(s) and 48 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1300) [13:11:12] In 0 hour(s) and 48 minute(s): Test Kitchen UI Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1400) [13:11:23] MichaelG_WMF: ack I'll keep you posted [13:11:57] !log ladsgroup@deploy1003 helmfile [codfw] START helmfile.d/services/thumbor: apply [13:12:48] Need a deployer for wmf-config deploy. [13:13:02] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad: WMCS switch refresh 2026 (C8 & D5) - https://phabricator.wikimedia.org/T440305 (10cmooney) 03NEW p:05Triage→03Medium [13:13:09] I can do it but after Luca is done with the work [13:14:03] !log installing glibc security updates [13:14:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:14:08] !log ladsgroup@deploy1003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [13:16:29] (03CR) 10Daniel Kertesz: [C:03+1] "syntax wise is OK, but I don't know if there's anything else that need to be checked for this kind of changes" [puppet] - 10https://gerrit.wikimedia.org/r/1352041 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [13:18:36] !log ladsgroup@deploy1003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [13:19:13] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [13:20:43] !log ladsgroup@deploy1003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [13:21:28] (03CR) 10Federico Ceratto: mysql: Fix fleetwide RO/RW setting (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1351376 (https://phabricator.wikimedia.org/T438831) (owner: 10Federico Ceratto) [13:21:39] RESOLVED: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:24:08] (03PS1) 10JavierMonton: stream: mw-page-html-content-change-enrich [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352091 (https://phabricator.wikimedia.org/T424987) [13:25:17] !log installing libssh2 security updates [13:25:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:27:07] (03PS3) 10Slyngshede: debug.json: order codfw (primary) DC backends first [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319818 (https://phabricator.wikimedia.org/T439674) [13:28:40] (03PS2) 10Slyngshede: geo-maps: update map default to list codfw first [dns] - 10https://gerrit.wikimedia.org/r/1319816 (https://phabricator.wikimedia.org/T439674) [13:29:15] (03CR) 10Slyngshede: [C:03+2] geo-maps: Update with Meta changes for October 2026 [dns] - 10https://gerrit.wikimedia.org/r/1351928 (owner: 10Slyngshede) [13:29:23] !log slyngshede@dns1004 START - running authdns-update [13:30:09] (03PS4) 10Giuseppe Lavagetto: [DNM] k8s cookbook to enable virtualization [cookbooks] - 10https://gerrit.wikimedia.org/r/1351902 (https://phabricator.wikimedia.org/T438287) [13:31:54] !log slyngshede@dns1004 END - running authdns-update [13:32:22] !log elukey@deploy1003 Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297 (duration: 214m 10s) [13:32:25] T437297: Test the Docker Registry's GC with Releng/MediaWiki images - https://phabricator.wikimedia.org/T437297 [13:32:31] (03CR) 10CI reject: [V:04-1] [DNM] k8s cookbook to enable virtualization [cookbooks] - 10https://gerrit.wikimedia.org/r/1351902 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [13:32:41] Amir1, MichaelG_WMF green light! [13:33:12] <3 [13:33:35] (03PS5) 10Giuseppe Lavagetto: [DNM] k8s cookbook to enable virtualization [cookbooks] - 10https://gerrit.wikimedia.org/r/1351902 (https://phabricator.wikimedia.org/T438287) [13:33:36] Amir1: could you deploy my backports as well? 🥺 [13:34:22] sure thing! [13:34:25] !log oblivian@cumin1004 START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2011-2018].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [13:34:27] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2011-2018].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [13:34:36] 🙏 [13:34:42] !log installing php8.2 security updates [13:34:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:35:14] jouncebot: now [13:35:14] For the next 0 hour(s) and 24 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1300) [13:35:36] 10ops-eqiad, 06SRE, 06DC-Ops: eqiad: WMCS switch refresh 2026 - optics order - https://phabricator.wikimedia.org/T440312 (10cmooney) 03NEW p:05Triage→03Medium [13:35:37] !log oblivian@cumin1004 START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2033-2039].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [13:35:38] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2033-2039].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [13:35:42] MichaelG_WMF: do you want me to merge and deploy all together or separately? [13:35:56] 10ops-eqiad, 06SRE, 06DC-Ops: eqiad: WMCS switch refresh 2026 - optics order - https://phabricator.wikimedia.org/T440312#12400723 (10cmooney) [13:35:57] Amir1: all together please [13:35:57] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad: WMCS switch refresh 2026 (C8 & D5) - https://phabricator.wikimedia.org/T440305#12400724 (10cmooney) [13:36:02] (03CR) 10CI reject: [V:04-1] [DNM] k8s cookbook to enable virtualization [cookbooks] - 10https://gerrit.wikimedia.org/r/1351902 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [13:36:10] (03CR) 10Ladsgroup: [C:03+2] refactor(AccountSetup): Move post-signup onboarding checks to a shared service [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352031 (owner: 10Michael Große) [13:36:14] (03CR) 10Ladsgroup: [C:03+2] fix(AccountSetup): Send onboarding exposure from the post-signup redirect [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352035 (owner: 10Michael Große) [13:36:22] (03CR) 10Ladsgroup: [C:03+2] refactor(AccountSetup): Move post-signup onboarding checks to a shared service [extensions/GrowthExperiments] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352036 (owner: 10Michael Große) [13:36:23] awesome [13:36:26] (03CR) 10Ladsgroup: [C:03+2] fix(AccountSetup): Send onboarding exposure from the post-signup redirect [extensions/GrowthExperiments] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352037 (owner: 10Michael Große) [13:36:43] (03PS1) 10Daniel Kinzler: REST: enable /api on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352093 (https://phabricator.wikimedia.org/T433547) [13:36:48] +2'ed, it'll a bit for all to merge, in the mean time I deploy the other one (cc Hide_on_rosie ) [13:36:58] unfortunately, all that I can do in terms of testing is to create a new account and ensure that nothing obvious is breaking. The real effect will only be visible in the experiment data coming in over time [13:37:27] !log oblivian@cumin1004 START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [13:37:31] Amir1: OK [13:37:31] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2041.codfw.wmnet [13:37:45] (03CR) 10CI reject: [V:04-1] REST: enable /api on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352093 (https://phabricator.wikimedia.org/T433547) (owner: 10Daniel Kinzler) [13:37:52] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352025 (https://phabricator.wikimedia.org/T440216) (owner: 10Tryvix1509) [13:37:55] (03Merged) 10jenkins-bot: refactor(AccountSetup): Move post-signup onboarding checks to a shared service [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352031 (owner: 10Michael Große) [13:38:05] (03Merged) 10jenkins-bot: fix(AccountSetup): Send onboarding exposure from the post-signup redirect [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352035 (owner: 10Michael Große) [13:38:05] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2041.codfw.wmnet [13:38:06] (03Merged) 10jenkins-bot: refactor(AccountSetup): Move post-signup onboarding checks to a shared service [extensions/GrowthExperiments] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352036 (owner: 10Michael Große) [13:38:15] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Test the Docker Registry's GC with Releng/MediaWiki images - https://phabricator.wikimedia.org/T437297#12400733 (10elukey) After ~130 mins the mark and sweep completed, and ~400GB of data has been d... [13:38:32] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad: WMCS switch refresh 2026 - rack and cable new switches - https://phabricator.wikimedia.org/T440313 (10cmooney) 03NEW p:05Triage→03Medium [13:38:40] (03Merged) 10jenkins-bot: fix(AccountSetup): Send onboarding exposure from the post-signup redirect [extensions/GrowthExperiments] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352037 (owner: 10Michael Große) [13:38:48] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad: WMCS switch refresh 2026 - rack new switches - https://phabricator.wikimedia.org/T440313#12400750 (10cmooney) [13:38:53] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2041.codfw.wmnet [13:38:55] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2041.codfw.wmnet [13:38:55] !log oblivian@cumin1004 END (ERROR) - Cookbook sre.k8s.enable_virtualization (exit_code=97) rolling provision on D{wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [13:39:07] (03Merged) 10jenkins-bot: ltwiki: Enable extended confirmed user group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352025 (https://phabricator.wikimedia.org/T440216) (owner: 10Tryvix1509) [13:40:15] (03PS1) 10Daniel Kinzler: REST: enable the content/v2-beta module on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352094 (https://phabricator.wikimedia.org/T433534) [13:41:49] (03PS5) 10Elukey: profile::pki::multirooca: simplify vhost config [puppet] - 10https://gerrit.wikimedia.org/r/1346615 (https://phabricator.wikimedia.org/T436809) [13:41:50] (03CR) 10AKhatun: [C:03+1] stream: mw-page-html-content-change-enrich [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352091 (https://phabricator.wikimedia.org/T424987) (owner: 10JavierMonton) [13:42:03] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1352025|ltwiki: Enable extended confirmed user group (T440216)]] [13:42:07] T440216: Enable extended confirmed user group on Lithuanian Wikipedia - https://phabricator.wikimedia.org/T440216 [13:44:37] (03PS6) 10Elukey: profile::pki::multirooca: simplify vhost config [puppet] - 10https://gerrit.wikimedia.org/r/1346615 (https://phabricator.wikimedia.org/T436809) [13:44:43] (03CR) 10JavierMonton: [C:03+2] stream: mw-page-html-content-change-enrich [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352091 (https://phabricator.wikimedia.org/T424987) (owner: 10JavierMonton) [13:46:12] !log ladsgroup@deploy1003 tryvix1509, ladsgroup: Backport for [[gerrit:1352025|ltwiki: Enable extended confirmed user group (T440216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:46:21] (03Merged) 10jenkins-bot: stream: mw-page-html-content-change-enrich [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352091 (https://phabricator.wikimedia.org/T424987) (owner: 10JavierMonton) [13:47:10] Hide_on_rosie: it's on mw-debug [13:47:15] (03CR) 10Elukey: [C:03+2] profile::pki::multirooca: simplify vhost config [puppet] - 10https://gerrit.wikimedia.org/r/1346615 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [13:47:21] Testing. [13:47:52] (03CR) 10CWilliams: mysql: Fix fleetwide RO/RW setting (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1351376 (https://phabricator.wikimedia.org/T438831) (owner: 10Federico Ceratto) [13:48:31] (03PS1) 10JMeybohm: kube-state-metrics: Make node labels available [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352098 (https://phabricator.wikimedia.org/T435796) [13:48:45] (03PS4) 10Federico Ceratto: team-data-persistence: Alarm on stale data in Zarcillo [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) [13:49:15] (03CR) 10Federico Ceratto: team-data-persistence: Alarm on stale data in Zarcillo (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) (owner: 10Federico Ceratto) [13:50:10] Amir1: Check OK. [13:50:30] !log ladsgroup@deploy1003 tryvix1509, ladsgroup: Continuing with deployment [13:50:35] moving forward [13:51:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:52:53] (03PS6) 10Giuseppe Lavagetto: [DNM] k8s cookbook to enable virtualization [cookbooks] - 10https://gerrit.wikimedia.org/r/1351902 (https://phabricator.wikimedia.org/T438287) [13:53:49] !log oblivian@cumin1004 START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [13:53:53] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2042.codfw.wmnet [13:54:01] * MichaelG_WMF makes a quick trip to the coffee machine, will be back momentarily [13:54:27] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2042.codfw.wmnet [13:54:38] (03CR) 10Phuedx: [C:03+1] Deploy GrowthBook to staging based on a specific commit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352033 (https://phabricator.wikimedia.org/T439990) (owner: 10Santiago Faci) [13:54:47] (03CR) 10Blake: [C:03+2] kube-state-metrics: Make node labels available [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352098 (https://phabricator.wikimedia.org/T435796) (owner: 10JMeybohm) [13:54:47] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:54:52] (03CR) 10Blake: [C:03+1] kube-state-metrics: Make node labels available [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352098 (https://phabricator.wikimedia.org/T435796) (owner: 10JMeybohm) [13:55:16] (03CR) 10CI reject: [V:04-1] [DNM] k8s cookbook to enable virtualization [cookbooks] - 10https://gerrit.wikimedia.org/r/1351902 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [13:55:23] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply [13:55:31] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply [13:55:40] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [13:55:45] FIRING: WidespreadPuppetFailure: Puppet has failed in ulsfo - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [13:55:51] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1352025|ltwiki: Enable extended confirmed user group (T440216)]] (duration: 13m 47s) [13:55:54] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-feature-counts-change-enrich: apply [13:55:54] T440216: Enable extended confirmed user group on Lithuanian Wikipedia - https://phabricator.wikimedia.org/T440216 [13:56:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:56:53] (03PS1) 10Elukey: Revert "profile::pki::multirooca: simplify vhost config" [puppet] - 10https://gerrit.wikimedia.org/r/1352102 [13:57:12] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops, 13Patch-For-Review: ms-be1074: /dev/sdn issue - https://phabricator.wikimedia.org/T440281#12400884 (10Jclark-ctr) a:03Jclark-ctr This server is out of warranty @cwilliams-wmf would you like me to replace failed drive with one from decommissioned serve... [13:57:26] * MichaelG_WMF is back [13:57:58] (03CR) 10Santiago Faci: [C:03+2] Deploy GrowthBook to staging based on a specific commit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352033 (https://phabricator.wikimedia.org/T439990) (owner: 10Santiago Faci) [13:59:43] (03CR) 10JMeybohm: [C:03+2] kube-state-metrics: Make node labels available [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352098 (https://phabricator.wikimedia.org/T435796) (owner: 10JMeybohm) [14:00:05] Deploy window Test Kitchen UI Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1400) [14:00:25] (03Merged) 10jenkins-bot: kube-state-metrics: Make node labels available [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352098 (https://phabricator.wikimedia.org/T435796) (owner: 10JMeybohm) [14:00:49] okay, gonna push MichaelG_WMF's patches now [14:01:42] FIRING: JobUnavailable: Reduced availability for job cfssl in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:01:56] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1352037|fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036|refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035|fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031|refactor(AccountSetup): Move post-signup onboarding check [14:01:56] s to a shared service]] [14:02:04] Amir1: I'm ready to test whenever you tell me 👍 [14:02:13] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2042.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:02:25] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2042.codfw.wmnet [14:03:19] !log ladsgroup@deploy1003 migr, ladsgroup: Backport for [[gerrit:1352037|fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036|refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035|fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031|refactor(AccountSetup): Move post-signup onboarding checks to a s [14:03:19] hared service]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:03:19] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2042.codfw.wmnet [14:03:33] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2044.codfw.wmnet [14:03:37] (03Merged) 10jenkins-bot: Deploy GrowthBook to staging based on a specific commit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352033 (https://phabricator.wikimedia.org/T439990) (owner: 10Santiago Faci) [14:04:07] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2044.codfw.wmnet [14:04:09] MichaelG_WMF: it's on testservers now [14:04:13] FIRING: KubernetesCalicoDown: wikikube-worker2042.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2042.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:04:19] testing... [14:04:27] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:05:45] FIRING: [3x] WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [14:06:31] !log cdobbins@cumin1004 START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-eqsin [14:06:42] FIRING: [2x] JobUnavailable: Reduced availability for job cfssl in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:06:43] Amir1: I successfully created an account without any noticable error. That's the best testing I can do. If the logs are clean, then I think we're ready to roll foward. [14:06:51] cool [14:06:56] !log ladsgroup@deploy1003 migr, ladsgroup: Continuing with deployment [14:08:09] (03PS1) 10Elukey: profile::multirootca: fix vhost on port 443 [puppet] - 10https://gerrit.wikimedia.org/r/1352109 [14:08:43] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1352037|fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352036|refactor(AccountSetup): Move post-signup onboarding checks to a shared service]], [[gerrit:1352035|fix(AccountSetup): Send onboarding exposure from the post-signup redirect]], [[gerrit:1352031|refactor(AccountSetup): Move post-signup onboarding chec [14:08:43] ks to a shared service]] (duration: 06m 47s) [14:09:13] RESOLVED: KubernetesCalicoDown: wikikube-worker2042.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2042.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:09:18] (03CR) 10Muehlenhoff: [C:03+2] Setup LVS for ldap-ro-test/ldap-ro-test-ssl [puppet] - 10https://gerrit.wikimedia.org/r/1352041 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [14:10:43] PROBLEM - Confd vcl based reload on cp6012 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [14:11:05] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1352109 (owner: 10Elukey) [14:11:50] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2044.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:12:19] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2044.codfw.wmnet [14:12:21] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2044.codfw.wmnet [14:12:35] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2046.codfw.wmnet [14:13:02] (03CR) 10Joal: [C:03+1] "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/1346752 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [14:13:02] (03CR) 10Bking: [C:03+1] liftwing-studio: log JSON for ECS [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352073 (https://phabricator.wikimedia.org/T440198) (owner: 10Dpogorzelski) [14:13:09] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2046.codfw.wmnet [14:13:09] (03CR) 10Xcollazo: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346752 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [14:13:13] (03PS2) 10Elukey: profile::multirootca: fix vhost on port 443 [puppet] - 10https://gerrit.wikimedia.org/r/1352109 [14:13:29] (03CR) 10Elukey: "Updated the syntax but it is the same :)" [puppet] - 10https://gerrit.wikimedia.org/r/1352109 (owner: 10Elukey) [14:13:29] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:13:43] RECOVERY - Confd vcl based reload on cp6012 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [14:15:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [14:17:01] (03CR) 10Elukey: [C:03+2] profile::multirootca: fix vhost on port 443 [puppet] - 10https://gerrit.wikimedia.org/r/1352109 (owner: 10Elukey) [14:19:43] PROBLEM - PyBal IPVS diff check on lvs2014 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [14:19:46] (03PS3) 10Ladsgroup: Thumbor: Switch from CONVERT_PATH to MAGICK_PATH [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342065 (https://phabricator.wikimedia.org/T438089) (owner: 10Samwilson) [14:19:47] PROBLEM - PyBal connections to etcd on lvs2012 is CRITICAL: CRITICAL: 6 connections established with conf2004.codfw.wmnet:4001 (min=8) https://wikitech.wikimedia.org/wiki/PyBal [14:19:47] PROBLEM - PyBal IPVS diff check on lvs2012 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [14:20:04] (03Abandoned) 10Ladsgroup: thumbor: Bump to the most recent version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345261 (https://phabricator.wikimedia.org/T439297) (owner: 10Ladsgroup) [14:20:17] PROBLEM - PyBal connections to etcd on lvs2014 is CRITICAL: CRITICAL: 98 connections established with conf2004.codfw.wmnet:4001 (min=100) https://wikitech.wikimedia.org/wiki/PyBal [14:20:21] (03CR) 10Btullis: [C:03+1] liftwing-studio: log JSON for ECS [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352073 (https://phabricator.wikimedia.org/T440198) (owner: 10Dpogorzelski) [14:20:33] ^ expected, silencing them [14:20:52] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops, 13Patch-For-Review: ms-be1074: /dev/sdn issue - https://phabricator.wikimedia.org/T440281#12401135 (10CWilliams-WMF) @Jclark-ctr thanks for the update. Matthew is off until next week and so to be honest, I am not entirely sure. Do you know when you will b... [14:20:54] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2046.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:21:03] jouncebot: nowandnext [14:21:03] For the next 0 hour(s) and 8 minute(s): Test Kitchen UI Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1400) [14:21:03] In 0 hour(s) and 8 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1430) [14:21:13] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2046.codfw.wmnet [14:21:16] ACKNOWLEDGEMENT - PyBal IPVS diff check on lvs2012 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal Muehlenhoff new service https://wikitech.wikimedia.org/wiki/PyBal [14:21:16] ACKNOWLEDGEMENT - PyBal connections to etcd on lvs2012 is CRITICAL: CRITICAL: 6 connections established with conf2004.codfw.wmnet:4001 (min=8) Muehlenhoff new service https://wikitech.wikimedia.org/wiki/PyBal [14:21:16] ACKNOWLEDGEMENT - PyBal IPVS diff check on lvs2014 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal Muehlenhoff new service https://wikitech.wikimedia.org/wiki/PyBal [14:21:41] ACKNOWLEDGEMENT - PyBal IPVS diff check on lvs2012 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal Muehlenhoff new service https://wikitech.wikimedia.org/wiki/PyBal [14:21:42] ACKNOWLEDGEMENT - PyBal connections to etcd on lvs2012 is CRITICAL: CRITICAL: 6 connections established with conf2004.codfw.wmnet:4001 (min=8) Muehlenhoff new service https://wikitech.wikimedia.org/wiki/PyBal [14:21:42] ACKNOWLEDGEMENT - PyBal IPVS diff check on lvs2014 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal Muehlenhoff new service https://wikitech.wikimedia.org/wiki/PyBal [14:21:42] ACKNOWLEDGEMENT - PyBal connections to etcd on lvs2014 is CRITICAL: CRITICAL: 98 connections established with conf2004.codfw.wmnet:4001 (min=100) Muehlenhoff new service https://wikitech.wikimedia.org/wiki/PyBal [14:21:49] !log jelto@deploy1003 Started scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig [14:21:49] !log jelto@deploy1003 scap deploy-service miscweb: [STAGING/DEFAULT] staging-eqiad Deploying miscweb [14:22:07] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2046.codfw.wmnet [14:22:18] ACKNOWLEDGEMENT - PyBal connections to etcd on lvs2014 is CRITICAL: CRITICAL: 98 connections established with conf2004.codfw.wmnet:4001 (min=100) Muehlenhoff new service https://wikitech.wikimedia.org/wiki/PyBal [14:22:21] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2049.codfw.wmnet [14:22:40] !log jelto@deploy1003 scap deploy-service miscweb: [PRODUCTION/DEFAULT] codfw Deploying miscweb [14:23:20] !log jelto@deploy1003 scap deploy-service miscweb: [PRODUCTION/DEFAULT] eqiad Deploying miscweb [14:23:24] !log restarting pybal on lvs2014 T439730 [14:23:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:23:27] T439730: Add a new LVS ldap-ro-test service - https://phabricator.wikimedia.org/T439730 [14:23:29] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2049.codfw.wmnet [14:23:43] !log jelto@deploy1003 Finished scap deploy-service miscweb: First test deployment of miscweb using deploy-service and spiderpig (duration: 01m 54s) [14:23:49] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:24:44] RECOVERY - PyBal IPVS diff check on lvs2014 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [14:25:08] (03PS2) 10Herron: Test new rsyslog image on mw-experimental and mw-web canary [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346712 (https://phabricator.wikimedia.org/T438375) [14:25:18] RECOVERY - PyBal connections to etcd on lvs2014 is OK: OK: 100 connections established with conf2004.codfw.wmnet:4001 (min=100) https://wikitech.wikimedia.org/wiki/PyBal [14:25:21] (03PS3) 10Herron: Test new rsyslog image on mw-web canary [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346712 (https://phabricator.wikimedia.org/T438375) [14:25:29] (03PS1) 10Herron: Test new rsyslog image on mw-experimental [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352121 (https://phabricator.wikimedia.org/T438375) [14:25:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [14:25:54] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325997 (https://phabricator.wikimedia.org/T432544) (owner: 10Mamouri) [14:26:55] (03Merged) 10jenkins-bot: Enable $wgCiteDefaultRefDirAuto on fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325997 (https://phabricator.wikimedia.org/T432544) (owner: 10Mamouri) [14:27:00] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [14:27:15] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1325997|Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] [14:27:18] T432544: Cite: reference-list items on RTL wikis should take each footnote's direction from its own content (mixed LTR/RTL bibliographies render broken) - https://phabricator.wikimedia.org/T432544 [14:27:29] (03PS2) 10Jforrester: abstractwiki-rust: Bump version number; trixie-backports has upgraded rustc etc. from 1.94.1 to 1.95.0 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1348701 [14:27:58] (03CR) 10Herron: "went ahead and split this into two changes, please see the related patch" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346712 (https://phabricator.wikimedia.org/T438375) (owner: 10Herron) [14:28:04] jouncebot: nowandnext [14:28:05] For the next 0 hour(s) and 1 minute(s): Test Kitchen UI Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1400) [14:28:05] In 0 hour(s) and 1 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1430) [14:28:29] (03CR) 10Jforrester: "I can't self-merge in this repo as I'm not a root, so merge appreciated. (Build not strictly needed, it'll happen automatically on Sunday " [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1348701 (owner: 10Jforrester) [14:28:38] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [14:28:42] (03CR) 10Herron: "went ahead and split this into two changes" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346712 (https://phabricator.wikimedia.org/T438375) (owner: 10Herron) [14:29:55] (03CR) 10Btullis: Add Matomo frontend helm chart (033 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345213 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [14:30:04] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1430) [14:30:10] (03CR) 10Btullis: [C:03+1] Add Matomo frontend helm chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345213 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [14:30:45] RESOLVED: [4x] WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [14:31:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:31:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2049.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:31:28] !log ladsgroup@deploy1003 ladsgroup, mamouri: Backport for [[gerrit:1325997|Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:31:33] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2049.codfw.wmnet [14:31:35] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2049.codfw.wmnet [14:31:41] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-eqsin [14:31:49] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2050.codfw.wmnet [14:32:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2050.codfw.wmnet [14:32:43] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:32:46] (03CR) 10Btullis: [C:03+1] "Looks good from my perspective. Maybe ask someone else from o11y, as Cole looks to be out at the moment." [puppet] - 10https://gerrit.wikimedia.org/r/1352072 (https://phabricator.wikimedia.org/T440198) (owner: 10Dpogorzelski) [14:33:08] !log restarting pybal on lvs2012 T439730 [14:33:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:33:12] T439730: Add a new LVS ldap-ro-test service - https://phabricator.wikimedia.org/T439730 [14:33:35] !log ladsgroup@deploy1003 ladsgroup, mamouri: Continuing with deployment [14:34:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:34:46] RECOVERY - PyBal connections to etcd on lvs2012 is OK: OK: 8 connections established with conf2004.codfw.wmnet:4001 (min=8) https://wikitech.wikimedia.org/wiki/PyBal [14:34:46] RECOVERY - PyBal IPVS diff check on lvs2012 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [14:36:02] (03PS1) 10Arnaudb: gerrit: reindex the replicas offline every night [puppet] - 10https://gerrit.wikimedia.org/r/1352124 (https://phabricator.wikimedia.org/T440177) [14:36:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:36:40] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops, 13Patch-For-Review: ms-be1074: /dev/sdn issue - https://phabricator.wikimedia.org/T440281#12401321 (10Jclark-ctr) This server is out of warranty @cwilliams-wmf would you like me to replace failed drive with one from decommissioned server? can this be s... [14:38:12] (03PS2) 10CDanis: kube-reload-certs: allow different labels then "release" [puppet] - 10https://gerrit.wikimedia.org/r/1351965 (https://phabricator.wikimedia.org/T439409) (owner: 10Elukey) [14:38:31] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2050.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:38:50] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2050.codfw.wmnet [14:38:53] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1325997|Enable $wgCiteDefaultRefDirAuto on fawiki (T432544)]] (duration: 11m 37s) [14:38:56] T432544: Cite: reference-list items on RTL wikis should take each footnote's direction from its own content (mixed LTR/RTL bibliographies render broken) - https://phabricator.wikimedia.org/T432544 [14:39:24] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:39:44] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2050.codfw.wmnet [14:39:46] PROBLEM - PyBal connections to etcd on lvs1018 is CRITICAL: CRITICAL: 24 connections established with conf1007.eqiad.wmnet:4001 (min=26) https://wikitech.wikimedia.org/wiki/PyBal [14:39:46] PROBLEM - PyBal connections to etcd on lvs1020 is CRITICAL: CRITICAL: 117 connections established with conf1007.eqiad.wmnet:4001 (min=119) https://wikitech.wikimedia.org/wiki/PyBal [14:39:46] PROBLEM - PyBal IPVS diff check on lvs1018 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [14:39:58] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2051.codfw.wmnet [14:40:02] (03CR) 10Bking: Add Matomo frontend helm chart (033 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345213 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [14:40:28] (03CR) 10CI reject: [V:04-1] kube-reload-certs: allow different labels then "release" [puppet] - 10https://gerrit.wikimedia.org/r/1351965 (https://phabricator.wikimedia.org/T439409) (owner: 10Elukey) [14:40:36] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2051.codfw.wmnet [14:40:48] ACKNOWLEDGEMENT - PyBal IPVS diff check on lvs1018 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal Muehlenhoff in setup https://wikitech.wikimedia.org/wiki/PyBal [14:40:48] ACKNOWLEDGEMENT - PyBal connections to etcd on lvs1018 is CRITICAL: CRITICAL: 24 connections established with conf1007.eqiad.wmnet:4001 (min=26) Muehlenhoff in setup https://wikitech.wikimedia.org/wiki/PyBal [14:40:49] ACKNOWLEDGEMENT - PyBal connections to etcd on lvs1020 is CRITICAL: CRITICAL: 117 connections established with conf1007.eqiad.wmnet:4001 (min=119) Muehlenhoff in setup https://wikitech.wikimedia.org/wiki/PyBal [14:40:56] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:41:05] (03CR) 10CDanis: "I think we could also fix this with some simple additions to jaeger's values.yaml, let me try it out." [puppet] - 10https://gerrit.wikimedia.org/r/1351965 (https://phabricator.wikimedia.org/T439409) (owner: 10Elukey) [14:41:15] !log cdobbins@cumin1004 START - Cookbook sre.cdn.roll-restart-ats rolling restart_daemons on A:cp-codfw [14:41:28] (03PS26) 10Bking: Add Matomo frontend helm chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345213 (https://phabricator.wikimedia.org/T436003) [14:41:54] PROBLEM - PyBal IPVS diff check on lvs1020 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [14:42:18] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops, 13Patch-For-Review: ms-be1074: /dev/sdn issue - https://phabricator.wikimedia.org/T440281#12401342 (10Jclark-ctr) If we didn’t have the correct model, we would have to order the drive. I’m 99% sure we have some available, but I want to verify in person to... [14:42:40] ACKNOWLEDGEMENT - PyBal IPVS diff check on lvs1020 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal Muehlenhoff in setup https://wikitech.wikimedia.org/wiki/PyBal [14:44:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:44:57] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T440195#12401383 (10Jclark-ctr) a:03Jclark-ctr ps1-c6-eqiad.mgmt.eqiad.wmnet #1: Sensor: Cord, Master_Cord_A, Active Power Value: 4.31 kW (power) Thresholds: High: 4300... [14:44:58] (03PS27) 10Bking: Add Matomo frontend helm chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345213 (https://phabricator.wikimedia.org/T436003) [14:45:05] (03CR) 10Ladsgroup: [C:03+2] Thumbor: Switch from CONVERT_PATH to MAGICK_PATH [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342065 (https://phabricator.wikimedia.org/T438089) (owner: 10Samwilson) [14:45:19] @mutante @dancy @rzl I would like to deploy a mediawiki config change later today, before the official backport window: . What would be a good time? Or should I leave it until tomorrow? It's not urgent, I just want it off my plate, and the backport windows are at inconvenient times. [14:45:50] (03PS28) 10Bking: Add Matomo frontend helm chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345213 (https://phabricator.wikimedia.org/T436003) [14:46:35] (03Merged) 10jenkins-bot: Thumbor: Switch from CONVERT_PATH to MAGICK_PATH [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342065 (https://phabricator.wikimedia.org/T438089) (owner: 10Samwilson) [14:46:52] duesen: That's probably fine. The train has already rolled to group0 so the train window is open. [14:47:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2051.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:47:21] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2051.codfw.wmnet [14:47:36] duesen: Clarifying: Using the time before the backport window (which is the train window) is fine. [14:47:45] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops, 13Patch-For-Review: ms-be1074: /dev/sdn issue - https://phabricator.wikimedia.org/T440281#12401435 (10CWilliams-WMF) OK, I think that this is probably the safest plan: - I will get the Puppet patch merged and hope that Puppet can complete to remove the fa... [14:48:09] !log restarting pybal on lvs1020 T439730 [14:48:09] dancy: thanks! That's a bit later than I hoped, but it's good to know that it's an option. [14:48:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:48:12] T439730: Add a new LVS ldap-ro-test service - https://phabricator.wikimedia.org/T439730 [14:48:15] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2051.codfw.wmnet [14:48:16] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [14:48:20] (03CR) 10Bking: [C:03+2] Add Matomo frontend helm chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345213 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [14:48:33] duesen: Now is fine too if you're trying to get it out of the way [14:49:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:49:24] (03PS1) 10CWilliams: swift: mark sdn1 in ms-be1074 as failed [puppet] - 10https://gerrit.wikimedia.org/r/1351957 (https://phabricator.wikimedia.org/T440281) [14:49:46] RECOVERY - PyBal connections to etcd on lvs1020 is OK: OK: 119 connections established with conf1007.eqiad.wmnet:4001 (min=119) https://wikitech.wikimedia.org/wiki/PyBal [14:49:55] (03Merged) 10jenkins-bot: Add Matomo frontend helm chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345213 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [14:49:57] (03CR) 10Ladsgroup: [C:03+1] swift: mark sdn1 in ms-be1074 as failed [puppet] - 10https://gerrit.wikimedia.org/r/1351957 (https://phabricator.wikimedia.org/T440281) (owner: 10CWilliams) [14:50:15] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T439559#12401467 (10Jclark-ctr) 05Open→03Resolved a:03Jclark-ctr [14:50:18] (03CR) 10Audrey Penven: [C:03+1] Enable mobile editing for all Wikidata editors (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347373 (https://phabricator.wikimedia.org/T437054) (owner: 10Sadiya.mohammed13) [14:51:28] (03PS1) 10CDanis: WIP [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352136 [14:51:52] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet [14:51:54] RECOVERY - PyBal IPVS diff check on lvs1020 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [14:51:57] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2041-2042,2044,2046,2049-2051].codfw.wmnet [14:54:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:54:41] (03PS2) 10CDanis: jaeger: match wmf release label [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352136 (https://phabricator.wikimedia.org/T439246) [14:55:04] (03PS1) 10Hnowlan: team-netops: create alert for combined impact of BGP impact and probe failure [alerts] - 10https://gerrit.wikimedia.org/r/1352138 (https://phabricator.wikimedia.org/T417051) [14:56:10] (03PS1) 10Bking: wikimedia.org: Add record for matomo-next [dns] - 10https://gerrit.wikimedia.org/r/1352139 (https://phabricator.wikimedia.org/T436003) [14:57:03] !log restarting pybal on lvs1018 T439730 [14:57:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:57:07] T439730: Add a new LVS ldap-ro-test service - https://phabricator.wikimedia.org/T439730 [14:57:19] (03CR) 10CI reject: [V:04-1] team-netops: create alert for combined impact of BGP impact and probe failure [alerts] - 10https://gerrit.wikimedia.org/r/1352138 (https://phabricator.wikimedia.org/T417051) (owner: 10Hnowlan) [14:58:41] (03PS2) 10Hnowlan: team-netops: create alert for combined impact of BGP impact and probe failure [alerts] - 10https://gerrit.wikimedia.org/r/1352138 (https://phabricator.wikimedia.org/T417051) [14:59:00] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [14:59:12] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [15:00:04] jelto, arnoldokoth, mutante, and arnaudb: #bothumor My software never has bugs. It just develops random features. Rise for SRE Collaboration Services office hours. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1500). [15:00:04] jelto and dancy: May I have your attention please! GitLab Runner CI maintenance. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1500) [15:00:08] (03CR) 10Btullis: [C:03+1] "LGTM." [dns] - 10https://gerrit.wikimedia.org/r/1352139 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [15:00:23] (03PS1) 10Daniel Kinzler: REST Gateway: change link syntax used in ratee limit responses. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352145 [15:00:51] 06SRE, 10Observability-Alerting, 13Patch-For-Review, 07Sustainability (Incident Followup): Paging alert on combination of hosts down and a BGP outage - https://phabricator.wikimedia.org/T417051#12401539 (10cmooney) I'm not sure we want to create compound alerting rules like that. Fundamentally I would sug... [15:00:58] (03PS2) 10Bking: wikimedia.org: Add record for matomo-next [dns] - 10https://gerrit.wikimedia.org/r/1352139 (https://phabricator.wikimedia.org/T436003) [15:02:02] !log ladsgroup@deploy1003 helmfile [codfw] START helmfile.d/services/thumbor: apply [15:02:06] !log aokoth@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2003.codfw.wmnet with reason: Deployment [15:02:12] !log brennen@deploy1003 Started deploy [phabricator/deployment@2b620eb]: deploy phab2003 for T440340 [15:02:15] T440340: Deploy Phab/Phorge 2026-10-06 - https://phabricator.wikimedia.org/T440340 [15:02:32] !log jayme@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [15:02:48] !log jayme@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [15:03:03] !log brennen@deploy1003 Finished deploy [phabricator/deployment@2b620eb]: deploy phab2003 for T440340 (duration: 00m 51s) [15:03:07] !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'. [15:03:25] !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [15:03:27] !log ladsgroup@deploy1003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [15:03:32] (03CR) 10Hashar: "I declined it because the root cause got addressed, at the same time maxPages would not have prevented crawling a million document. I sho" [puppet] - 10https://gerrit.wikimedia.org/r/1308081 (https://phabricator.wikimedia.org/T431398) (owner: 10Arnaudb) [15:03:44] !log aokoth@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1005.eqiad.wmnet with reason: Deployment [15:03:50] (03CR) 10Cathal Mooney: "Honestly I'm not convinced this is the best way forward. I think what we should do is fix the network setup so this can't happen again (n" [alerts] - 10https://gerrit.wikimedia.org/r/1352138 (https://phabricator.wikimedia.org/T417051) (owner: 10Hnowlan) [15:03:50] !log brennen@deploy1003 Started deploy [phabricator/deployment@2b620eb]: deploy phab1005 for T440340 [15:03:51] (03PS3) 10CDanis: jaeger: match wmf release label [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352136 (https://phabricator.wikimedia.org/T439246) [15:04:10] dancy: meeting finished. so yea, now would be great! I'll take it slow, I haven't deployed in a while. But Spiderpig is going to do most of the work for me anyway. [15:04:18] Indeed. [15:04:32] !log brennen@deploy1003 Finished deploy [phabricator/deployment@2b620eb]: deploy phab1005 for T440340 (duration: 00m 42s) [15:04:57] (03CR) 10BCornwall: [C:03+1] geo-maps: update map default to list codfw first [dns] - 10https://gerrit.wikimedia.org/r/1319816 (https://phabricator.wikimedia.org/T439674) (owner: 10Slyngshede) [15:05:14] (03CR) 10BCornwall: [C:03+1] wmnet: remove testreduce CNAME record [dns] - 10https://gerrit.wikimedia.org/r/1351524 (https://phabricator.wikimedia.org/T436917) (owner: 10Jasmine) [15:05:44] (03CR) 10Eevans: [C:03+1] swift: mark sdn1 in ms-be1074 as failed [puppet] - 10https://gerrit.wikimedia.org/r/1351957 (https://phabricator.wikimedia.org/T440281) (owner: 10CWilliams) [15:05:45] (03CR) 10BCornwall: [C:03+1] wikimedia.org: Add record for matomo-next [dns] - 10https://gerrit.wikimedia.org/r/1352139 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [15:06:10] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [15:07:28] (03CR) 10CWilliams: [C:03+2] swift: mark sdn1 in ms-be1074 as failed [puppet] - 10https://gerrit.wikimedia.org/r/1351957 (https://phabricator.wikimedia.org/T440281) (owner: 10CWilliams) [15:07:53] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.cdn.roll-restart-ats (exit_code=0) rolling restart_daemons on A:cp-codfw [15:08:09] (03CR) 10Eevans: [C:03+2] abstractwiki-rust: Bump version number; trixie-backports has upgraded rustc etc. from 1.94.1 to 1.95.0 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1348701 (owner: 10Jforrester) [15:08:22] (03PS2) 10CWilliams: swift: mark sdn1 in ms-be1074 as failed [puppet] - 10https://gerrit.wikimedia.org/r/1351957 (https://phabricator.wikimedia.org/T440281) [15:08:46] (03CR) 10Eevans: [V:03+2 C:03+2] abstractwiki-rust: Bump version number; trixie-backports has upgraded rustc etc. from 1.94.1 to 1.95.0 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1348701 (owner: 10Jforrester) [15:08:47] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [15:08:56] (03PS4) 10CDanis: jaeger: match wmf release label [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352136 (https://phabricator.wikimedia.org/T439246) [15:09:51] (03CR) 10TrainBranchBot: [C:03+2] "Approved by daniel@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352094 (https://phabricator.wikimedia.org/T433534) (owner: 10Daniel Kinzler) [15:10:16] Spiderpigging... [15:10:46] !log jayme@deploy1003 helmfile [codfw] START helmfile.d/admin 'apply'. [15:10:56] (03Merged) 10jenkins-bot: REST: enable the content/v2-beta module on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352094 (https://phabricator.wikimedia.org/T433534) (owner: 10Daniel Kinzler) [15:11:03] !log jayme@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'apply'. [15:11:09] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.6 point update - https://phabricator.wikimedia.org/T434866#12401626 (10MoritzMuehlenhoff) [15:11:13] !log daniel@deploy1003 Started scap sync-world: Backport for [[gerrit:1352094|REST: enable the content/v2-beta module on testwiki (T433534)]] [15:11:16] T433534: [Hypothesis] PP3.5.2 REST API v2 Beta Launch - https://phabricator.wikimedia.org/T433534 [15:11:21] !log jayme@deploy1003 helmfile [eqiad] START helmfile.d/admin 'apply'. [15:11:31] (03CR) 10CWilliams: [C:03+2] swift: mark sdn1 in ms-be1074 as failed [puppet] - 10https://gerrit.wikimedia.org/r/1351957 (https://phabricator.wikimedia.org/T440281) (owner: 10CWilliams) [15:11:36] !log jayme@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'apply'. [15:11:48] !log installing python3.13 security updates [15:11:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:12:49] (03PS5) 10CDanis: jaeger: match wmf release label [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352136 (https://phabricator.wikimedia.org/T439246) [15:14:52] (03CR) 10CI reject: [V:04-1] jaeger: match wmf release label [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352136 (https://phabricator.wikimedia.org/T439246) (owner: 10CDanis) [15:17:37] !log daniel@deploy1003 daniel: Backport for [[gerrit:1352094|REST: enable the content/v2-beta module on testwiki (T433534)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:17:40] T433534: [Hypothesis] PP3.5.2 REST API v2 Beta Launch - https://phabricator.wikimedia.org/T433534 [15:21:39] !log daniel@deploy1003 daniel: Continuing with deployment [15:22:02] (03CR) 10Giuseppe Lavagetto: [C:04-1] "Of course if we can fix this in jaeger that's better, but I'd like to take a different approach." [puppet] - 10https://gerrit.wikimedia.org/r/1351965 (https://phabricator.wikimedia.org/T439409) (owner: 10Elukey) [15:23:48] 10ops-esams, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: esams router and switch maint - https://phabricator.wikimedia.org/T440349 (10RobH) 03NEW [15:25:24] (03PS6) 10CDanis: jaeger: match wmf release label [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352136 (https://phabricator.wikimedia.org/T439246) [15:26:58] !log daniel@deploy1003 Finished scap sync-world: Backport for [[gerrit:1352094|REST: enable the content/v2-beta module on testwiki (T433534)]] (duration: 15m 44s) [15:27:00] T433534: [Hypothesis] PP3.5.2 REST API v2 Beta Launch - https://phabricator.wikimedia.org/T433534 [15:27:53] (03PS1) 10Muehlenhoff: New Cumin alias memcached-all [puppet] - 10https://gerrit.wikimedia.org/r/1352155 [15:28:56] 10ops-esams, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: esams router and switch maint - https://phabricator.wikimedia.org/T440349#12401819 (10RobH) [15:30:09] dancy: deployment successful, code broken. No need to roll back, it's an experimental feature on testwiki. I'll come back later with a fix. Tomorrow probably. [15:30:23] Good news! [15:31:26] (03CR) 10Bking: [C:03+2] "Done" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345213 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [15:33:41] 10ops-esams, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: esams router and switch maint - https://phabricator.wikimedia.org/T440349#12401861 (10RobH) [15:34:46] 10ops-esams, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: esams router and switch maint - https://phabricator.wikimedia.org/T440349#12401863 (10RobH) [15:35:27] !log ladsgroup@deploy1003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [15:36:02] !log ladsgroup@deploy1003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [15:39:04] (03PS1) 10Blake: mcrouter: Switch updateStrategy to RollingUpdate, deploy flags. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352158 (https://phabricator.wikimedia.org/T420223) [15:43:11] (03CR) 10Elukey: [C:03+1] mcrouter: Switch updateStrategy to RollingUpdate, deploy flags. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352158 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [15:44:20] PROBLEM - Check if Pybal has been restarted after pybal.conf was changed on lvs1018 is CRITICAL: CRITICAL: Service pybal.service has not been restarted after /etc/pybal/pybal.conf was changed (gt 1h). https://wikitech.wikimedia.org/wiki/PyBal%23Pybal_service_has_not_been_restarted [15:45:21] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [15:45:48] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [15:46:06] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply [15:52:26] (03CR) 10Blake: [C:03+2] mcrouter: Switch updateStrategy to RollingUpdate, deploy flags. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352158 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [15:54:29] (03Merged) 10jenkins-bot: mcrouter: Switch updateStrategy to RollingUpdate, deploy flags. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352158 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [15:54:52] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply [15:55:00] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply [15:55:20] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply [15:56:24] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply [15:56:44] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply [15:56:45] !log blake@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-mcrouter: apply [15:58:09] !log blake@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-mcrouter: apply [16:00:04] jhathaway and rzl: May I have your attention please! Puppet request window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1600) [16:00:04] No Gerrit patches in the queue for this window AFAICS. [16:00:14] (03PS7) 10Giuseppe Lavagetto: [DNM] k8s cookbook to enable virtualization [cookbooks] - 10https://gerrit.wikimedia.org/r/1351902 (https://phabricator.wikimedia.org/T438287) [16:00:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [16:00:58] 06SRE, 06Infrastructure-Foundations, 06Traffic, 07LDAP: Add a new LVS ldap-ro-test service - https://phabricator.wikimedia.org/T439730#12402103 (10MoritzMuehlenhoff) ldap-ro-test has been rolled out to the LVS servers (step: lvs_setup) [16:02:44] (03CR) 10CI reject: [V:04-1] [DNM] k8s cookbook to enable virtualization [cookbooks] - 10https://gerrit.wikimedia.org/r/1351902 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [16:04:19] !log oblivian@cumin1004 START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2055-2062,2064-2065].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [16:04:23] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2055.codfw.wmnet [16:04:58] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2055.codfw.wmnet [16:05:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [16:05:19] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:05:43] (03CR) 10Hnowlan: "I don't feel very strongly about this alert either way - I think this kind of situation is still a pretty bad one that we should catch, bu" [alerts] - 10https://gerrit.wikimedia.org/r/1352138 (https://phabricator.wikimedia.org/T417051) (owner: 10Hnowlan) [16:10:13] 06SRE, 10SRE-Access-Requests, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Backup SSH key for Guillaume Lederrey (gehel) - https://phabricator.wikimedia.org/T440278#12402207 (10herron) 05Open→03Resolved a:03herron [16:10:20] (03CR) 10AKhatun: "Wanted to note we are going to rename it from `article_feature_counts` -> `revision_feature_counts`. The other CRs will need changing as w" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [16:11:42] FIRING: [4x] JobUnavailable: Reduced availability for job cfssl in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:12:37] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2055.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:12:50] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2055.codfw.wmnet [16:12:52] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2055.codfw.wmnet [16:13:06] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2056.codfw.wmnet [16:13:41] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2056.codfw.wmnet [16:14:02] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:14:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:16:42] FIRING: [4x] JobUnavailable: Reduced availability for job cfssl in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:21:04] 06SRE, 10SRE-Access-Requests: Remaining access pieces for Eva to use Jupyter - https://phabricator.wikimedia.org/T440031#12402385 (10herron) [16:21:55] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2056.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:22:12] 06SRE, 10SRE-Access-Requests: Remaining access pieces for Eva to use Jupyter - https://phabricator.wikimedia.org/T440031#12402398 (10herron) Hi @eniroomand could you please fill in any blank sections in the top of the description? Thanks! [16:22:13] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2056.codfw.wmnet [16:22:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2056.codfw.wmnet [16:22:29] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2057.codfw.wmnet [16:23:39] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2057.codfw.wmnet [16:23:59] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:24:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:25:30] 10ops-codfw, 06SRE, 10database-backups, 06DBA, 06DC-Ops: db2201 (backup source s5 and x1) crashed - https://phabricator.wikimedia.org/T440283#12402415 (10Jhancock.wm) @Marostegui This server is still under warranty and we should add it. I can start pulling reports for this. [16:26:24] 10ops-codfw, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging200[1-3].codfw.wmnet - https://phabricator.wikimedia.org/T440181#12402417 (10Jhancock.wm) [16:28:01] (03CR) 10Dzahn: [C:03+1] "per comment on https://phabricator.wikimedia.org/T431398 we could set it to 5000" [puppet] - 10https://gerrit.wikimedia.org/r/1308081 (https://phabricator.wikimedia.org/T431398) (owner: 10Arnaudb) [16:31:48] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2057.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:31:59] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2057.codfw.wmnet [16:32:53] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2057.codfw.wmnet [16:33:08] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2058.codfw.wmnet [16:33:20] (03PS3) 10DLynch: services_proxy: Enable the linked-artifacts listener for MediaWiki [puppet] - 10https://gerrit.wikimedia.org/r/1351442 (https://phabricator.wikimedia.org/T432733) [16:33:41] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2058.codfw.wmnet [16:33:48] (03PS2) 10DLynch: VisualEditor: Precompute edit suggestion counts on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351444 (https://phabricator.wikimedia.org/T432733) [16:34:00] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:34:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:34:17] (03CR) 10CI reject: [V:04-1] services_proxy: Enable the linked-artifacts listener for MediaWiki [puppet] - 10https://gerrit.wikimedia.org/r/1351442 (https://phabricator.wikimedia.org/T432733) (owner: 10DLynch) [16:34:24] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:36:25] (03CR) 10Andrew Bogott: [C:03+2] zookeeper: remove need to opt-in to enable logging on trixie and beyond [puppet] - 10https://gerrit.wikimedia.org/r/1345209 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [16:37:04] (03CR) 10Bking: [C:03+2] wikimedia.org: Add record for matomo-next [dns] - 10https://gerrit.wikimedia.org/r/1352139 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [16:37:25] !log bking@dns2004 START - running authdns-update [16:39:41] !log bking@dns2004 END - running authdns-update [16:41:48] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2058.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:42:17] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2058.codfw.wmnet [16:42:19] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2058.codfw.wmnet [16:42:33] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2059.codfw.wmnet [16:43:06] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2059.codfw.wmnet [16:43:26] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:44:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:47:17] (03PS4) 10DLynch: services_proxy: Enable the linked-artifacts listener for MediaWiki [puppet] - 10https://gerrit.wikimedia.org/r/1351442 (https://phabricator.wikimedia.org/T432733) [16:49:22] (03PS1) 10Ebernhardson: cirrus: Attach X-Cluster-Name header to responses [puppet] - 10https://gerrit.wikimedia.org/r/1352186 (https://phabricator.wikimedia.org/T440376) [16:51:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2059.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:51:24] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2059.codfw.wmnet [16:52:18] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2059.codfw.wmnet [16:52:32] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2060.codfw.wmnet [16:53:09] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2060.codfw.wmnet [16:53:29] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:54:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:54:24] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:58:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [16:58:35] 06SRE, 10Observability-Alerting, 13Patch-For-Review, 07Sustainability (Incident Followup): Paging alert on combination of hosts down and a BGP outage - https://phabricator.wikimedia.org/T417051#12402674 (10cmooney) >>! In T417051#12401539, @cmooney wrote: > Fundamentally I would suggest the issue here is t... [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1700) [17:00:48] (03CR) 10Cathal Mooney: team-netops: create alert for combined impact of BGP impact and probe failure (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1352138 (https://phabricator.wikimedia.org/T417051) (owner: 10Hnowlan) [17:02:16] 10ops-codfw, 06SRE, 10database-backups, 06DBA, 06DC-Ops: db2201 (backup source s5 and x1) crashed - https://phabricator.wikimedia.org/T440283#12402714 (10Marostegui) >>! In T440283#12402415, @Jhancock.wm wrote: > @Marostegui This server is still under warranty and we should add it. I can start pulling r... [17:03:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [17:03:04] (03CR) 10Ottomata: [C:03+1] VisualEditor: Precompute edit suggestion counts on testwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351444 (https://phabricator.wikimedia.org/T432733) (owner: 10DLynch) [17:03:52] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2060.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:04:10] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2060.codfw.wmnet [17:04:12] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2060.codfw.wmnet [17:04:24] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:04:26] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2061.codfw.wmnet [17:05:03] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2061.codfw.wmnet [17:05:24] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:08:22] !log bking@cumin2003 START - Cookbook sre.ganeti.reboot-vm for VM matomo1003.eqiad.wmnet [17:12:06] (03CR) 10Scott French: [C:03+1] "Adding Blake to the reviewer set, since (IIUC) this will proceed once the comms in T436189 are complete, which will likely happen after th" [puppet] - 10https://gerrit.wikimedia.org/r/1329690 (https://phabricator.wikimedia.org/T436178) (owner: 10BryanDavis) [17:12:20] !log bking@cumin2003 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM matomo1003.eqiad.wmnet [17:12:42] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2061.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:13:01] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2061.codfw.wmnet [17:13:56] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2061.codfw.wmnet [17:14:09] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2062.codfw.wmnet [17:14:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:14:44] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2062.codfw.wmnet [17:15:04] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:15:23] (03CR) 10Bking: dumps: web: Add a brownout for v1 meta-current and meta-history (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1346007 (https://phabricator.wikimedia.org/T437461) (owner: 10Btullis) [17:19:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:22:53] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2062.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:23:11] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2062.codfw.wmnet [17:24:06] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2062.codfw.wmnet [17:24:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:24:19] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2064.codfw.wmnet [17:24:54] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2064.codfw.wmnet [17:25:14] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:29:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:29:22] 06SRE, 10Infrastructure Security, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07SecTeam-Processed, and 2 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12402840 (10bking) [17:33:45] !log fceratto@cumin1004 END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db1901: test [17:34:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:34:25] 10ops-codfw, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging200[1-3].codfw.wmnet - https://phabricator.wikimedia.org/T440181#12402856 (10Jhancock.wm) 05Open→03Resolved [17:34:38] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2064.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:34:56] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2064.codfw.wmnet [17:35:49] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker2064.codfw.wmnet [17:36:03] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2065.codfw.wmnet [17:36:37] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2065.codfw.wmnet [17:36:56] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:39:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:39:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:39:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:39:56] 06SRE, 10Infrastructure Security, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07SecTeam-Processed, and 2 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12402883 (10bking) [17:44:16] !log marostegui@cumin1004 START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test [17:45:02] (03Abandoned) 10Marostegui: dbconfig.schema: Add test-s4 to main [puppet] - 10https://gerrit.wikimedia.org/r/1329257 (https://phabricator.wikimedia.org/T427059) (owner: 10Marostegui) [17:45:24] (03PS1) 10Catrope: Email confirmation: Temporarily disable banner modification [extensions/WikimediaEvents] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352217 [17:45:28] (03PS1) 10Marostegui: Revert "db2207,db2209,db1218: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1352218 [17:45:36] (03PS1) 10Catrope: Email confirmation: Temporarily disable banner modification [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352219 [17:45:48] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 06 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352217 (owner: 10Catrope) [17:45:49] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2065.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:45:57] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 06 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352219 (owner: 10Catrope) [17:46:13] !log marostegui@cumin1004 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2207: Pooling back db2207 after stress test [17:46:17] (03CR) 10Marostegui: [C:03+2] Revert "db2207,db2209,db1218: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1352218 (owner: 10Marostegui) [17:46:18] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2065.codfw.wmnet [17:46:20] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2065.codfw.wmnet [17:46:21] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2055-2062,2064-2065].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [17:47:13] !log marostegui@cumin1004 START - Cookbook sre.mysql.pool pool db2207: Pooling back db2207 after stress test [17:47:32] !log marostegui@cumin1004 START - Cookbook sre.mysql.pool pool db2209: Pooling back db2209 after stress test [17:47:44] !log marostegui@cumin1004 START - Cookbook sre.mysql.pool pool db1218: Pooling back db1218 after stress test [17:47:55] !log Repool db1218, db2207 and db2209 after stress testing [17:47:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:48:14] (03CR) 10CI reject: [V:04-1] Email confirmation: Temporarily disable banner modification [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352219 (owner: 10Catrope) [17:49:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:53:09] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad: WMCS switch refresh 2026 - rack new switches - https://phabricator.wikimedia.org/T440313#12402958 (10Jclark-ctr) These rails were used to replace the broken ones from the A/B install. [17:55:48] (03CR) 10Jsn.sherman: [C:03+1] InitialiseSettings: Enable ModeratorToolkit on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352019 (https://phabricator.wikimedia.org/T437617) (owner: 10Dillon) [17:55:56] (03CR) 10Jsn.sherman: [C:03+1] InitialiseSettings: Enable ModeratorToolkit on four group2 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352018 (https://phabricator.wikimedia.org/T437617) (owner: 10Dillon) [18:00:05] andre and brennen: OwO what's this, a deployment window?? MediaWiki train - Utc-0+Utc-7 Version (secondary timeslot). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T1800). nyaa~ [18:01:54] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet [18:01:59] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2055-2062,2064-2065].codfw.wmnet [18:13:49] (03PS1) 10Ebernhardson: opensearch-cluster: Add X-Cluster-Name response header [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352232 (https://phabricator.wikimedia.org/T440376) [18:21:02] (03PS3) 10Scott French: changeprop: Wire in mesh.networkpolicy.egress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352101 (https://phabricator.wikimedia.org/T434925) [18:23:55] (03PS4) 10Scott French: changeprop-jobqueue: Enable service mesh and mw-jobrunner listener [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325936 (https://phabricator.wikimedia.org/T434925) [18:23:56] (03PS5) 10Scott French: changeprop-jobqueue: Access mw-jobrunner via the service mesh [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325937 (https://phabricator.wikimedia.org/T434925) [18:24:45] (03PS2) 10Scott French: changeprop: Enable service mesh and mw-api-int-async listener [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352075 (https://phabricator.wikimedia.org/T434925) [18:24:51] (03PS2) 10Scott French: changeprop: Access mw-api-int via the service mesh [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352095 (https://phabricator.wikimedia.org/T434925) [18:30:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [18:31:22] (03PS1) 10CWilliams: swift: mark sdn1 in ms-be1074 as failed [puppet] - 10https://gerrit.wikimedia.org/r/1352180 (https://phabricator.wikimedia.org/T440281) [18:32:17] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2207: Pooling back db2207 after stress test [18:32:36] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2209: Pooling back db2209 after stress test [18:32:48] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1218: Pooling back db1218 after stress test [18:33:12] (03CR) 10Bking: [C:03+1] "Conditional +1 on updating the chart's changelog ( charts/opensearch-cluster/CHANGELOG.md )" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352232 (https://phabricator.wikimedia.org/T440376) (owner: 10Ebernhardson) [18:35:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [18:40:36] (03PS2) 10CWilliams: swift: mark sdn1 in ms-be1074 as failed [puppet] - 10https://gerrit.wikimedia.org/r/1352180 (https://phabricator.wikimedia.org/T440281) [18:40:53] (03CR) 10CWilliams: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352180 (https://phabricator.wikimedia.org/T440281) (owner: 10CWilliams) [18:41:28] (03PS3) 10Ahmon Dancy: mediawiki::deployment::server: Automate scope=pretrain deployments [puppet] - 10https://gerrit.wikimedia.org/r/1329690 (https://phabricator.wikimedia.org/T436178) (owner: 10BryanDavis) [18:43:27] (03PS1) 10Eevans: linked_artifacts: deploy Hoarde v1.5.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352249 [18:45:35] (03CR) 10Eevans: [C:03+2] linked_artifacts: deploy Hoarde v1.5.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352249 (owner: 10Eevans) [18:47:00] (03Merged) 10jenkins-bot: linked_artifacts: deploy Hoarde v1.5.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352249 (owner: 10Eevans) [18:48:13] !log eevans@deploy1003 helmfile [staging] START helmfile.d/services/linked-artifacts: apply [18:48:27] !log eevans@deploy1003 helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply [18:49:26] (03CR) 10CDanis: [C:03+1] swift: mark sdn1 in ms-be1074 as failed [puppet] - 10https://gerrit.wikimedia.org/r/1352180 (https://phabricator.wikimedia.org/T440281) (owner: 10CWilliams) [18:50:48] (03PS3) 10CWilliams: swift: mark sdn1 in ms-be1074 as failed [puppet] - 10https://gerrit.wikimedia.org/r/1352180 (https://phabricator.wikimedia.org/T440281) [18:51:55] (03CR) 10CWilliams: [C:03+2] swift: mark sdn1 in ms-be1074 as failed [puppet] - 10https://gerrit.wikimedia.org/r/1352180 (https://phabricator.wikimedia.org/T440281) (owner: 10CWilliams) [18:54:38] (03PS2) 10Ebernhardson: opensearch-cluster: Add X-Cluster-Name response header [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352232 (https://phabricator.wikimedia.org/T440376) [18:56:13] 06SRE, 10SRE-Access-Requests: Requesting access to ops for lizf - https://phabricator.wikimedia.org/T439933#12403190 (10herron) @Lizimedia Hello! Would you like the honors of preparing the puppet patch to `modules/admin/data/data.yaml` adding your account to ops? Happy to help along the way, or prepare it my... [18:58:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1016:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1016 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [18:59:42] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12403197 (10herron) [19:03:40] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12403229 (10herron) Great! I'll prepare the patch now, but in the mean time could you please confirm with me the ssh key via a second method ("out of band"). Email would be fine, I'm k... [19:05:50] (03CR) 10Catrope: "recheck" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352219 (owner: 10Catrope) [19:06:13] (03PS1) 10Catrope: EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate() [extensions/WikimediaEvents] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352259 [19:06:19] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 06 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352259 (owner: 10Catrope) [19:06:42] (03PS1) 10Herron: admin: add user greyolsen to deployment group [puppet] - 10https://gerrit.wikimedia.org/r/1352166 (https://phabricator.wikimedia.org/T439143) [19:06:44] (03PS1) 10Herron: admin: add dlin to ldap_only_users [puppet] - 10https://gerrit.wikimedia.org/r/1352170 (https://phabricator.wikimedia.org/T440228) [19:06:46] (03PS1) 10Catrope: EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate() [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352260 [19:06:56] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 06 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352260 (owner: 10Catrope) [19:08:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1016:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1016 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [19:09:10] (03PS2) 10Herron: admin: add ealbizzati to group deployment [puppet] - 10https://gerrit.wikimedia.org/r/1352258 (https://phabricator.wikimedia.org/T437741) [19:09:33] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12403254 (10EAlbizzati-WMF) @herron done. Thanks [19:10:28] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12403263 (10herron) >>! In T437741#12403254, @EAlbizzati-WMF wrote: > @herron done. Thanks Confirmed! [19:10:34] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12403264 (10herron) [19:14:26] (03PS3) 10Dduvall: Refactor tests to run inline with build [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345668 (https://phabricator.wikimedia.org/T439490) [19:18:18] (03CR) 10Dduvall: "@glavagetto@wikimedia.org this is ready for review." [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345668 (https://phabricator.wikimedia.org/T439490) (owner: 10Dduvall) [19:21:44] (03PS1) 10Clare Ming: Update Test Kitchen instruments intake URL on beta cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352261 (https://phabricator.wikimedia.org/T437720) [19:27:53] (03CR) 10Herron: [C:03+2] admin: add dlin to ldap_only_users [puppet] - 10https://gerrit.wikimedia.org/r/1352170 (https://phabricator.wikimedia.org/T440228) (owner: 10Herron) [19:29:12] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad A/B switch cabling documentation - https://phabricator.wikimedia.org/T418018#12403321 (10Jclark-ctr) @cmooney @RobH Finished A8 LSW, spine management, and SCS cabling. Updated the SCS port labels and Google Sheet. Adjusted the... [19:29:42] (03PS1) 10Milazg: WIP: change config [puppet] - 10https://gerrit.wikimedia.org/r/1352263 [19:30:21] 06SRE, 10LDAP-Access-Requests, 13Patch-For-Review: Requesting LDAP access to Superset for DLin - https://phabricator.wikimedia.org/T440228#12403322 (10herron) [19:31:03] 06SRE, 10LDAP-Access-Requests: Requesting LDAP access to Superset for DLin - https://phabricator.wikimedia.org/T440228#12403331 (10herron) 05Open→03Resolved a:03herron Requested group has been added, I'll transition this to resolved now but please don't hesitate to re-open if any followup is needed.... [19:32:39] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to Deployment for GreyOlson/GOlson-WMF - https://phabricator.wikimedia.org/T439143#12403338 (10herron) [19:38:06] (03PS2) 10Milazg: WIP: change config [puppet] - 10https://gerrit.wikimedia.org/r/1352263 [19:38:20] (03CR) 10Milazg: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352263 (owner: 10Milazg) [19:44:08] (03PS3) 10Milazg: WIP: change config [puppet] - 10https://gerrit.wikimedia.org/r/1352263 [19:44:16] (03CR) 10Btullis: dumps: web: Add a brownout for v1 meta-current and meta-history (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1346007 (https://phabricator.wikimedia.org/T437461) (owner: 10Btullis) [19:44:53] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 268432904 and 39 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [19:45:38] (03CR) 10Milazg: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352263 (owner: 10Milazg) [19:45:53] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 3085824 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [19:51:55] (03CR) 10Btullis: [C:03+2] install_server: Reuse the 8 disk recipe for an-druid100[67] [puppet] - 10https://gerrit.wikimedia.org/r/1351322 (https://phabricator.wikimedia.org/T440033) (owner: 10Btullis) [19:59:44] (03CR) 10Dzahn: "it seems like the very first run should be done manually. if that still means to catch-up from a longer time ago it might not finish in th" [puppet] - 10https://gerrit.wikimedia.org/r/1352124 (https://phabricator.wikimedia.org/T440177) (owner: 10Arnaudb) [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: gettimeofday() says it's time for UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T2000) [20:00:05] RoanKattouw: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:14] (03PS1) 10Xcollazo: dumps: Copy only complete File Exports to the distribution hosts [puppet] - 10https://gerrit.wikimedia.org/r/1352267 (https://phabricator.wikimedia.org/T440383) [20:01:35] (03CR) 10TrainBranchBot: [C:03+2] "Approved by catrope@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352217 (owner: 10Catrope) [20:01:35] (03CR) 10TrainBranchBot: [C:03+2] "Approved by catrope@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352219 (owner: 10Catrope) [20:01:35] (03CR) 10TrainBranchBot: [C:03+2] "Approved by catrope@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352259 (owner: 10Catrope) [20:01:36] (03CR) 10TrainBranchBot: [C:03+2] "Approved by catrope@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352260 (owner: 10Catrope) [20:02:54] (03Merged) 10jenkins-bot: Email confirmation: Temporarily disable banner modification [extensions/WikimediaEvents] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352217 (owner: 10Catrope) [20:02:55] (03Merged) 10jenkins-bot: Email confirmation: Temporarily disable banner modification [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352219 (owner: 10Catrope) [20:02:57] (03Merged) 10jenkins-bot: EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate() [extensions/WikimediaEvents] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352259 (owner: 10Catrope) [20:03:04] (03Merged) 10jenkins-bot: EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate() [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352260 (owner: 10Catrope) [20:03:29] (03CR) 10Bking: [C:03+1] dumps: web: Add a brownout for v1 meta-current and meta-history [puppet] - 10https://gerrit.wikimedia.org/r/1346007 (https://phabricator.wikimedia.org/T437461) (owner: 10Btullis) [20:03:32] !log catrope@deploy1003 Started scap sync-world: Backport for [[gerrit:1352217|Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219|Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259|EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260|EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] [20:07:46] !log catrope@deploy1003 catrope: Backport for [[gerrit:1352217|Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219|Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259|EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260|EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] synced to the testservers (see htt [20:07:46] ps://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:09:24] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:10:21] PROBLEM - MariaDB Replica Lag: s8 on db1285 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 604.81 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:10:29] PROBLEM - MariaDB Replica Lag: s8 on db2198 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 611.02 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:10:31] !log catrope@deploy1003 catrope: Continuing with deployment [20:10:39] PROBLEM - MariaDB Replica Lag: s1 on db2250 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 622.11 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:11:01] PROBLEM - MariaDB Replica Lag: s1 on db1240 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 644.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:13:23] 10ops-ulsfo, 06SRE, 06DC-Ops: ulsfo: OOB migration from copper to fiber - https://phabricator.wikimedia.org/T436499#12403452 (10RobH) Ok, confirmed the parts have arrived. DigitalRealty will refund the charges for the remote hands to swap the copper to fiber for oob in ulsfo. The parts for this were delive... [20:15:49] !log catrope@deploy1003 Finished scap sync-world: Backport for [[gerrit:1352217|Email confirmation: Temporarily disable banner modification]], [[gerrit:1352219|Email confirmation: Temporarily disable banner modification]], [[gerrit:1352259|EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]], [[gerrit:1352260|EmailConfirmationHooks: Replace hard-coded timestamp with getStartDate()]] (duration: 12m 17 [20:15:49] s) [20:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job cfssl in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:22:45] 10ops-ulsfo, 06SRE, 06DC-Ops: ulsfo: OOB migration from copper to fiber - https://phabricator.wikimedia.org/T436499#12403480 (10Papaul) I don't there is any coordination needed for this. the port is also configured the OOB is already on port ge-0/0/7 which is a SFP port so the only thing needed is just the... [20:24:59] 06SRE, 06Infrastructure-Foundations, 10netops: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#12403488 (10Papaul) We are waiting to upgrade the Nokia swatches in codfw and we can resolve this task. [20:47:17] 10ops-ulsfo, 06SRE, 06DC-Ops: ulsfo: OOB migration from copper to fiber - https://phabricator.wikimedia.org/T436499#12403659 (10RobH) [20:49:09] (03CR) 10Xcollazo: [C:03+1] dumps: web: Add a brownout for v1 meta-current and meta-history [puppet] - 10https://gerrit.wikimedia.org/r/1346007 (https://phabricator.wikimedia.org/T437461) (owner: 10Btullis) [20:57:43] (03CR) 10Ryan Kemper: [C:03+1] sre.hadoop.roll-restart-workers: implement start-datetime logic [cookbooks] - 10https://gerrit.wikimedia.org/r/1347713 (https://phabricator.wikimedia.org/T437713) (owner: 10Bking) [20:58:58] (03CR) 10Bking: sre.hadoop.roll-restart-workers: implement start-datetime logic (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1347713 (https://phabricator.wikimedia.org/T437713) (owner: 10Bking) [20:59:14] (03CR) 10Bking: [C:03+2] sre.hadoop.roll-restart-workers: implement start-datetime logic [cookbooks] - 10https://gerrit.wikimedia.org/r/1347713 (https://phabricator.wikimedia.org/T437713) (owner: 10Bking) [21:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261006T2100) [21:14:29] RECOVERY - MariaDB Replica Lag: s8 on db2198 is OK: OK slave_sql_lag Replication lag: 0.07 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:16:21] RECOVERY - MariaDB Replica Lag: s8 on db1285 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:26:54] (03PS1) 10Andrew Bogott: Openstack Nova: update minimum CPU model for new VMs [puppet] - 10https://gerrit.wikimedia.org/r/1352280 (https://phabricator.wikimedia.org/T440400) [21:29:01] (03PS2) 10Andrew Bogott: Openstack Nova: update minimum CPU model for new VMs [puppet] - 10https://gerrit.wikimedia.org/r/1352280 (https://phabricator.wikimedia.org/T440400) [21:29:31] PROBLEM - MariaDB Replica Lag: x3 on db2200 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 642.02 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:32:48] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352280 (https://phabricator.wikimedia.org/T440400) (owner: 10Andrew Bogott) [21:37:39] (03PS1) 10Bking: Data-platform: migrate masters-eligible check to Prometheus [alerts] - 10https://gerrit.wikimedia.org/r/1352282 (https://phabricator.wikimedia.org/T431304) [21:40:31] (03PS2) 10Bking: Data-platform: migrate masters-eligible check to Prometheus [alerts] - 10https://gerrit.wikimedia.org/r/1352282 (https://phabricator.wikimedia.org/T431304) [21:42:36] (03PS3) 10Andrew Bogott: Openstack Nova: update minimum CPU model for new VMs [puppet] - 10https://gerrit.wikimedia.org/r/1352280 (https://phabricator.wikimedia.org/T440400) [21:45:31] RECOVERY - MariaDB Replica Lag: x3 on db2200 is OK: OK slave_sql_lag Replication lag: 0.20 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:46:31] I'm a bit late but can I use the readers deployment window? [21:46:35] (03CR) 10Scott French: [C:03+1] "Thanks, Ahmon! As discussed out of band and on T435419, the math gets a little weird in terms of mapping the observed production canary vs" [puppet] - 10https://gerrit.wikimedia.org/r/1345206 (https://phabricator.wikimedia.org/T435419) (owner: 10Ahmon Dancy) [21:47:52] (03PS1) 10Jdlrobson: Revert "Scroll the TOC along with the page scrolling" [skins/Vector] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352284 (https://phabricator.wikimedia.org/T440296) [21:48:00] (03CR) 10Andrew Bogott: "This assumes that newer CPU are fully backwards-compatible with older ones. That has always been true in the past, but before applying thi" [puppet] - 10https://gerrit.wikimedia.org/r/1352280 (https://phabricator.wikimedia.org/T440400) (owner: 10Andrew Bogott) [21:49:24] FIRING: KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1152.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [21:49:41] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [skins/Vector] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352284 (https://phabricator.wikimedia.org/T440296) (owner: 10Jdlrobson) [21:52:01] PROBLEM - MariaDB Replica Lag: x3 on db1216 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 601.13 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:54:56] (03CR) 10JHathaway: "could the enc path be set in hiera and then switch to the include syntax?" [puppet] - 10https://gerrit.wikimedia.org/r/1137013 (owner: 10Krinkle) [21:55:37] (03Merged) 10jenkins-bot: Revert "Scroll the TOC along with the page scrolling" [skins/Vector] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352284 (https://phabricator.wikimedia.org/T440296) (owner: 10Jdlrobson) [21:55:55] !log jdlrobson@deploy1003 Started scap sync-world: Backport for [[gerrit:1352284|Revert "Scroll the TOC along with the page scrolling" (T440296)]] [21:55:58] T440296: Pages scroll unexpectedly when the intro section is short on Firefox and Safari - https://phabricator.wikimedia.org/T440296 [21:56:01] RECOVERY - MariaDB Replica Lag: s1 on db1240 is OK: OK slave_sql_lag Replication lag: 0.18 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:00:06] !log jdlrobson@deploy1003 jdlrobson: Backport for [[gerrit:1352284|Revert "Scroll the TOC along with the page scrolling" (T440296)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:01:42] !log jdlrobson@deploy1003 jdlrobson: Continuing with deployment [22:03:01] RECOVERY - MariaDB Replica Lag: x3 on db1216 is OK: OK slave_sql_lag Replication lag: 0.24 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:03:39] RECOVERY - MariaDB Replica Lag: s1 on db2250 is OK: OK slave_sql_lag Replication lag: 0.10 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:07:19] !log jdlrobson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1352284|Revert "Scroll the TOC along with the page scrolling" (T440296)]] (duration: 11m 24s) [22:07:22] T440296: Pages scroll unexpectedly when the intro section is short on Firefox and Safari - https://phabricator.wikimedia.org/T440296 [22:17:29] PROBLEM - MariaDB Replica Lag: s4 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 617.84 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:17:33] (03PS5) 10Scott French: mediawiki: Update mesh modules and dependents [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338000 (https://phabricator.wikimedia.org/T427666) [22:17:33] (03PS2) 10Scott French: changeprop: Update mesh modules and dependents [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351523 (https://phabricator.wikimedia.org/T427666) [22:17:33] (03PS4) 10Scott French: Update mesh.{configuration,networkpolicy} and dependents in all charts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338030 (https://phabricator.wikimedia.org/T427666) [22:17:43] (03PS3) 10Scott French: changeprop: Support enabling the mesh as a sidecar [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351528 (https://phabricator.wikimedia.org/T434925) [22:17:43] (03PS4) 10Scott French: changeprop: Wire in mesh.networkpolicy.egress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352101 (https://phabricator.wikimedia.org/T434925) [22:24:01] PROBLEM - MariaDB Replica Lag: s4 on db1265 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 629.05 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:33:05] (03PS1) 10MusikAnimal: CommonSettings: ensure 'usecodemirror' user option is unset globally [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352288 (https://phabricator.wikimedia.org/T440077) [22:34:30] (03PS3) 10Ryan Kemper: Data-platform: migrate masters-eligible check to Prometheus [alerts] - 10https://gerrit.wikimedia.org/r/1352282 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [22:37:37] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352290 [22:49:20] (03PS1) 10Dzahn: gerrit: recover a killed replica reindex before the replica check [puppet] - 10https://gerrit.wikimedia.org/r/1352297 (https://phabricator.wikimedia.org/T440177) [23:02:13] (03PS1) 10MusikAnimal: CommonSettings: remove redundant CodeMirror config settings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352303 (https://phabricator.wikimedia.org/T440077) [23:03:28] (03CR) 10Ryan Kemper: [C:03+1] "I pushed a patchset to fix the dashboard links, LGTM now" [alerts] - 10https://gerrit.wikimedia.org/r/1352282 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [23:38:59] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1352310 [23:38:59] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1352310 (owner: 10TrainBranchBot) [23:45:01] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1352310 (owner: 10TrainBranchBot) [23:51:03] (03CR) 10Scott French: Update mesh.{configuration,networkpolicy} and dependents in all charts (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338030 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [23:52:29] RECOVERY - MariaDB Replica Lag: s4 on db2239 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response