[00:01:25] !log dzahn@cumin1004 START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie [00:01:42] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12372162 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1004 for host zuul1005.eqiad.wmnet with OS tr... [00:10:23] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 613.29 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:10:37] PROBLEM - MariaDB Replica Lag: s3 on db1240 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 630.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:17:12] !log dzahn@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage [00:21:26] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!!" [puppet] - 10https://gerrit.wikimedia.org/r/1345087 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [00:22:37] !log dzahn@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage [00:32:16] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12372206 (10Dzahn) I reimaged zuul1006 and zuul1007 with the cookbook and no issues. zuul1005 kept having the issue we have previously seen... [00:46:24] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.32 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:50:32] (03PS1) 10Andrea Denisse: kubernetes: Add dummy KLAXON_CORTO_TOKEN for oncall [labs/private] - 10https://gerrit.wikimedia.org/r/1345674 (https://phabricator.wikimedia.org/T439495) [01:08:56] (03PS1) 10TrainBranchBot: Branch commit for wmf/1.47.0-wmf.22 [core] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1345678 (https://phabricator.wikimedia.org/T438218) [01:08:58] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/1.47.0-wmf.22 [core] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1345678 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [01:09:03] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1345679 [01:09:03] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1345679 (owner: 10TrainBranchBot) [01:16:43] (03Merged) 10jenkins-bot: Branch commit for wmf/1.47.0-wmf.22 [core] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1345678 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [01:17:20] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1345679 (owner: 10TrainBranchBot) [01:20:52] PROBLEM - Host wikikube-worker1016 is DOWN: PING CRITICAL - Packet loss = 60%, RTA = 3082.05 ms [01:21:30] RECOVERY - Host wikikube-worker1016 is UP: PING OK - Packet loss = 0%, RTA = 18.10 ms [01:29:01] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [01:49:18] !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_esams - 7.1.1-2~bpo13+wmf3 () [02:00:05] Deploy window Automatic branching of MediaWiki, extensions, skins, and vendor – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T0200) [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T0200) [02:00:35] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:06:38] RECOVERY - MariaDB Replica Lag: s3 on db1240 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:08:03] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 28s) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:22:38] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [02:24:26] RECOVERY - dump of s4 in codfw on backupmon1001 is OK: Last dump for s4 at codfw (db2239) taken on 2026-09-29 01:02:41 (180 GiB, +0.1 %) https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Rerun_a_failed_backup [02:27:40] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 226.32 ms [02:29:01] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [02:37:30] FIRING: Traffic bill over quota: Alert for device cr2-magru.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [02:39:01] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:57:30] RESOLVED: Traffic bill over quota: Alert for device cr2-magru.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [03:00:05] Deploy window Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T0300) [03:01:57] (03PS1) 10TrainBranchBot: testwikis to 1.47.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345691 (https://phabricator.wikimedia.org/T438218) [03:01:59] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by mwpresync@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345691 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [03:02:58] (03Merged) 10jenkins-bot: testwikis to 1.47.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345691 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [03:03:22] !log mwpresync@deploy1003 Started scap sync-world: testwikis to 1.47.0-wmf.22 refs T438218 [03:03:26] T438218: 1.47.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T438218 [03:37:09] (03PS7) 10Ryan Kemper: kafka: converge topic config from hieradata [puppet] - 10https://gerrit.wikimedia.org/r/1318976 (https://phabricator.wikimedia.org/T276088) [03:39:25] !log mwpresync@deploy1003 Finished scap sync-world: testwikis to 1.47.0-wmf.22 refs T438218 (duration: 36m 03s) [03:39:28] T438218: 1.47.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T438218 [04:00:05] Deploy window Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T0400) [04:01:05] !log mwpresync@deploy1003 Pruned MediaWiki: 1.47.0-wmf.19 (duration: 01m 00s) [04:31:05] (03PS1) 10KartikMistry: Update cxserver to 2026-09-25-114231-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345697 [04:31:55] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs1033.eqiad.wmnet with OS bookworm [04:32:17] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs1033 [04:32:18] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1033 [04:43:35] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage [04:44:16] (03CR) 10LWatson: [C:03+1] carousel: fix carousel image preload observer root [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345633 (https://phabricator.wikimedia.org/T439422) (owner: 10Matthias Mullie) [04:44:38] (03CR) 10LWatson: [C:03+1] Make carousel images are visible in the margins [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345627 (https://phabricator.wikimedia.org/T439422) (owner: 10Matthias Mullie) [04:47:59] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1033.eqiad.wmnet with reason: host reimage [04:51:35] !log manually restarting dump_cloud_ip_range on puppetserver1001 after failure [04:51:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:03:02] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1033.eqiad.wmnet with OS bookworm [05:07:25] (03PS1) 10Fabfur: external_clouds_vendors: minor logging fix [puppet] - 10https://gerrit.wikimedia.org/r/1345850 [05:17:55] PROBLEM - Host wikikube-worker1289 is DOWN: PING CRITICAL - Packet loss = 100% [05:19:28] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs1034.eqiad.wmnet with OS bookworm [05:19:40] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs1034 [05:19:40] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1034 [05:20:25] RECOVERY - Host wikikube-worker1289 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms [05:21:12] !log ryankemper@cumin2003 START - Cookbook sre.hosts.reimage for host wdqs1035.eqiad.wmnet with OS bookworm [05:21:35] !log ryankemper@cumin2003 START - Cookbook sre.hosts.move-vlan for host wdqs1035 [05:21:35] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1035 [05:29:01] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:31:14] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage [05:32:59] !log ryankemper@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage [05:34:47] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1034.eqiad.wmnet with reason: host reimage [05:38:15] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1035.eqiad.wmnet with reason: host reimage [05:50:29] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1034.eqiad.wmnet with OS bookworm [05:54:01] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs1035.eqiad.wmnet with OS bookworm [05:55:39] 06SRE, 06Data-Engineering, 10Kafka-Infrastructure, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Monitor kafka-configurator runs and pending changes - https://phabricator.wikimedia.org/T439506 (10RKemper) 03NEW [05:57:36] (03PS2) 10Giuseppe Lavagetto: kubernetes::deployment_server: setup daily timer to reload certificates [puppet] - 10https://gerrit.wikimedia.org/r/1345600 (https://phabricator.wikimedia.org/T439409) [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T0600) [06:00:05] marostegui, cezmunsta, and federico3: Time to snap out of that daydream and deploy Primary database switchover. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T0600). [06:01:28] 06SRE, 06Data-Engineering, 10Kafka-Infrastructure, 06serviceops-radar, and 4 others: Configuration Management for Kafka settings - https://phabricator.wikimedia.org/T276088#12372533 (10RKemper) [06:02:50] (03PS1) 10Ryan Kemper: aptrepo: import kafka-configurator from apt-staging [puppet] - 10https://gerrit.wikimedia.org/r/1345852 (https://phabricator.wikimedia.org/T435282) [06:06:10] FIRING: BFDdown: BFD session down between cr1-codfw and fe80::26fc:4e07:1041:5917 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [06:11:10] RESOLVED: BFDdown: BFD session down between cr1-codfw and fe80::26fc:4e07:1041:5917 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [06:19:06] (03PS1) 10Kosta Harlan: CheckUser: Enable ClientHintsInEditRequest on group0 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345854 (https://phabricator.wikimedia.org/T401808) [06:19:37] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344405 (https://phabricator.wikimedia.org/T436480) (owner: 10Southparkfan) [06:19:39] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345854 (https://phabricator.wikimedia.org/T401808) (owner: 10Kosta Harlan) [06:29:01] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [06:39:01] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:45:05] (03PS1) 10Jelto: etherpad: update image to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345857 (https://phabricator.wikimedia.org/T438866) [06:48:30] Deploying cxserver. [06:49:18] (03CR) 10KartikMistry: [C:03+2] Update cxserver to 2026-09-25-114231-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345697 (owner: 10KartikMistry) [06:49:49] (03CR) 10Jelto: [C:03+2] etherpad: update image to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345857 (https://phabricator.wikimedia.org/T438866) (owner: 10Jelto) [06:51:57] (03PS2) 10Matthias Mullie: Cache extracted carousel data [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345632 (https://phabricator.wikimedia.org/T439182) [06:52:05] (03Merged) 10jenkins-bot: Update cxserver to 2026-09-25-114231-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345697 (owner: 10KartikMistry) [06:52:18] (03Merged) 10jenkins-bot: etherpad: update image to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345857 (https://phabricator.wikimedia.org/T438866) (owner: 10Jelto) [06:55:11] (03CR) 10Brouberol: [C:03+1] aptrepo: import kafka-configurator from apt-staging [puppet] - 10https://gerrit.wikimedia.org/r/1345852 (https://phabricator.wikimedia.org/T435282) (owner: 10Ryan Kemper) [06:56:52] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply [06:57:31] !log jelto@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply [07:00:05] Amir1, urbanecm, and awight: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) UTC morning backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T0700). [07:00:05] Matthias and kostajh: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:00:06] o/ [07:00:21] I'll get started [07:00:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345626 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [07:00:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345632 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [07:00:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345627 (https://phabricator.wikimedia.org/T439422) (owner: 10Matthias Mullie) [07:00:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345633 (https://phabricator.wikimedia.org/T439422) (owner: 10Matthias Mullie) [07:01:39] !log kartik@deploy1003 helmfile [staging] START helmfile.d/services/cxserver: apply [07:02:00] !log kartik@deploy1003 helmfile [staging] DONE helmfile.d/services/cxserver: apply [07:03:52] (03Merged) 10jenkins-bot: Skip expensive work if known to be useless [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345626 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [07:03:54] (03Merged) 10jenkins-bot: Cache extracted carousel data [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345632 (https://phabricator.wikimedia.org/T439182) (owner: 10Matthias Mullie) [07:04:03] (03Merged) 10jenkins-bot: Make carousel images are visible in the margins [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345627 (https://phabricator.wikimedia.org/T439422) (owner: 10Matthias Mullie) [07:04:04] (03Merged) 10jenkins-bot: carousel: fix carousel image preload observer root [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345633 (https://phabricator.wikimedia.org/T439422) (owner: 10Matthias Mullie) [07:04:49] !log mlitn@deploy1003 Started scap sync-world: Backport for [[gerrit:1345626|Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632|Cache extracted carousel data (T439182)]], [[gerrit:1345627|Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633|carousel: fix carousel image preload observer root (T439422)]] [07:04:54] T439182: [Regression] MediaViewer hook adds 30% to Backend Pageview Timing on mobile (300 -> 400ms) - https://phabricator.wikimedia.org/T439182 [07:04:54] T439422: [Image Browsing] Carousel images should extend beyond content margins - https://phabricator.wikimedia.org/T439422 [07:06:33] PROBLEM - OSPF status on cr4-ulsfo is CRITICAL: OSPFv2: 2/4 UP : OSPFv3: 2/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [07:07:03] matthiasmullie: can you ping me when you're finished please? [07:07:03] PROBLEM - OSPF status on cr2-eqiad is CRITICAL: OSPFv2: 8/9 UP : OSPFv3: 8/9 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [07:07:09] kostajh: will do! [07:07:29] PROBLEM - OSPF status on cr2-codfw is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [07:07:39] FIRING: CoreBGPDown: Core BGP session down between cr2-codfw and cr4-ulsfo (198.35.26.157) - group Confed_ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr2-codfw:9804&var-bgp_group=Confed_ulsfo&var-bgp_neighbor=cr4-ulsfo - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [07:08:10] FIRING: [4x] BFDdown: BFD session down between cr2-codfw and 198.35.26.157 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [07:09:06] !log mlitn@deploy1003 mlitn: Backport for [[gerrit:1345626|Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632|Cache extracted carousel data (T439182)]], [[gerrit:1345627|Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633|carousel: fix carousel image preload observer root (T439422)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now [07:09:06] be verified there. [07:11:08] !log mlitn@deploy1003 mlitn: Continuing with deployment [07:12:39] FIRING: [3x] CoreBGPDown: Core BGP session down between cr2-codfw and cr4-ulsfo (198.35.26.157) - group Confed_ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [07:14:50] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host idp-test2005.wikimedia.org [07:18:06] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12372676 (10fgiunchedi) I set jbod for cloudcephosd1055 via the mgmt web interface, and tried cephosd-efi recipe, which prompted me for a partitioning choi... [07:18:28] !log mlitn@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345626|Skip expensive work if known to be useless (T439182)]], [[gerrit:1345632|Cache extracted carousel data (T439182)]], [[gerrit:1345627|Make carousel images are visible in the margins (T439422)]], [[gerrit:1345633|carousel: fix carousel image preload observer root (T439422)]] (duration: 13m 39s) [07:18:33] T439182: [Regression] MediaViewer hook adds 30% to Backend Pageview Timing on mobile (300 -> 400ms) - https://phabricator.wikimedia.org/T439182 [07:18:33] T439422: [Image Browsing] Carousel images should extend beyond content margins - https://phabricator.wikimedia.org/T439422 [07:18:38] kostajh: the floor is yours [07:18:48] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2005.wikimedia.org [07:19:01] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:21:30] matthiasmullie: thx! [07:21:50] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345854 (https://phabricator.wikimedia.org/T401808) (owner: 10Kosta Harlan) [07:22:36] (03Merged) 10jenkins-bot: CheckUser: Enable ClientHintsInEditRequest on group0 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345854 (https://phabricator.wikimedia.org/T401808) (owner: 10Kosta Harlan) [07:22:45] !log kartik@deploy1003 helmfile [codfw] START helmfile.d/services/cxserver: apply [07:22:58] !log kharlan@deploy1003 Started scap sync-world: Backport for [[gerrit:1345854|CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] [07:23:01] T401808: Allow Client Hints data to be used to prevent the save of edits - https://phabricator.wikimedia.org/T401808 [07:23:19] !log kartik@deploy1003 helmfile [codfw] DONE helmfile.d/services/cxserver: apply [07:24:17] !log kartik@deploy1003 helmfile [eqiad] START helmfile.d/services/cxserver: apply [07:24:47] !log kartik@deploy1003 helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [07:26:00] !log Updated cxserver to 2026-09-25-114231-production [07:26:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:27:10] !log kharlan@deploy1003 kharlan: Backport for [[gerrit:1345854|CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:27:57] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host sretest2004.codfw.wmnet [07:30:23] !log kharlan@deploy1003 kharlan: Continuing with deployment [07:31:35] (03PS1) 10Matthias Mullie: carousel: keep the hide button attached to the carousel [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345862 (https://phabricator.wikimedia.org/T439507) [07:31:47] (03PS1) 10Matthias Mullie: carousel: keep the hide button attached to the carousel [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1345863 (https://phabricator.wikimedia.org/T439507) [07:33:44] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest2004.codfw.wmnet [07:34:10] kostajh: can you ping me when you're done? I'll do another :p [07:35:39] !log kharlan@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345854|CheckUser: Enable ClientHintsInEditRequest on group0 (T401808)]] (duration: 12m 41s) [07:35:43] T401808: Allow Client Hints data to be used to prevent the save of edits - https://phabricator.wikimedia.org/T401808 [07:37:47] matthiasmullie: sure, I'm done [07:38:00] aight thanks [07:38:05] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345862 (https://phabricator.wikimedia.org/T439507) (owner: 10Matthias Mullie) [07:38:05] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1345863 (https://phabricator.wikimedia.org/T439507) (owner: 10Matthias Mullie) [07:39:07] (03Merged) 10jenkins-bot: carousel: keep the hide button attached to the carousel [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345862 (https://phabricator.wikimedia.org/T439507) (owner: 10Matthias Mullie) [07:39:19] (03Merged) 10jenkins-bot: carousel: keep the hide button attached to the carousel [extensions/MultimediaViewer] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1345863 (https://phabricator.wikimedia.org/T439507) (owner: 10Matthias Mullie) [07:39:42] !log mlitn@deploy1003 Started scap sync-world: Backport for [[gerrit:1345862|carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863|carousel: keep the hide button attached to the carousel (T439507)]] [07:39:46] T439507: Image Browsing: keep the carousel hide button attached to the carousel - https://phabricator.wikimedia.org/T439507 [07:41:07] (03Abandoned) 10Fabfur: external_clouds_vendors: minor logging fix [puppet] - 10https://gerrit.wikimedia.org/r/1345850 (owner: 10Fabfur) [07:43:51] !log mlitn@deploy1003 mlitn: Backport for [[gerrit:1345862|carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863|carousel: keep the hide button attached to the carousel (T439507)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:44:22] !log mlitn@deploy1003 mlitn: Continuing with deployment [07:49:40] !log mlitn@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345862|carousel: keep the hide button attached to the carousel (T439507)]], [[gerrit:1345863|carousel: keep the hide button attached to the carousel (T439507)]] (duration: 09m 57s) [07:49:43] T439507: Image Browsing: keep the carousel hide button attached to the carousel - https://phabricator.wikimedia.org/T439507 [07:49:50] !log UTC morning backport window done [07:49:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:52:30] (03CR) 10Ilias Sarantopoulos: ml-services: expose gpt-oss-safeguard-20b publicly (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345562 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [07:58:01] (03CR) 10Ozge: [C:03+1] ml-services: expose gpt-oss-safeguard-20b publicly [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345562 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [07:58:55] (03CR) 10Ozge: [C:03+1] "cool, let's test if this will work." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345562 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [07:59:15] (03CR) 10Ozge: [C:03+1] ml-services: expose gpt-oss-safeguard-20b publicly (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345562 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [08:03:43] (03PS1) 10Muehlenhoff: thumbor-plugins: Rebuild against latest package versions in Trixie [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1345920 [08:08:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker2269:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker2269 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [08:10:54] (03PS1) 10Slyngshede: P:IDP Tomcat 11 for CAS 8.X [puppet] - 10https://gerrit.wikimedia.org/r/1345921 (https://phabricator.wikimedia.org/T439509) [08:10:56] !log installing PHP 8.4 security updates [08:10:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:14:03] 06SRE, 06Traffic: Prometheus Ferm MSS export does not work for IPv6 - https://phabricator.wikimedia.org/T433672#12372837 (10cmooney) 05Open→03Resolved a:03cmooney Thanks @CDobbins for getting to the bottom of this! [08:15:11] (03CR) 10Ilias Sarantopoulos: [C:03+2] ml-services: expose gpt-oss-safeguard-20b publicly (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345562 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [08:15:19] (03PS1) 10Slyngshede: P:tofurkey enable Tofurkey for DRMRS [puppet] - 10https://gerrit.wikimedia.org/r/1345923 (https://phabricator.wikimedia.org/T355446) [08:17:36] (03Merged) 10jenkins-bot: ml-services: expose gpt-oss-safeguard-20b publicly [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345562 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [08:18:57] !log isaranto@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [08:21:06] (03CR) 10Slyngshede: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345923 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [08:21:26] (03CR) 10Slyngshede: "Enable Tofurkey on DRMRS before enabling everywhere." [puppet] - 10https://gerrit.wikimedia.org/r/1345923 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [08:21:46] (03CR) 10Muehlenhoff: [C:03+2] Remove cumin1003 from the alertmanager and tcpircbot configs [puppet] - 10https://gerrit.wikimedia.org/r/1345087 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [08:26:28] (03CR) 10Muehlenhoff: [C:03+1] "My spot checks were also fine and PCC looks also good (the only puzzling thing is that mx1001/2001 are still present despite being long go" [puppet] - 10https://gerrit.wikimedia.org/r/1345157 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [08:26:37] (03CR) 10Cathal Mooney: [C:03+1] "LGTM!" [homer/public] - 10https://gerrit.wikimedia.org/r/1344682 (https://phabricator.wikimedia.org/T439136) (owner: 10Ayounsi) [08:28:14] (03CR) 101Veertje: "I've minimised the changes to the LoginForm specific for the WM install" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 (owner: 101Veertje) [08:37:24] (03PS1) 10Slyngshede: P:idp add missing dummy secret [labs/private] - 10https://gerrit.wikimedia.org/r/1345925 [08:38:28] (03CR) 10Slyngshede: [V:03+2 C:03+2] P:idp add missing dummy secret [labs/private] - 10https://gerrit.wikimedia.org/r/1345925 (owner: 10Slyngshede) [08:39:01] FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_krb5-admin-server.service on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:39:36] !log ayounsi@cumin1004 START - Cookbook sre.network.tls for network device lsw1-f7-eqiad [08:39:43] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f7-eqiad [08:39:46] (03CR) 10Slyngshede: [V:03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9485/console" [puppet] - 10https://gerrit.wikimedia.org/r/1345921 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [08:40:15] jouncebot: nowandnext [08:40:15] No deployments scheduled for the next 1 hour(s) and 19 minute(s) [08:40:15] In 1 hour(s) and 19 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1000) [08:40:31] (03CR) 10Urbanecm: [C:03+2] ReassignMentees: Actually deduplicate reassignMenteesJob [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344712 (https://phabricator.wikimedia.org/T418194) (owner: 10Urbanecm) [08:40:38] !log ayounsi@cumin1004 START - Cookbook sre.network.tls for network device lsw1-e6-eqiad [08:40:44] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e6-eqiad [08:40:53] !log ayounsi@cumin1004 START - Cookbook sre.network.tls for network device lsw1-e7-eqiad [08:40:59] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e7-eqiad [08:41:08] !log ayounsi@cumin1004 START - Cookbook sre.network.tls for network device lsw1-f6-eqiad [08:41:16] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f6-eqiad [08:41:23] !log ayounsi@cumin1004 START - Cookbook sre.network.tls for network device lsw1-f5-eqiad [08:41:30] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f5-eqiad [08:41:54] !log ayounsi@cumin1004 START - Cookbook sre.network.tls for network device lsw1-e5-eqiad [08:42:01] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e5-eqiad [08:42:52] (03CR) 10Ayounsi: [C:03+2] Remove "hostkey_algorithm" in all sites except eqiad/codfw [homer/public] - 10https://gerrit.wikimedia.org/r/1344682 (https://phabricator.wikimedia.org/T439136) (owner: 10Ayounsi) [08:44:15] (03Merged) 10jenkins-bot: Remove "hostkey_algorithm" in all sites except eqiad/codfw [homer/public] - 10https://gerrit.wikimedia.org/r/1344682 (https://phabricator.wikimedia.org/T439136) (owner: 10Ayounsi) [08:45:08] (03CR) 10A-pizzata: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343954 (https://phabricator.wikimedia.org/T437961) (owner: 10A-pizzata) [08:46:34] (03PS1) 10Dpogorzelski: liftwing-studio: rebuild images on the 20260927 bases [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345926 [08:47:46] (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344712 (https://phabricator.wikimedia.org/T418194) (owner: 10Urbanecm) [08:47:59] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: rebuild images on the 20260927 bases [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345926 (owner: 10Dpogorzelski) [08:48:04] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] liftwing-studio: rebuild images on the 20260927 bases [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345926 (owner: 10Dpogorzelski) [08:48:16] (03Merged) 10jenkins-bot: ReassignMentees: Actually deduplicate reassignMenteesJob [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344712 (https://phabricator.wikimedia.org/T418194) (owner: 10Urbanecm) [08:48:35] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1344712|ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] [08:48:38] T418194: Mentors still having mentees after removing themselves - https://phabricator.wikimedia.org/T418194 [08:49:16] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [08:50:13] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [08:52:47] !log urbanecm@deploy1003 urbanecm: Backport for [[gerrit:1344712|ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:53:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker2269:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker2269 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [08:54:19] !log urbanecm@deploy1003 urbanecm: Continuing with deployment [08:57:55] (03CR) 10Arnaudb: "Comparing the two charts it seems that the major two differences are the Recreate strategy and the ConfigMap for Aphlict, which would requ" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341856 (https://phabricator.wikimedia.org/T436657) (owner: 10Arnaudb) [08:59:43] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344712|ReassignMentees: Actually deduplicate reassignMenteesJob (T418194)]] (duration: 11m 07s) [08:59:46] T418194: Mentors still having mentees after removing themselves - https://phabricator.wikimedia.org/T418194 [09:04:15] (03PS1) 10Muehlenhoff: Replace grant for cumin1003 with cumin1004 [puppet] - 10https://gerrit.wikimedia.org/r/1345931 (https://phabricator.wikimedia.org/T427897) [09:04:26] (03PS1) 10Cathal Mooney: Add reverse zone delegations for new dse-k8s-eqiad range 10.68.0.0/17 [dns] - 10https://gerrit.wikimedia.org/r/1345933 (https://phabricator.wikimedia.org/T430658) [09:05:37] (03CR) 10CI reject: [V:04-1] Add reverse zone delegations for new dse-k8s-eqiad range 10.68.0.0/17 [dns] - 10https://gerrit.wikimedia.org/r/1345933 (https://phabricator.wikimedia.org/T430658) (owner: 10Cathal Mooney) [09:06:03] (03CR) 10CWilliams: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1345931 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [09:06:41] (03PS1) 10Ayounsi: Exclude ae interfaces from CoreOutboundSaturation alert [alerts] - 10https://gerrit.wikimedia.org/r/1345934 (https://phabricator.wikimedia.org/T439202) [09:06:49] (03CR) 10CI reject: [V:04-1] Exclude ae interfaces from CoreOutboundSaturation alert [alerts] - 10https://gerrit.wikimedia.org/r/1345934 (https://phabricator.wikimedia.org/T439202) (owner: 10Ayounsi) [09:07:57] (03Abandoned) 10Cathal Mooney: Add reverse zone delegations for new dse-k8s-eqiad range 10.68.0.0/17 [dns] - 10https://gerrit.wikimedia.org/r/1345933 (https://phabricator.wikimedia.org/T430658) (owner: 10Cathal Mooney) [09:08:49] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 359403696 and 47 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [09:09:34] (03CR) 10Ayounsi: "recheck" [alerts] - 10https://gerrit.wikimedia.org/r/1345934 (https://phabricator.wikimedia.org/T439202) (owner: 10Ayounsi) [09:09:47] 06SRE, 06Infrastructure-Foundations: Sunset rootless Cumin - https://phabricator.wikimedia.org/T439522 (10MoritzMuehlenhoff) 03NEW [09:10:32] (03CR) 10Muehlenhoff: [C:03+2] Replace grant for cumin1003 with cumin1004 [puppet] - 10https://gerrit.wikimedia.org/r/1345931 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [09:10:33] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade Cumin hosts to Trixie - https://phabricator.wikimedia.org/T427897#12373070 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff All core Cumin servers are on Trixie. The cloucumin hosts are tracked via T436667 and cuminunpri... [09:11:50] (03CR) 10Ayounsi: "CI error seems unrelated to the change itself." [alerts] - 10https://gerrit.wikimedia.org/r/1345934 (https://phabricator.wikimedia.org/T439202) (owner: 10Ayounsi) [09:15:02] (03CR) 10Cathal Mooney: [C:03+1] "indeed yeah." [alerts] - 10https://gerrit.wikimedia.org/r/1345934 (https://phabricator.wikimedia.org/T439202) (owner: 10Ayounsi) [09:18:18] (03CR) 10Tiziano Fogli: "recheck" [alerts] - 10https://gerrit.wikimedia.org/r/1345934 (https://phabricator.wikimedia.org/T439202) (owner: 10Ayounsi) [09:19:15] (03PS1) 10Dreamy Jazz: maintenance: Set FOREACHWIKI_IGNORE_ERRORS for some PSI jobs [puppet] - 10https://gerrit.wikimedia.org/r/1345937 (https://phabricator.wikimedia.org/T432110) [09:20:49] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 11528 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [09:22:49] (03CR) 10Ayounsi: [C:03+2] Exclude ae interfaces from CoreOutboundSaturation alert [alerts] - 10https://gerrit.wikimedia.org/r/1345934 (https://phabricator.wikimedia.org/T439202) (owner: 10Ayounsi) [09:23:05] (03PS2) 10Dreamy Jazz: maintenance: Set FOREACHWIKI_IGNORE_ERRORS for some PSI jobs [puppet] - 10https://gerrit.wikimedia.org/r/1345937 (https://phabricator.wikimedia.org/T432110) [09:23:14] (03CR) 10Dreamy Jazz: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345937 (https://phabricator.wikimedia.org/T432110) (owner: 10Dreamy Jazz) [09:23:46] jouncebot: next [09:23:46] In 0 hour(s) and 36 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1000) [09:24:28] (03Merged) 10jenkins-bot: Exclude ae interfaces from CoreOutboundSaturation alert [alerts] - 10https://gerrit.wikimedia.org/r/1345934 (https://phabricator.wikimedia.org/T439202) (owner: 10Ayounsi) [09:27:45] (03PS3) 10Dreamy Jazz: maintenance: Set FOREACHWIKI_IGNORE_ERRORS for some PSI jobs [puppet] - 10https://gerrit.wikimedia.org/r/1345937 (https://phabricator.wikimedia.org/T432110) [09:27:55] (03CR) 10Dreamy Jazz: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345937 (https://phabricator.wikimedia.org/T432110) (owner: 10Dreamy Jazz) [09:29:01] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [09:36:01] !log urbanecm@deploy1003 mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Ikan --performer=Ikan --as-job # T418194 [09:36:05] T418194: Mentors still having mentees after removing themselves - https://phabricator.wikimedia.org/T418194 [09:42:19] (03CR) 10Clément Goubert: [C:03+1] ProductionServices: Simplify indirect parsercache-dbs stub [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344298 (https://phabricator.wikimedia.org/T362786) (owner: 10Krinkle) [09:45:47] (03PS1) 10Daniel Kertesz: Remove obsolete busted container image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345940 (https://phabricator.wikimedia.org/T437442) [09:47:00] (03CR) 10Giuseppe Lavagetto: [C:03+1] Remove obsolete busted container image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345940 (https://phabricator.wikimedia.org/T437442) (owner: 10Daniel Kertesz) [09:49:25] PROBLEM - ganeti-wconfd running on ganeti6004 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 110 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [09:49:25] PROBLEM - ganeti-noded running on ganeti6004 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 0 (root), command name ganeti-noded https://wikitech.wikimedia.org/wiki/Ganeti [09:49:25] PROBLEM - HTTPS Ganeti RAPI drmrs on ganeti6004 is CRITICAL: connect to address ganeti02.svc.drmrs.wmnet and port 5080: Connection refused https://www.mediawiki.org/wiki/Ganeti%23RAPI_daemon [09:49:27] PROBLEM - ganeti-confd running on ganeti6004 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 109 (gnt-confd), command name ganeti-confd https://wikitech.wikimedia.org/wiki/Ganeti [09:49:38] (03PS2) 10A-pizzata: sqoop_mediawiki: Add weekly change_tag sqoop [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [09:50:25] RECOVERY - ganeti-wconfd running on ganeti6004 is OK: PROCS OK: 1 process with UID = 110 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [09:50:25] RECOVERY - ganeti-noded running on ganeti6004 is OK: PROCS OK: 1 process with UID = 0 (root), command name ganeti-noded https://wikitech.wikimedia.org/wiki/Ganeti [09:50:27] RECOVERY - HTTPS Ganeti RAPI drmrs on ganeti6004 is OK: HTTP OK: Status line output matched 401 - 308 bytes in 0.022 second response time https://www.mediawiki.org/wiki/Ganeti%23RAPI_daemon [09:50:27] RECOVERY - ganeti-confd running on ganeti6004 is OK: PROCS OK: 1 process with UID = 109 (gnt-confd), command name ganeti-confd https://wikitech.wikimedia.org/wiki/Ganeti [09:51:54] 07sre-alert-triage, 06Infrastructure-Foundations: Alert in need of triage: GanetiCACertificateAboutToExpire (instance ganeti6002:9100) - https://phabricator.wikimedia.org/T439379#12373272 (10MoritzMuehlenhoff) The built-in renew-crypto command runs commands from the ganeti master to the other nodes, which runs... [09:52:24] (03CR) 10A-pizzata: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [09:52:47] 06SRE, 06Collaboration-Services, 06Infrastructure-Foundations, 06Release-Engineering-Team: Builds fail on integration-agent-docker-1086 due to expired snapshot.debian.org Release file - https://phabricator.wikimedia.org/T439529 (10tappof) 03NEW [09:53:16] 06SRE, 06Collaboration-Services, 06Infrastructure-Foundations, 06Release-Engineering-Team: Builds fail on integration-agent-docker-1086 due to expired snapshot.debian.org Release file - https://phabricator.wikimedia.org/T439529#12373300 (10tappof) [09:59:54] (03CR) 10Clément Goubert: [C:03+1] "The fixture LGTM." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333837 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:00:04] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1000) [10:00:55] jouncebot: nowandnext [10:00:55] For the next 0 hour(s) and 59 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1000) [10:00:55] In 1 hour(s) and 59 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1200) [10:01:07] (03PS1) 10Filippo Giunchedi: installserver: use standard partitioning for cloudcephosd > 1053 [puppet] - 10https://gerrit.wikimedia.org/r/1345942 (https://phabricator.wikimedia.org/T419892) [10:02:25] (03PS8) 10Blake: sidecars: enable a restartPolicy: Always option. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333837 (https://phabricator.wikimedia.org/T417800) [10:04:17] (03PS3) 10Samtar: InitialiseSettings: Enable ModeratorToolkit on test.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343957 (https://phabricator.wikimedia.org/T438334) [10:04:27] (03PS3) 10Samtar: CommonSettings: Load ModeratorToolkit extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343958 (https://phabricator.wikimedia.org/T438334) [10:04:59] (03CR) 10Isabelle Hurbain-Palatin: [C:03+2] mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345551 (owner: 10PipelineBot) [10:07:49] (03Merged) 10jenkins-bot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345551 (owner: 10PipelineBot) [10:09:26] !log ihurbain@deploy1003 helmfile [staging] START helmfile.d/services/mobileapps: apply [10:10:03] !log jmm@cumin2003 START - Cookbook sre.ganeti.makevm for new host testvm2001.codfw.wmnet [10:10:06] !log jmm@cumin2003 START - Cookbook sre.dns.netbox [10:10:29] !log ihurbain@deploy1003 helmfile [staging] DONE helmfile.d/services/mobileapps: apply [10:11:32] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware), 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12373389 (10fgiunchedi) >>! In T419892#12372676, @fgiunchedi wrote: > I then looked at cloudcephosd and I don't see any reason why th... [10:13:54] (03CR) 10A-pizzata: sqoop_mediawiki: Add weekly change_tag sqoop (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [10:14:26] !log jmm@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003" [10:14:30] !log jmm@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2001.codfw.wmnet - jmm@cumin2003" [10:14:30] !log jmm@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [10:14:31] !log jmm@cumin2003 START - Cookbook sre.dns.wipe-cache testvm2001.codfw.wmnet on all recursors [10:14:34] !log jmm@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2001.codfw.wmnet on all recursors [10:14:56] !log jmm@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003" [10:15:00] !log jmm@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2001.codfw.wmnet - jmm@cumin2003" [10:15:06] !log jmm@cumin2003 START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie [10:15:09] RECOVERY - Postfix SMTP on crm2001 is OK: OK - Certificate crm2001.codfw.wmnet will expire on Tue 27 Oct 2026 09:40:00 AM GMT +0000. https://wikitech.wikimedia.org/wiki/Mail%23Troubleshooting [10:17:21] (03CR) 10Daniel Kertesz: [C:03+2] Remove obsolete busted container image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345940 (https://phabricator.wikimedia.org/T437442) (owner: 10Daniel Kertesz) [10:17:28] !log deploy2003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # T438232 [10:17:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:17:31] T438232: Request to increase the size of /srv on production deployment servers - https://phabricator.wikimedia.org/T438232 [10:17:53] !log deploy1003:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # T438232 [10:17:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:18:16] (03CR) 10Daniel Kertesz: [V:03+2 C:03+2] Remove obsolete busted container image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345940 (https://phabricator.wikimedia.org/T437442) (owner: 10Daniel Kertesz) [10:19:19] !log deploy2002:~$ sudo lvextend -L +50GB --resizefs /dev/vg0/srv # T438232 [10:19:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:21:23] (03CR) 10A-pizzata: sqoop_mediawiki: Add weekly change_tag sqoop (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [10:22:51] 06SRE, 06Traffic, 13Patch-For-Review: Migrated busted container image to Trixie - https://phabricator.wikimedia.org/T437442#12373489 (10dkertesz) I have verified with @Joe and [[ https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/production-images/+/fc9d7b7af374288a0070781f68014fd8647940... [10:22:57] (03PS1) 10Isabelle Hurbain-Palatin: Revert "mobileapps: pipeline bot promote" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345949 [10:23:06] 06SRE, 06Traffic, 13Patch-For-Review: Migrated busted container image to Trixie - https://phabricator.wikimedia.org/T437442#12373502 (10dkertesz) a:03dkertesz [10:23:09] (03CR) 10Isabelle Hurbain-Palatin: [C:03+2] Revert "mobileapps: pipeline bot promote" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345949 (owner: 10Isabelle Hurbain-Palatin) [10:24:37] (03Merged) 10jenkins-bot: Revert "mobileapps: pipeline bot promote" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345949 (owner: 10Isabelle Hurbain-Palatin) [10:27:28] (03CR) 10Blake: [C:03+2] sidecars: enable a restartPolicy: Always option. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333837 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:28:23] will be running a helmfile-only deploy momentarily for ^ [10:29:01] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [10:32:20] (03Merged) 10jenkins-bot: sidecars: enable a restartPolicy: Always option. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333837 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:32:33] !log jmm@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage [10:33:33] !log blake@deploy1003 Started scap sync-world: add sidecar flag for T417800 [10:33:36] T417800: Implement proper sidecar container support in mediawiki pods - https://phabricator.wikimedia.org/T417800 [10:37:14] !log blake@deploy1003 Finished scap sync-world: add sidecar flag for T417800 (duration: 04m 14s) [10:38:48] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage [10:40:17] 06SRE, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh configuration - https://phabricator.wikimedia.org/T439535 (10cmooney) 03NEW p:05Triage→03Medium [10:40:57] (03CR) 10Atsuko: [C:03+2] airflow-test-k8s: upgrade to airflow 3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345077 (https://phabricator.wikimedia.org/T433383) (owner: 10Atsuko) [10:41:26] 06SRE, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh configuration - https://phabricator.wikimedia.org/T439535#12373591 (10cmooney) [10:41:30] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12373592 (10cmooney) [10:43:43] !log shutting down https://airflow-test-k8s.wikimedia.org for upgrade, https://phabricator.wikimedia.org/T433383#12363530 [10:43:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:43:59] (03Merged) 10jenkins-bot: airflow-test-k8s: upgrade to airflow 3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345077 (https://phabricator.wikimedia.org/T433383) (owner: 10Atsuko) [10:44:18] (03CR) 10Gkyziridis: "I am not very familiar with the puppet repository, but I think we are missing some points in this patch:" [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) (owner: 10Ozge) [10:46:28] 06SRE, 06Infrastructure-Foundations, 10netops: Support channelized / breakout interfaces on SR-Linux - https://phabricator.wikimedia.org/T439537 (10cmooney) 03NEW p:05Triage→03Medium [10:46:37] PROBLEM - Check unit status of statograph_post on alert1002 is CRITICAL: CRITICAL: Status of the systemd unit statograph_post https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [10:46:47] !log Rebalancing has been completed on kafka-logging in eqiad: kafka-logging1001 and kafka-logging1002 are no longer assigned any partitions — T432444 [10:46:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:46:52] T432444: Provision kafka-logging100[6-8] and kafka-logging200[6-8] - https://phabricator.wikimedia.org/T432444 [10:47:20] 06SRE, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh configuration - https://phabricator.wikimedia.org/T439535#12373630 (10cmooney) [10:47:21] 06SRE, 06Infrastructure-Foundations, 10netops: Support channelized / breakout interfaces on SR-Linux - https://phabricator.wikimedia.org/T439537#12373629 (10cmooney) [10:47:25] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [10:49:38] (03PS1) 10Michael Große: fix: prevent AccountSetup from being scrollable [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345952 (https://phabricator.wikimedia.org/T436872) [10:50:10] (03PS1) 10Michael Große: fix: prevent AccountSetup from being scrollable [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1345953 (https://phabricator.wikimedia.org/T436872) [10:54:01] FIRING: KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1152.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:55:42] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [10:55:46] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [10:56:24] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [10:56:33] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [10:56:37] RECOVERY - Check unit status of statograph_post on alert1002 is OK: OK: Status of the systemd unit statograph_post https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [10:56:37] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [10:59:49] FIRING: HelmReleaseBadStatus: Helm release airflow-test-k8s/production on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=airflow-test-k8s - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:03:14] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345952 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große) [11:03:48] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1345953 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große) [11:08:10] FIRING: [4x] BFDdown: BFD session down between cr2-codfw and 198.35.26.157 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:09:49] RESOLVED: HelmReleaseBadStatus: Helm release airflow-test-k8s/production on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=airflow-test-k8s - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:09:53] (03PS8) 10Muehlenhoff: profile::zookeeper::firewall: Also allow passing a list of hosts [puppet] - 10https://gerrit.wikimedia.org/r/1272766 [11:12:39] FIRING: [3x] CoreBGPDown: Core BGP session down between cr2-codfw and cr4-ulsfo (198.35.26.157) - group Confed_ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [11:21:07] Hii, I need help debugging idp-test with the new service code, https://airflow-test-k8s.wikimedia.org/auth/login/?next=%2F [11:21:25] (wrong chat, will go to -sre) [11:23:13] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342962 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [11:27:25] 07sre-alert-triage, 06Infrastructure-Foundations: Alert in need of triage: GanetiCACertificateAboutToExpire (instance ganeti6002:9100) - https://phabricator.wikimedia.org/T439379#12373809 (10MoritzMuehlenhoff) 05Open→03Resolved The Ganeti cluster CA cert for drmrs02 has been extended: ` jmm@ganeti6004... [11:28:08] (03PS3) 10Aqu: sqoop_mediawiki: Add weekly change_tag sqoop [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) [11:29:44] (03CR) 10Aqu: sqoop_mediawiki: Add weekly change_tag sqoop (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [11:30:10] (03PS1) 10Daniel Kertesz: profile::cache::base: reserve ports used by local daemons [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) [11:31:58] (03CR) 10CI reject: [V:04-1] profile::cache::base: reserve ports used by local daemons [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) (owner: 10Daniel Kertesz) [11:34:40] (03PS2) 10Daniel Kertesz: profile::cache::base: reserve ports used by local daemons [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) [11:37:04] (03PS1) 10Muehlenhoff: Open a Phab task if Ganeti CA expires soon [alerts] - 10https://gerrit.wikimedia.org/r/1345966 (https://phabricator.wikimedia.org/T439440) [11:40:09] (03CR) 10CI reject: [V:04-1] Open a Phab task if Ganeti CA expires soon [alerts] - 10https://gerrit.wikimedia.org/r/1345966 (https://phabricator.wikimedia.org/T439440) (owner: 10Muehlenhoff) [11:41:41] (03PS2) 10Muehlenhoff: Open a Phab task if Ganeti CA expires soon [alerts] - 10https://gerrit.wikimedia.org/r/1345966 (https://phabricator.wikimedia.org/T439440) [11:42:03] (03CR) 10CI reject: [V:04-1] Open a Phab task if Ganeti CA expires soon [alerts] - 10https://gerrit.wikimedia.org/r/1345966 (https://phabricator.wikimedia.org/T439440) (owner: 10Muehlenhoff) [11:48:36] (03CR) 10Vgutierrez: profile::cache::base: reserve ports used by local daemons (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) (owner: 10Daniel Kertesz) [11:52:59] (03PS1) 10Btullis: puppetserver: Permit the new dse-k8s-eqiad Pod range to rsync [puppet] - 10https://gerrit.wikimedia.org/r/1345970 (https://phabricator.wikimedia.org/T429773) [11:53:02] (03PS1) 10Btullis: dse-k8s-eqiad: Retire the old Pod IP range [puppet] - 10https://gerrit.wikimedia.org/r/1345971 (https://phabricator.wikimedia.org/T430658) [11:55:15] 06SRE, 06Collaboration-Services, 06Release-Engineering-Team: Builds fail on integration-agent-docker-1086 due to expired snapshot.debian.org Release file - https://phabricator.wikimedia.org/T439529#12373901 (10MoritzMuehlenhoff) [11:55:39] (03CR) 10Muehlenhoff: "(PCC failure is unrelated, caused by https://phabricator.wikimedia.org/T439529)" [alerts] - 10https://gerrit.wikimedia.org/r/1345966 (https://phabricator.wikimedia.org/T439440) (owner: 10Muehlenhoff) [11:56:18] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345970 (https://phabricator.wikimedia.org/T429773) (owner: 10Btullis) [11:56:37] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1272766 (owner: 10Muehlenhoff) [11:57:49] (03PS1) 10Btullis: dse-k8s-eqiad: Remove the old Pod IP range [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345973 (https://phabricator.wikimedia.org/T430658) [11:58:11] (03PS1) 10Btullis: dse-k8s: Remove the reverse DNS delegations for the old Pod range [dns] - 10https://gerrit.wikimedia.org/r/1345974 (https://phabricator.wikimedia.org/T430658) [12:00:05] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1200) [12:02:47] !log osleger@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [12:02:58] (03PS4) 10A-pizzata: sqoop_mediawiki: Add weekly change_tag sqoop [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [12:02:59] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [12:03:15] !log osleger@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [12:03:16] !log osleger@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [12:03:48] !log osleger@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [12:04:16] (03CR) 10A-pizzata: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [12:04:46] (03PS3) 10Daniel Kertesz: profile::cache::base: reserve ports used by local daemons [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) [12:06:02] btullis@cumin1004 provision (PID 1566121) is awaiting input [12:06:46] !log osleger@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [12:06:49] !log osleger@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [12:06:50] !log osleger@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [12:06:53] !log osleger@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [12:07:16] (03CR) 10Muehlenhoff: [C:03+2] profile::zookeeper::firewall: Also allow passing a list of hosts [puppet] - 10https://gerrit.wikimedia.org/r/1272766 (owner: 10Muehlenhoff) [12:11:25] (03CR) 10Klausman: [C:03+1] dse-k8s-eqiad: Retire the old Pod IP range [puppet] - 10https://gerrit.wikimedia.org/r/1345971 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [12:12:09] (03CR) 10Muehlenhoff: [C:03+2] Use the new MDB/Trixie servers for tests [puppet] - 10https://gerrit.wikimedia.org/r/1344656 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [12:13:18] btullis@cumin1004 provision (PID 1566121) is awaiting input [12:21:46] (03PS1) 10Urbanecm: ReassignMentees: Make the jobReleaseTimestamp work [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345986 (https://phabricator.wikimedia.org/T418194) [12:22:43] jouncebot: nowandnext [12:22:43] For the next 0 hour(s) and 37 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1200) [12:22:43] In 0 hour(s) and 37 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1300) [12:23:23] (03CR) 10Urbanecm: [C:03+2] ReassignMentees: Make the jobReleaseTimestamp work [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345986 (https://phabricator.wikimedia.org/T418194) (owner: 10Urbanecm) [12:23:56] (03CR) 10Filippo Giunchedi: [C:03+2] installserver: use standard partitioning for cloudcephosd > 1053 [puppet] - 10https://gerrit.wikimedia.org/r/1345942 (https://phabricator.wikimedia.org/T419892) (owner: 10Filippo Giunchedi) [12:26:17] (03CR) 10Klausman: [C:03+1] dse-k8s-eqiad: Remove the old Pod IP range [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345973 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [12:26:37] (03CR) 10Klausman: [C:03+1] dse-k8s: Remove the reverse DNS delegations for the old Pod range [dns] - 10https://gerrit.wikimedia.org/r/1345974 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [12:27:24] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [12:28:23] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [12:29:54] (03PS1) 10Ayounsi: network: refactor Junos BGP and add get_junos_bgp_details [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) [12:30:46] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [12:30:54] (03PS2) 10Muehlenhoff: idp-test: Configure to use the new Trixie/MDB LDAP servers for tests [puppet] - 10https://gerrit.wikimedia.org/r/1344665 (https://phabricator.wikimedia.org/T331699) [12:30:58] (03Merged) 10jenkins-bot: ReassignMentees: Make the jobReleaseTimestamp work [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345986 (https://phabricator.wikimedia.org/T418194) (owner: 10Urbanecm) [12:31:29] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [12:31:43] (03CR) 10Brouberol: [C:03+1] puppetserver: Permit the new dse-k8s-eqiad Pod range to rsync [puppet] - 10https://gerrit.wikimedia.org/r/1345970 (https://phabricator.wikimedia.org/T429773) (owner: 10Btullis) [12:32:14] (03CR) 10Brouberol: [C:03+1] dse-k8s-eqiad: Remove the old Pod IP range [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345973 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [12:32:30] (03CR) 10CI reject: [V:04-1] network: refactor Junos BGP and add get_junos_bgp_details [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [12:32:31] (03CR) 10Brouberol: [C:03+1] dse-k8s: Remove the reverse DNS delegations for the old Pod range [dns] - 10https://gerrit.wikimedia.org/r/1345974 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [12:32:46] (03CR) 10Arnaudb: [C:03+2] mesh: document the idle timeout key the templates actually read [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338960 (https://phabricator.wikimedia.org/T436657) (owner: 10Arnaudb) [12:32:50] (03CR) 10Brouberol: [C:03+1] dse-k8s-eqiad: Retire the old Pod IP range [puppet] - 10https://gerrit.wikimedia.org/r/1345971 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [12:34:05] getting an error on "Squashed backport of 1345436, 1345890, 1345969" being unexpected [12:34:12] weirdly, I don't see anything like that in Gerrit [12:34:18] (03Merged) 10jenkins-bot: mesh: document the idle timeout key the templates actually read [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338960 (https://phabricator.wikimedia.org/T436657) (owner: 10Arnaudb) [12:34:32] all recent on https://gerrit.wikimedia.org/r/q/branch:+wmf/1.47.0-wmf.21 is mine [12:35:55] it seems we had a direct push... [12:36:13] (03PS2) 10Ayounsi: network: refactor Junos BGP and add get_junos_bgp_details [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) [12:36:28] (03CR) 10Muehlenhoff: [C:03+2] idp-test: Configure to use the new Trixie/MDB LDAP servers for tests [puppet] - 10https://gerrit.wikimedia.org/r/1344665 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [12:36:39] the pusher is pushing a revert [12:38:30] (03CR) 10CI reject: [V:04-1] network: refactor Junos BGP and add get_junos_bgp_details [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [12:38:48] FIRING: PuppetFailure: Puppet has failed on testvm2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [12:39:01] FIRING: [3x] SystemdUnitFailed: wmf_auto_restart_krb5-admin-server.service on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:39:12] okay, checking again [12:39:41] !log filippo@cumin1004 START - Cookbook sre.hosts.reimage for host cloudcephosd1055.eqiad.wmnet with OS bookworm [12:39:59] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12374073 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1004 for host cloudcephosd1055.eqiad.wmnet with OS bookworm [12:40:06] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1345986|ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] [12:40:10] T418194: Mentors still having mentees after removing themselves - https://phabricator.wikimedia.org/T418194 [12:41:05] (03PS1) 10Isabelle Hurbain-Palatin: Add query parameters to the core html endpoint templates [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345994 (https://phabricator.wikimedia.org/T431693) [12:41:54] i see the revert, so i proceeded [12:42:03] filled T439548 [12:42:04] T439548: wmf-deployment members can push directly to deployment branches - https://phabricator.wikimedia.org/T439548 [12:43:06] (03PS3) 10Ayounsi: network: refactor Junos BGP and add get_junos_bgp_details [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) [12:43:27] (03PS2) 10Isabelle Hurbain-Palatin: Add query parameters to the core html endpoint templates [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345994 (https://phabricator.wikimedia.org/T431693) [12:44:16] !log urbanecm@deploy1003 urbanecm: Backport for [[gerrit:1345986|ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:46:03] 10ops-eqiad, 06SRE, 06DC-Ops, 06Machine-Learning-Team, and 2 others: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12374107 (10klausman) a:05klausman→03RobH [12:46:35] (03CR) 10Jgiannelos: [C:03+2] Add query parameters to the core html endpoint templates [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345994 (https://phabricator.wikimedia.org/T431693) (owner: 10Isabelle Hurbain-Palatin) [12:46:48] (03CR) 10CI reject: [V:04-1] network: refactor Junos BGP and add get_junos_bgp_details [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [12:47:11] !log urbanecm@deploy1003 urbanecm: Continuing with deployment [12:49:27] (03Merged) 10jenkins-bot: Add query parameters to the core html endpoint templates [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345994 (https://phabricator.wikimedia.org/T431693) (owner: 10Isabelle Hurbain-Palatin) [12:49:36] (03PS1) 10Isabelle Hurbain-Palatin: Revert^2 "mobileapps: pipeline bot promote" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345996 [12:50:10] (03CR) 10Isabelle Hurbain-Palatin: [C:03+2] Revert^2 "mobileapps: pipeline bot promote" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345996 (owner: 10Isabelle Hurbain-Palatin) [12:52:30] (03Merged) 10jenkins-bot: Revert^2 "mobileapps: pipeline bot promote" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345996 (owner: 10Isabelle Hurbain-Palatin) [12:52:33] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345986|ReassignMentees: Make the jobReleaseTimestamp work (T418194)]] (duration: 12m 26s) [12:52:36] T418194: Mentors still having mentees after removing themselves - https://phabricator.wikimedia.org/T418194 [12:53:37] (03PS4) 10Ayounsi: network: refactor Junos BGP and add get_junos_bgp_details [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) [12:53:38] !log klausman@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on ml-serve1015.eqiad.wmnet with reason: Downtime for more BIOS setup rummaging [12:54:05] !log ihurbain@deploy1003 helmfile [staging] START helmfile.d/services/mobileapps: apply [12:54:52] !log ihurbain@deploy1003 helmfile [staging] DONE helmfile.d/services/mobileapps: apply [12:55:48] (03CR) 10CI reject: [V:04-1] network: refactor Junos BGP and add get_junos_bgp_details [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [12:56:28] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345998 [12:56:32] (03PS1) 10Marco Fossati: Squashed backport of 1345436, 1345890, 1345969 [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345999 (https://phabricator.wikimedia.org/T439534) [12:56:33] (03CR) 10Daniel Kertesz: profile::cache::base: reserve ports used by local daemons (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) (owner: 10Daniel Kertesz) [12:57:02] (03PS5) 10Ayounsi: network: refactor Junos BGP and add get_junos_bgp_details [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) [12:57:09] filippo@cumin1004 reimage (PID 1570093) is awaiting input [12:58:19] (03PS9) 10Arnaudb: mesh: Copy mesh.configuration 1.16.0 to 1.16.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338959 (https://phabricator.wikimedia.org/T436657) [12:58:20] (03PS15) 10Arnaudb: mesh: add opt-in websocket support in configuration 1.16.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338751 (https://phabricator.wikimedia.org/T436657) [12:58:41] (03PS1) 10Cathal Mooney: Interface validator: support breakout port names on SR-Linux [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1346001 (https://phabricator.wikimedia.org/T439537) [12:59:07] (03CR) 10TrainBranchBot: [C:03+2] "Approved by samtar@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343957 (https://phabricator.wikimedia.org/T438334) (owner: 10Samtar) [12:59:25] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345999 (https://phabricator.wikimedia.org/T439534) (owner: 10Marco Fossati) [13:00:02] (03Merged) 10jenkins-bot: InitialiseSettings: Enable ModeratorToolkit on test.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343957 (https://phabricator.wikimedia.org/T438334) (owner: 10Samtar) [13:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: Time to do the UTC afternoon backport window deploy. Don't look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1300). [13:00:05] TheresNoTime, MichaelG_WMF, and mfossati: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:11] o/ [13:00:15] o/ [13:00:16] * TheresNoTime is just self-deploying their two patches first if thats ok [13:00:23] !log samtar@deploy1003 Started scap sync-world: Backport for [[gerrit:1343957|InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] [13:00:24] * MichaelG_WMF is here [13:00:28] T438334: Deploy ModeratorToolkit to test.wiki - https://phabricator.wikimedia.org/T438334 [13:00:50] (03CR) 10CI reject: [V:04-1] Interface validator: support breakout port names on SR-Linux [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1346001 (https://phabricator.wikimedia.org/T439537) (owner: 10Cathal Mooney) [13:01:21] (03PS1) 10Isabelle Hurbain-Palatin: Add debug on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346003 [13:01:38] (03PS2) 10Cathal Mooney: Interface validator: support breakout port names on SR-Linux [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1346001 (https://phabricator.wikimedia.org/T439537) [13:02:17] (03CR) 10Btullis: [C:03+2] puppetserver: Permit the new dse-k8s-eqiad Pod range to rsync [puppet] - 10https://gerrit.wikimedia.org/r/1345970 (https://phabricator.wikimedia.org/T429773) (owner: 10Btullis) [13:02:29] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [core] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345216 (https://phabricator.wikimedia.org/T425216) (owner: 10Krinkle) [13:02:34] happy to be last, my backport is all about translations :-) [13:02:45] (03PS2) 10Isabelle Hurbain-Palatin: mobileapps: Add debug on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346003 [13:02:56] array_push($this) [13:02:59] mfossati: congrats, you're before last now. [13:03:02] (03CR) 10Jgiannelos: [C:03+2] mobileapps: Add debug on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346003 (owner: 10Isabelle Hurbain-Palatin) [13:03:19] Krinkle: are you sure you want to go after those l10n backports :P [13:03:29] np :) [13:04:11] anyway I can deploy if needed (probably for MichaelG_WMF) [13:04:14] Krinkle: :-D no worries, you can go before [13:04:42] Lucas_WMDE: yes please :) [13:04:54] thx, anytime is fine. I suppose after MichaelG_WMF is done. [13:04:56] !log samtar@deploy1003 samtar: Backport for [[gerrit:1343957|InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:04:59] (03PS1) 10Btullis: install_server: Add a partman recipe that keeps SSD space for TopoLVM [puppet] - 10https://gerrit.wikimedia.org/r/1346004 (https://phabricator.wikimedia.org/T438058) [13:05:19] * Lucas_WMDE wonders how TheresNoTime will test ModeratorToolkit on testwiki before the extension is loaded in CS.php ;) [13:05:26] Lucas_WMDE: that would be great this time [13:05:29] !log samtar@deploy1003 samtar: Continuing with deployment [13:05:34] Lucas_WMDE: like that :D [13:05:47] presto! [13:06:06] (03PS1) 10Btullis: dumps: web: Add a brownout for v1 meta-current and meta-history [puppet] - 10https://gerrit.wikimedia.org/r/1346007 (https://phabricator.wikimedia.org/T437461) [13:06:17] (03PS2) 10Btullis: dse-k8s-eqiad: Retire the old Pod IP range [puppet] - 10https://gerrit.wikimedia.org/r/1345971 (https://phabricator.wikimedia.org/T430658) [13:06:17] (03PS6) 10Ayounsi: network: refactor Junos BGP and add get_junos_bgp_details [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) [13:06:25] btullis@cumin1004 provision (PID 1566121) is awaiting input [13:06:27] (03PS1) 10Lerickson: Annotate wdqs-proxy pods so metrics are scraped. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346008 (https://phabricator.wikimedia.org/T439449) [13:06:38] (I feel like those two patches should have been the other way around, but the guides seem pretty consistent in that order o.O) [13:06:40] (03Merged) 10jenkins-bot: mobileapps: Add debug on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346003 (owner: 10Isabelle Hurbain-Palatin) [13:06:56] I was gonna say they could’ve been deployed together [13:06:59] we have guides for that? [13:07:41] Lucas_WMDE: somewhere in https://www.mediawiki.org/wiki/Writing_an_extension_for_deployment :D [13:08:28] (03PS2) 10Lerickson: Annotate wdqs-proxy pods so metrics are scraped. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346008 (https://phabricator.wikimedia.org/T439449) [13:08:46] also, this feels a bit risky while the rest of the train is still on wmf.21… [13:08:47] lucaswerkmeister-wmde@deploy1003:/srv/mediawiki-staging$ ls -d */extensions/ModeratorToolkit [13:08:47] php-1.47.0-wmf.22/extensions/ModeratorToolkit php-next/extensions/ModeratorToolkit [13:09:15] idk how likely it is that we roll back testwiki [13:09:37] Lucas_WMDE: fair point... risky enough that you think I should delay? [13:09:55] not sure… it’s easy for me to nag from the sidelines ;) [13:10:01] idk how urgent this is for your team [13:10:08] (03PS1) 10Jgiannelos: mobileapps: Enable trace level logging on staging for debugging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346009 [13:10:19] I think it’s at least worth a “risky patch” comment on https://phabricator.wikimedia.org/T438218 [13:10:23] (train-blocker task) [13:10:45] !log samtar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1343957|InitialiseSettings: Enable ModeratorToolkit on test.wiki (T438334)]] (duration: 10m 21s) [13:10:47] I have a deadline for the hypothesis thing, but hm.. I'll leave a comment ^ [13:10:48] T438334: Deploy ModeratorToolkit to test.wiki - https://phabricator.wikimedia.org/T438334 [13:11:07] (03CR) 10TrainBranchBot: [C:03+2] "Approved by samtar@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343958 (https://phabricator.wikimedia.org/T438334) (owner: 10Samtar) [13:11:07] *looks at calendar* ah yes ;) [13:11:25] (03PS1) 10Jgiannelos: mobileapps: Enable trace level logging for debugging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346010 [13:11:26] (03Abandoned) 10Jgiannelos: mobileapps: Enable trace level logging on staging for debugging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346009 (owner: 10Jgiannelos) [13:12:37] (03CR) 10Isabelle Hurbain-Palatin: [C:03+2] mobileapps: Enable trace level logging for debugging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346010 (owner: 10Jgiannelos) [13:12:38] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342962 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [13:14:09] (03Merged) 10jenkins-bot: CommonSettings: Load ModeratorToolkit extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343958 (https://phabricator.wikimedia.org/T438334) (owner: 10Samtar) [13:14:31] !log samtar@deploy1003 Started scap sync-world: Backport for [[gerrit:1343958|CommonSettings: Load ModeratorToolkit extension (T438334)]] [13:14:39] (03PS4) 10Daniel Kertesz: profile::cache::base: reserve ports used by local daemons [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) [13:15:14] (03Merged) 10jenkins-bot: mobileapps: Enable trace level logging for debugging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346010 (owner: 10Jgiannelos) [13:15:22] (03CR) 10Daniel Kertesz: profile::cache::base: reserve ports used by local daemons (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) (owner: 10Daniel Kertesz) [13:15:37] !log ihurbain@deploy1003 helmfile [staging] START helmfile.d/services/mobileapps: apply [13:15:47] !log ihurbain@deploy1003 helmfile [staging] DONE helmfile.d/services/mobileapps: apply [13:15:52] !log ihurbain@deploy1003 helmfile [staging] START helmfile.d/services/mobileapps: apply [13:15:56] !log ihurbain@deploy1003 helmfile [staging] DONE helmfile.d/services/mobileapps: apply [13:16:22] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342963 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [13:16:22] !log ihurbain@deploy1003 helmfile [staging] START helmfile.d/services/mobileapps: apply [13:16:45] !log ihurbain@deploy1003 helmfile [staging] DONE helmfile.d/services/mobileapps: apply [13:17:15] (03CR) 10VolkerE: Enable ReadingLists on all Wikipedia wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345588 (https://phabricator.wikimedia.org/T434924) (owner: 10Aude) [13:17:28] (03PS1) 10DCausse: Search: optimize morelike queries [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346012 [13:18:01] (03CR) 10Vgutierrez: [C:03+1] profile::cache::base: reserve ports used by local daemons (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) (owner: 10Daniel Kertesz) [13:18:15] TheresNoTime: thanks for adding the comment! [13:18:21] (03CR) 10CI reject: [V:04-1] Search: optimize morelike queries [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346012 (owner: 10DCausse) [13:19:10] !log samtar@deploy1003 samtar: Backport for [[gerrit:1343958|CommonSettings: Load ModeratorToolkit extension (T438334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:19:14] T438334: Deploy ModeratorToolkit to test.wiki - https://phabricator.wikimedia.org/T438334 [13:19:18] * TheresNoTime testing ^ [13:19:25] (03PS2) 10DCausse: Search: optimize morelike queries [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346012 [13:20:32] !log samtar@deploy1003 samtar: Continuing with deployment [13:21:10] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [13:21:52] (03CR) 10Giuseppe Lavagetto: [C:04-1] "Overall I think this goes in the right direction and two main things need to be checked:" [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958) (owner: 10Dduvall) [13:21:58] (03CR) 10Ayounsi: "tested with the peering cookbook to make sure there are no regressions." [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [13:23:27] (03PS1) 10Isabelle Hurbain-Palatin: mobileapps: put the query parameters top-level and not headers [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346013 (https://phabricator.wikimedia.org/T431693) [13:23:28] (03PS8) 10JMeybohm: gVisor: use kvm as a platform when hardware virtualization is available [puppet] - 10https://gerrit.wikimedia.org/r/1342962 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [13:23:37] (03CR) 10CI reject: [V:04-1] mobileapps: put the query parameters top-level and not headers [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346013 (https://phabricator.wikimedia.org/T431693) (owner: 10Isabelle Hurbain-Palatin) [13:23:39] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342963 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [13:24:54] (03PS2) 10Isabelle Hurbain-Palatin: mobileapps: put the query parameters top-level and not headers [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346013 (https://phabricator.wikimedia.org/T431693) [13:25:48] !log samtar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1343958|CommonSettings: Load ModeratorToolkit extension (T438334)]] (duration: 11m 17s) [13:25:51] T438334: Deploy ModeratorToolkit to test.wiki - https://phabricator.wikimedia.org/T438334 [13:25:55] Lucas_WMDE: all yours [13:25:59] TheresNoTime: pretty sure the guide you're referring to dates back to when we were syncing individual files so you needed to define the variable in CS before you could use it in IS. today it could just as well be a single patch [13:26:04] thanks! [13:26:08] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345952 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große) [13:26:08] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1345953 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große) [13:26:25] (03CR) 10Ayounsi: [C:03+1] Interface validator: support breakout port names on SR-Linux [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1346001 (https://phabricator.wikimedia.org/T439537) (owner: 10Cathal Mooney) [13:26:27] (03CR) 10Jgiannelos: [C:03+2] mobileapps: put the query parameters top-level and not headers [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346013 (https://phabricator.wikimedia.org/T431693) (owner: 10Isabelle Hurbain-Palatin) [13:26:34] taavi: I for one am shocked that our docs are slightly out of date [13:26:48] (03PS1) 10Vgutierrez: purged: use the new host prefixes [puppet] - 10https://gerrit.wikimedia.org/r/1346014 [13:27:05] !bash taavi: I for one am shocked that our docs are slightly out of date [13:27:05] Lucas_WMDE: Stored quip at https://bash.toolforge.org/quip/dPNY7aABDAZZyZXn9Ucp [13:27:08] (03PS1) 10Jgiannelos: Revert "mobileapps: Enable trace level logging for debugging" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346015 [13:27:30] TheresNoTime: be the change you want to ~~see~~ read [13:27:40] well, I guess reading is a subcategory of seeing [13:27:59] (03PS5) 10Giuseppe Lavagetto: kubernetes::node: add gvisor label based on hardware virtualization [puppet] - 10https://gerrit.wikimedia.org/r/1342963 (https://phabricator.wikimedia.org/T438287) [13:28:07] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1342963 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [13:28:16] I should’ve +2ed those backports in advance. too late now [13:29:01] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:29:03] (03Merged) 10jenkins-bot: mobileapps: put the query parameters top-level and not headers [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346013 (https://phabricator.wikimedia.org/T431693) (owner: 10Isabelle Hurbain-Palatin) [13:29:48] (03CR) 10Vgutierrez: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346014 (owner: 10Vgutierrez) [13:30:16] !log ihurbain@deploy1003 helmfile [staging] START helmfile.d/services/mobileapps: apply [13:30:40] !log ihurbain@deploy1003 helmfile [staging] DONE helmfile.d/services/mobileapps: apply [13:33:19] (03CR) 10Vgutierrez: "PCC shows a expected NOOP for the text cluster" [puppet] - 10https://gerrit.wikimedia.org/r/1346014 (owner: 10Vgutierrez) [13:33:46] (03PS1) 10Ilias Sarantopoulos: ml-services: fix gpt-oss-safeguard name [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346016 (https://phabricator.wikimedia.org/T439395) [13:33:59] (03Merged) 10jenkins-bot: fix: prevent AccountSetup from being scrollable [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345952 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große) [13:34:01] (03Merged) 10jenkins-bot: fix: prevent AccountSetup from being scrollable [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1345953 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große) [13:34:26] !log lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1345952|fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953|fix: prevent AccountSetup from being scrollable (T436872)]] [13:34:29] T436872: Account Setup experiment: test end-to-end flow and verify translations on pilot wikis (DE1.3.1) - https://phabricator.wikimedia.org/T436872 [13:34:40] * MichaelG_WMF is ready to test [13:35:02] (03PS2) 10Vgutierrez: purged: use the new host prefixes [puppet] - 10https://gerrit.wikimedia.org/r/1346014 [13:35:05] (03Abandoned) 10Jgiannelos: Revert "mobileapps: Enable trace level logging for debugging" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346015 (owner: 10Jgiannelos) [13:35:18] (03CR) 10Ilias Sarantopoulos: "Turns out the llm prefix is required in the model name as well." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346016 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [13:36:02] (03CR) 10JMeybohm: [C:03+1] "Fixed wrong interpretation of the `is_virtual` fact." [puppet] - 10https://gerrit.wikimedia.org/r/1342962 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [13:36:05] (03PS1) 10Jgiannelos: mobileapps: Disable debugging on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346017 [13:36:07] (03CR) 10JMeybohm: [C:03+1] kubernetes::node: add gvisor label based on hardware virtualization [puppet] - 10https://gerrit.wikimedia.org/r/1342963 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [13:38:23] !log lucaswerkmeister-wmde@deploy1003 migr, lucaswerkmeister-wmde: Backport for [[gerrit:1345952|fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953|fix: prevent AccountSetup from being scrollable (T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:38:33] (03CR) 10Vgutierrez: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346014 (owner: 10Vgutierrez) [13:39:00] MichaelG_WMF: please test :) [13:39:06] testing... [13:39:38] also Krinkle, you can probably CR+2 your backport already to speed it up [13:40:14] (03CR) 10Krinkle: [C:03+2] File: Implement LocalFile::getUrlForPurge workaround for caching [core] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345216 (https://phabricator.wikimedia.org/T425216) (owner: 10Krinkle) [13:40:19] (03CR) 10Joal: "Some comment inline" [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [13:40:42] Lucas_WMDE: looks fine! Ready to roll forward :) [13:40:49] !log lucaswerkmeister-wmde@deploy1003 migr, lucaswerkmeister-wmde: Continuing with deployment [13:40:50] thanks! [13:41:43] (03CR) 10Joal: "This patch lacks the data-purge aspect the other one has :) Plus some comment inline." [puppet] - 10https://gerrit.wikimedia.org/r/1343954 (https://phabricator.wikimedia.org/T437961) (owner: 10A-pizzata) [13:45:56] (03PS1) 10Klausman: profiles/amd_gpu: check xGMI, note stuck DfPstate [puppet] - 10https://gerrit.wikimedia.org/r/1346019 (https://phabricator.wikimedia.org/T431553) [13:46:05] !log lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345952|fix: prevent AccountSetup from being scrollable (T436872)]], [[gerrit:1345953|fix: prevent AccountSetup from being scrollable (T436872)]] (duration: 11m 39s) [13:46:09] T436872: Account Setup experiment: test end-to-end flow and verify translations on pilot wikis (DE1.3.1) - https://phabricator.wikimedia.org/T436872 [13:46:23] (03PS2) 10Klausman: profiles/amd_gpu: check xGMI, note stuck DfPstate [puppet] - 10https://gerrit.wikimedia.org/r/1346019 (https://phabricator.wikimedia.org/T431553) [13:46:28] Krinkle: over to you [13:47:18] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [13:47:42] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie [13:47:42] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2001.codfw.wmnet [13:48:01] (03Merged) 10jenkins-bot: File: Implement LocalFile::getUrlForPurge workaround for caching [core] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345216 (https://phabricator.wikimedia.org/T425216) (owner: 10Krinkle) [13:48:42] (03CR) 10A-pizzata: sqoop_mediawiki: Add weekly change_tag sqoop (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [13:48:48] RESOLVED: PuppetFailure: Puppet has failed on testvm2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [13:49:01] FIRING: [3x] SystemdUnitFailed: wmf_auto_restart_krb5-admin-server.service on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:50:50] Lucas_WMDE: ack [13:51:06] !log krinkle@deploy1003 Started deploy [performance/arc-lamp@35c590e]: T331941 [13:51:09] T331941: arclamp-log.py prunes data too soon (after 30d instead of 90d) - https://phabricator.wikimedia.org/T331941 [13:51:15] !log krinkle@deploy1003 Finished deploy [performance/arc-lamp@35c590e]: T331941 (duration: 00m 09s) [13:51:36] (03CR) 10Fabfur: purged: use the new host prefixes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1346014 (owner: 10Vgutierrez) [13:52:08] (03PS3) 10Vgutierrez: purged: use the new host prefixes [puppet] - 10https://gerrit.wikimedia.org/r/1346014 [13:52:38] (03CR) 10Fabfur: [C:03+1] "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/1346014 (owner: 10Vgutierrez) [13:52:39] (03CR) 10Vgutierrez: purged: use the new host prefixes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1346014 (owner: 10Vgutierrez) [13:53:47] 10ops-eqiad, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T439559 (10tappof) 03NEW [13:53:55] 10ops-eqiad, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T439559#12374492 (10tappof) [13:54:24] (03CR) 10Xcollazo: dumps: web: Add a brownout for v1 meta-current and meta-history (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1346007 (https://phabricator.wikimedia.org/T437461) (owner: 10Btullis) [13:54:36] (03CR) 10Ozge: [C:03+1] ml-services: fix gpt-oss-safeguard name [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346016 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [13:54:40] (03CR) 10Btullis: [C:03+1] "Goodness!" [puppet] - 10https://gerrit.wikimedia.org/r/1346019 (https://phabricator.wikimedia.org/T431553) (owner: 10Klausman) [13:54:52] (03CR) 10Cathal Mooney: [C:03+2] Interface validator: support breakout port names on SR-Linux [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1346001 (https://phabricator.wikimedia.org/T439537) (owner: 10Cathal Mooney) [13:54:56] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [13:54:57] (03CR) 10Vgutierrez: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346014 (owner: 10Vgutierrez) [13:55:04] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344298 (https://phabricator.wikimedia.org/T362786) (owner: 10Krinkle) [13:55:41] (03CR) 10Klausman: [C:03+1] install_server: Add a partman recipe that keeps SSD space for TopoLVM [puppet] - 10https://gerrit.wikimedia.org/r/1346004 (https://phabricator.wikimedia.org/T438058) (owner: 10Btullis) [13:56:13] (03CR) 10Dpogorzelski: [C:03+1] "I can't tell if the partman config is ok but i trust it is :)" [puppet] - 10https://gerrit.wikimedia.org/r/1346004 (https://phabricator.wikimedia.org/T438058) (owner: 10Btullis) [13:56:15] (03Merged) 10jenkins-bot: ProductionServices: Simplify indirect parsercache-dbs stub [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344298 (https://phabricator.wikimedia.org/T362786) (owner: 10Krinkle) [13:56:40] (03CR) 10JMeybohm: [C:03+1] "Thanks for taking care of this!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344831 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [13:56:40] !log krinkle@deploy1003 Started scap sync-world: Backport for [[gerrit:1345216|File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298|ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] [13:56:45] (03CR) 10JMeybohm: [C:03+1] python-webapp releases: Remove ingress.staging and ingress.mlstaging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344832 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [13:56:47] T425216: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216 [13:56:48] T362786: Enable dbctl for parsercache - https://phabricator.wikimedia.org/T362786 [13:56:48] T437209: Move poolcounter configuration to etcd - https://phabricator.wikimedia.org/T437209 [13:57:18] (03Merged) 10jenkins-bot: Interface validator: support breakout port names on SR-Linux [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1346001 (https://phabricator.wikimedia.org/T439537) (owner: 10Cathal Mooney) [13:58:06] (03CR) 10JMeybohm: [C:03+1] python-webapp: Update ingress.istio to 1.4.0 (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344831 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [13:58:45] !log cmooney@cumin1004 START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary [13:59:07] !log cmooney@cumin1004 END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary [14:00:04] Deploy window Test Kitchen UI Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1400) [14:00:51] !log krinkle@deploy1003 krinkle: Backport for [[gerrit:1345216|File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298|ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:01:06] (03CR) 10JMeybohm: [C:03+1] mesh.service: Add public_port_app_protocol (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345646 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [14:01:12] (03CR) 10JMeybohm: [C:03+1] python-webapp: Update mesh.service to 1.2.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345647 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [14:01:23] (03CR) 10JMeybohm: [C:03+1] editcheck-headless: Declare the TLS port as gRPC [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345648 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [14:01:31] (03CR) 10JMeybohm: [C:03+1] mesh: Copy service 1.2.1 to 1.2.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345645 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [14:02:11] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [14:02:58] (03PS2) 10DLynch: python-webapp: Update ingress.istio to 1.4.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344831 (https://phabricator.wikimedia.org/T436689) [14:03:09] (03CR) 10DLynch: python-webapp: Update ingress.istio to 1.4.0 (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344831 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [14:03:33] (03CR) 10Daniel Kertesz: [V:03+1] "PCC SUCCESS (CORE_DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9486/co" [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) (owner: 10Daniel Kertesz) [14:03:36] (03CR) 10Klausman: [C:03+2] profiles/amd_gpu: check xGMI, note stuck DfPstate [puppet] - 10https://gerrit.wikimedia.org/r/1346019 (https://phabricator.wikimedia.org/T431553) (owner: 10Klausman) [14:04:17] (03CR) 10Joal: sqoop_mediawiki: Add weekly change_tag sqoop (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [14:05:49] !log krinkle@deploy1003 krinkle: Continuing with deployment [14:06:00] (03CR) 10Btullis: [C:03+2] install_server: Add a partman recipe that keeps SSD space for TopoLVM [puppet] - 10https://gerrit.wikimedia.org/r/1346004 (https://phabricator.wikimedia.org/T438058) (owner: 10Btullis) [14:06:04] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004" [14:07:09] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 - btullis@cumin1004" [14:07:09] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:07:56] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:08:59] 06SRE, 06Commons, 10MediaWiki-File-management, 06Traffic, and 3 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12374600 (10Krinkle) LGTM now, for some definition of "good". 169 URLs when purging (03PS1) 10Andrew Bogott: Horizon: update codfw1dev build tag [puppet] - 10https://gerrit.wikimedia.org/r/1346027 (https://phabricator.wikimedia.org/T439363) [14:10:59] btullis@cumin1004 provision (PID 1581863) is awaiting input [14:11:06] !log krinkle@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345216|File: Implement LocalFile::getUrlForPurge workaround for caching (T425216)]], [[gerrit:1344298|ProductionServices: Simplify indirect parsercache-dbs stub (T362786 T437209)]] (duration: 14m 26s) [14:11:12] T425216: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216 [14:11:12] T362786: Enable dbctl for parsercache - https://phabricator.wikimedia.org/T362786 [14:11:13] T437209: Move poolcounter configuration to etcd - https://phabricator.wikimedia.org/T437209 [14:12:17] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:12:19] (03CR) 10Andrew Bogott: [C:03+2] Horizon: update codfw1dev build tag [puppet] - 10https://gerrit.wikimedia.org/r/1346027 (https://phabricator.wikimedia.org/T439363) (owner: 10Andrew Bogott) [14:12:31] !log bking@cumin2003 conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-dse-aa,name=codfw [14:13:42] (03CR) 10Scott French: "Thanks, Ahmon! I have some questions about the analysis that led to the thresholds here, but let's discuss that on T435419." [puppet] - 10https://gerrit.wikimedia.org/r/1345206 (https://phabricator.wikimedia.org/T435419) (owner: 10Ahmon Dancy) [14:13:43] (03PS1) 10Ayounsi: ulsfo/eqsin - remove dhcp_relay_ra [homer/public] - 10https://gerrit.wikimedia.org/r/1346028 [14:14:08] (03PS2) 10Ayounsi: ulsfo/eqsin - remove dhcp_relay_ra [homer/public] - 10https://gerrit.wikimedia.org/r/1346028 [14:14:31] 06SRE, 06Infrastructure-Foundations: Unresolved failures from url-downloader - https://phabricator.wikimedia.org/T439560 (10KMontalva-WMF) 03NEW [14:16:10] !log klausman@cumin1004 START - Cookbook sre.hosts.remove-downtime for ml-serve1015.eqiad.wmnet [14:16:11] !log klausman@cumin1004 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ml-serve1015.eqiad.wmnet [14:18:10] FIRING: [5x] BFDdown: BFD session down between cr2-codfw and 198.35.26.157 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:19:34] !log cdobbins@cumin1004 START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie [14:21:57] !log installing rsyslog security updates [14:21:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:21:59] !log cmooney@cumin1004 START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox [14:23:06] !log cmooney@cumin1004 END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox [14:23:08] (03PS1) 10Andrew Bogott: Update eqiad1 horizon version [puppet] - 10https://gerrit.wikimedia.org/r/1346032 (https://phabricator.wikimedia.org/T439363) [14:23:10] FIRING: [5x] BFDdown: BFD session down between cr2-codfw and 198.35.26.157 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:24:23] (03CR) 10Andrew Bogott: [C:03+2] Update eqiad1 horizon version [puppet] - 10https://gerrit.wikimedia.org/r/1346032 (https://phabricator.wikimedia.org/T439363) (owner: 10Andrew Bogott) [14:25:45] FIRING: WidespreadPuppetFailure: Puppet has failed in esams - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [14:26:24] (03CR) 10Cathal Mooney: [C:03+1] "indeed yep an oversight on my part thanks." [homer/public] - 10https://gerrit.wikimedia.org/r/1346028 (owner: 10Ayounsi) [14:28:15] jouncebot: nowandnext [14:28:15] For the next 0 hour(s) and 1 minute(s): Test Kitchen UI Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1400) [14:28:16] In 0 hour(s) and 1 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1430) [14:28:37] mfossati: do you still want to deploy your l10n backport? (idk if that conflicts with test kitchen though) [14:29:01] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [14:29:10] Lucas_WMDE: IDK, that would be great, but the window has gone [14:30:04] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1430) [14:30:07] I have a meetings wall, so I guess we can postpone [14:30:11] ok [14:30:18] we could deploy out-of-window if there’s nothing else going on [14:30:41] (IIUC the Test Kitchen Experiment Deployment Window is mainly there as a “marker” that the automatic start/stop of experiments happens at this time, not a blocker for other deployments) [14:30:45] FIRING: [2x] WidespreadPuppetFailure: Puppet has failed in esams - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [14:32:46] !log cgoubert@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues [14:33:18] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 3 others: wikikube-worker1152.eqiad.wmnet networking issue - https://phabricator.wikimedia.org/T438819#12374810 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=f255ecea-f2f3-40b3-9dd5-7fafa26657b1) set by cgoubert@cumin2003 for 7 days... [14:34:14] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 3 others: wikikube-worker1152.eqiad.wmnet networking issue - https://phabricator.wikimedia.org/T438819#12374832 (10Clement_Goubert) [14:38:47] (03CR) 10Andrew Bogott: [C:03+1] cloudlb: Allow specifying multiple addresses per frontend [puppet] - 10https://gerrit.wikimedia.org/r/1260004 (https://phabricator.wikimedia.org/T420921) (owner: 10Majavah) [14:40:45] FIRING: [2x] WidespreadPuppetFailure: Puppet has failed in esams - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [14:45:45] FIRING: [3x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [14:47:45] ACKNOWLEDGEMENT - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-worker1200 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 1 Failed : virtual_disk: 1 OfLn : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T439570 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [14:47:51] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1200 - https://phabricator.wikimedia.org/T439570 (10ops-monitoring-bot) 03NEW [14:48:10] (03PS3) 10Atsuko: airflow: fixes for Oauth [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345956 (https://phabricator.wikimedia.org/T433383) [14:50:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [14:51:11] (03PS4) 10Atsuko: airflow3: fixes for OAuth [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345956 (https://phabricator.wikimedia.org/T433383) [14:55:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [14:56:42] (03CR) 10A-pizzata: sqoop_mediawiki: Add weekly change_tag sqoop (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [14:58:41] 10SRE-SLO, 03Abstract Wikipedia Fix-It tasks, 06Abstract Wikipedia team (27Q1 (Jul–Sep)), 07Essential-Work: Sloth 30-day SLI over-counts errors for our low-traffic SLO, wikilambda-parsoid-combined - https://phabricator.wikimedia.org/T439571 (10Jdforrester-WMF) 03NEW [15:00:04] jelto, arnoldokoth, mutante, and arnaudb: OwO what's this, a deployment window?? SRE Collaboration Services office hours. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1500). nyaa~ [15:00:21] yea, but nothing is happening this window [15:01:50] we are having some puppetserver woes, investigating [15:03:06] thanks for confirming that jhathway or I would have wondering right this moment about my failed run [15:03:10] been [15:06:51] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 366872360 and 23 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [15:08:10] not obvious that anything has changed, I'm going to bounce the puppetservers, for lack of a better idea at present [15:08:23] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [15:08:51] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 2410832 and 2 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [15:08:57] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [15:09:03] !log cdobbins@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage [15:10:16] (03PS5) 10Atsuko: airflow3: fixes for OAuth [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345956 (https://phabricator.wikimedia.org/T433383) [15:12:39] FIRING: [3x] CoreBGPDown: Core BGP session down between cr2-codfw and cr4-ulsfo (198.35.26.157) - group Confed_ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:14:27] !log cdobbins@cumin1004 END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ncredir5004.eqsin.wmnet with reason: host reimage [15:15:37] (03CR) 10Ebernhardson: [C:03+1] Search: optimize morelike queries [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346012 (owner: 10DCausse) [15:20:05] Anything being deployed right now? I have a beta MW config patch so I'm going to +2 that and spiderpig it real quick if I'm not stepping on anyone's toes [15:20:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [15:21:19] ^^^ known, working on it [15:22:39] FIRING: [5x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:23:25] !log dzahn@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie [15:23:38] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12375115 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1004 for host zuul1005.eqiad.wmnet with OS trixie... [15:24:01] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [15:24:31] (03CR) 10TrainBranchBot: [C:03+2] "Approved by tchin@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345154 (https://phabricator.wikimedia.org/T429497) (owner: 10TChin) [15:25:03] (03CR) 10Joal: sqoop_mediawiki: Add weekly change_tag sqoop (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [15:25:15] 10ops-codfw, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: wikikube-worker2280 unreachable - https://phabricator.wikimedia.org/T423395#12375127 (10Jhancock.wm) they made me change ticketing systems for some reason. new case number #CS-00207636 [15:25:23] (03Merged) 10jenkins-bot: [beta eventgate] Migrate urls to deployment-eventgate05 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345154 (https://phabricator.wikimedia.org/T429497) (owner: 10TChin) [15:27:11] (03PS3) 10A-pizzata: sqoop_mediawiki: Add weekly full snapshot of the actor table [puppet] - 10https://gerrit.wikimedia.org/r/1343954 (https://phabricator.wikimedia.org/T437961) [15:28:13] * tchin Finished spiderpigging [15:28:36] (03CR) 10A-pizzata: "was planning to push the data-purge as soon as the tables are available on prod, either in this MR or in a following one." [puppet] - 10https://gerrit.wikimedia.org/r/1343954 (https://phabricator.wikimedia.org/T437961) (owner: 10A-pizzata) [15:33:42] (03PS2) 10Majavah: python: Log to stdout/stderr only [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1226300 (https://phabricator.wikimedia.org/T401102) [15:33:51] (03PS2) 10Majavah: lighttpd: Log to stdout/stderr only [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1226301 (https://phabricator.wikimedia.org/T401102) [15:39:30] !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 () [15:43:12] !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 () [15:45:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [15:47:50] (03PS2) 10DLynch: mesh.service: Add public_port_app_protocol [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345646 (https://phabricator.wikimedia.org/T436689) [15:47:51] (03PS2) 10DLynch: python-webapp: Update mesh.service to 1.2.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345647 (https://phabricator.wikimedia.org/T436689) [15:47:51] (03PS2) 10DLynch: editcheck-headless: Declare the TLS port as gRPC [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345648 (https://phabricator.wikimedia.org/T436689) [15:47:55] (03CR) 10Ilias Sarantopoulos: [C:03+2] ml-services: fix gpt-oss-safeguard name [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346016 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [15:49:32] (03CR) 10DLynch: mesh.service: Add public_port_app_protocol (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345646 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [15:49:40] (03Merged) 10jenkins-bot: ml-services: fix gpt-oss-safeguard name [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346016 (https://phabricator.wikimedia.org/T439395) (owner: 10Ilias Sarantopoulos) [15:50:45] RESOLVED: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [15:53:11] (03PS3) 10Lerickson: Annotate wdqs-proxy pods so metrics are scraped. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346008 (https://phabricator.wikimedia.org/T439449) [15:55:19] (03CR) 10Trueg: [C:03+1] "nice" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346008 (https://phabricator.wikimedia.org/T439449) (owner: 10Lerickson) [15:56:31] (03CR) 10JHathaway: [C:03+2] stdlib: upgrade to v10.1.0 [puppet] - 10https://gerrit.wikimedia.org/r/1345157 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [15:57:18] (03PS4) 10Lerickson: Annotate wdqs-proxy pods so metrics are scraped. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346008 (https://phabricator.wikimedia.org/T439449) [15:59:09] !log updating puppet stdlib, should be a noop [15:59:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:00:05] jhathaway and rzl: Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1600). Please do the needful. [16:00:05] Dreamy_Jazz: A patch you scheduled for Puppet request window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [16:00:11] \o [16:00:33] o/ [16:00:45] Should just be a merge and then wait for next run [16:01:16] (and helmfile apply on mw-cron) [16:01:40] jhathaway: happy to run that or leave it for you if you haven't had a chance to do one of those [16:02:19] I haven't done one, so happy to take it, are there any docs to follow? [16:02:47] jhathaway: it's real long and complicated! https://wikitech.wikimedia.org/wiki/Mw-cron_jobs#Deploying_periodic_jobs [16:03:14] \o/ [16:03:35] I'm nearby though, including if the helmfile diff shows anything you're not sure about [16:04:19] (03CR) 10JHathaway: [C:03+2] maintenance: Set FOREACHWIKI_IGNORE_ERRORS for some PSI jobs [puppet] - 10https://gerrit.wikimedia.org/r/1345937 (https://phabricator.wikimedia.org/T432110) (owner: 10Dreamy Jazz) [16:06:23] (03CR) 10Bking: [C:03+2] Annotate wdqs-proxy pods so metrics are scraped. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346008 (https://phabricator.wikimedia.org/T439449) (owner: 10Lerickson) [16:07:18] Thanks, let me know if you need me to test (it should be fine to skip the manual run of these scripts, though there is also no harm in manually running them) [16:09:24] (03CR) 10A-pizzata: sqoop_mediawiki: Add weekly change_tag sqoop (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [16:10:02] (03CR) 10A-pizzata: sqoop_mediawiki: Add weekly change_tag sqoop (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:11:55] (03PS1) 10Michael Große: fix(AccountSetup): add missing i18n key to extension.json [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346060 (https://phabricator.wikimedia.org/T438908) [16:14:13] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, September 30 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#de" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346060 (https://phabricator.wikimedia.org/T438908) (owner: 10Michael Große) [16:15:15] !log jhathaway@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-cron: apply [16:15:51] 06SRE, 06Infrastructure-Foundations: Sunset rootless Cumin - https://phabricator.wikimedia.org/T439522#12375493 (10MoritzMuehlenhoff) p:05Triage→03Medium [16:16:13] (03CR) 10Bking: [C:03+1] airflow3: fixes for OAuth [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345956 (https://phabricator.wikimedia.org/T433383) (owner: 10Atsuko) [16:16:29] rzl: diffs looks normal, except some changes to the neworking policy, expected? [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:45] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [16:16:51] jhathaway: looking [16:18:03] jhathaway: yep that's T432444, fire when ready [16:18:03] T432444: Provision kafka-logging100[6-8] and kafka-logging200[6-8] - https://phabricator.wikimedia.org/T432444 [16:18:15] rzl: thanks [16:18:24] !log jhathaway@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply [16:18:31] Dreamy_Jazz: deployed [16:19:07] Thanks [16:19:29] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [16:19:43] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [16:19:49] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [16:19:58] !log urbanecm@deploy1003 mwscript-k8s job started: GrowthExperiments:reassignMentees --wiki=enwiki --mentor=Blacephalon --performer=Blacephalon --as-job # T418194 [16:20:00] T418194: Mentors still having mentees after removing themselves - https://phabricator.wikimedia.org/T418194 [16:20:07] (03PS6) 10Atsuko: airflow3: fixes for OAuth [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345956 (https://phabricator.wikimedia.org/T433383) [16:20:19] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [16:21:06] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [16:21:31] jhathaway: congratulations, you're a cron artist [16:22:02] needs more flair [16:22:05] :P [16:22:20] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [16:22:37] (03CR) 10Atsuko: [C:03+1] "recovering lost +1" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345956 (https://phabricator.wikimedia.org/T433383) (owner: 10Atsuko) [16:24:45] (03CR) 10Ryan Kemper: [C:03+2] wdqs: configure eqiad expansion hosts [puppet] - 10https://gerrit.wikimedia.org/r/1345043 (https://phabricator.wikimedia.org/T420707) (owner: 10Ryan Kemper) [16:26:14] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: db2226 PSU loss of redundancy - https://phabricator.wikimedia.org/T439205#12375612 (10Jhancock.wm) 05Open→03Resolved PSU replaced and all looks healthy. [16:27:37] PROBLEM - Check unit status of statograph_post on alert1002 is CRITICAL: CRITICAL: Status of the systemd unit statograph_post https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [16:27:49] PROBLEM - Thanos swift https on thanos-fe1007 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 260 bytes in 1.051 second response time https://wikitech.wikimedia.org/wiki/Thanos [16:27:49] PROBLEM - Thanos swift https on thanos-fe1005 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 260 bytes in 1.063 second response time https://wikitech.wikimedia.org/wiki/Thanos [16:27:49] PROBLEM - Thanos swift https on thanos-fe1006 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 260 bytes in 1.082 second response time https://wikitech.wikimedia.org/wiki/Thanos [16:27:57] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - thanos-swift_443: Servers thanos-fe1006.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [16:28:06] 06SRE, 06Infrastructure-Foundations: Unresolved failures from url-downloader - https://phabricator.wikimedia.org/T439560#12375614 (10cmooney) Thanks for the task @KMontalva-WMF. Looking at the overall usage of the urldownloder hosts throughout that time they were still doing a lot of traffic and making connec... [16:28:49] RECOVERY - Thanos swift https on thanos-fe1006 is OK: HTTP OK: HTTP/1.1 200 OK - 281 bytes in 0.976 second response time https://wikitech.wikimedia.org/wiki/Thanos [16:29:24] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346012 (owner: 10DCausse) [16:29:57] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [16:30:49] RECOVERY - Thanos swift https on thanos-fe1007 is OK: HTTP OK: HTTP/1.1 200 OK - 279 bytes in 0.060 second response time https://wikitech.wikimedia.org/wiki/Thanos [16:30:49] RECOVERY - Thanos swift https on thanos-fe1005 is OK: HTTP OK: HTTP/1.1 200 OK - 279 bytes in 0.091 second response time https://wikitech.wikimedia.org/wiki/Thanos [16:31:33] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, September 30 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346012 (owner: 10DCausse) [16:31:34] FIRING: VarnishUnavailable: varnish-upload has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/Varnish#Diagnosing_Varnish_alerts - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DVarnishUnavailable [16:31:52] !ack [16:31:53] 8606 (ACKED) VarnishUnavailable global sre (varnish-upload thanos-rule@main) [16:32:31] in a meeting [16:32:34] brett: ^ [16:32:38] could this be related to the cookbook? [16:33:36] https://grafana.wikimedia.org/goto/s9kqlc?orgId=default seems a very transient small unavailability [16:34:16] but the rest of the panels show more errors [16:34:22] checking also live traffic [16:34:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [16:35:21] !log bking@cephosd1001 `sudo radosgw-admin user create --uid=matomo --display-name="Matomo on K8s"` T436186 [16:35:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:35:24] T436186: Matomo K8s migration: Create Helm chart - https://phabricator.wikimedia.org/T436186 [16:36:21] !log sukhe@puppetserver1001 conftool action : set/pooled=no; selector: name=cp5028 [16:36:34] RESOLVED: VarnishUnavailable: varnish-upload has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/Varnish#Diagnosing_Varnish_alerts - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DVarnishUnavailable [16:36:39] !log sukhe@puppetserver1001 conftool action : set/pooled=no; selector: name=cp5028.eqsin.wmnet [16:36:40] (03CR) 10JHathaway: [C:03+2] "Yeah I agree that is puzzling, something is not properly cleaning up hosts, needs further investigation." [puppet] - 10https://gerrit.wikimedia.org/r/1345157 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [16:45:07] (03PS1) 10JHathaway: postfix: upgrade module to v3.1.8 [puppet] - 10https://gerrit.wikimedia.org/r/1346068 (https://phabricator.wikimedia.org/T438912) [16:45:21] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346068 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [16:47:38] RECOVERY - Check unit status of statograph_post on alert1002 is OK: OK: Status of the systemd unit statograph_post https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [16:47:55] (03CR) 10JHathaway: [C:03+2] postfix: upgrade module to v3.1.8 [puppet] - 10https://gerrit.wikimedia.org/r/1346068 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [16:49:24] PROBLEM - Ensure acme-chief-api is running on acmechief2002 is CRITICAL: PROCS CRITICAL: 2 processes with args /usr/bin/uwsgi --die-on-term --ini /etc/uwsgi/apps-enabled/acme-chief.ini https://wikitech.wikimedia.org/wiki/Acme-chief [16:49:24] PROBLEM - Ensure acme-chief-backend is running only in the active node on acmechief2002 is CRITICAL: PROCS CRITICAL: 2 processes with args acme-chief-backend https://wikitech.wikimedia.org/wiki/Acme-chief [16:50:24] RECOVERY - Ensure acme-chief-backend is running only in the active node on acmechief2002 is OK: PROCS OK: 0 processes with args acme-chief-backend https://wikitech.wikimedia.org/wiki/Acme-chief [16:50:24] RECOVERY - Ensure acme-chief-api is running on acmechief2002 is OK: PROCS OK: 1 process with args /usr/bin/uwsgi --die-on-term --ini /etc/uwsgi/apps-enabled/acme-chief.ini https://wikitech.wikimedia.org/wiki/Acme-chief [16:54:27] !log ryankemper@deploy1003 Started deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host [16:54:30] T420707: Configure wdqs1033-1035 - https://phabricator.wikimedia.org/T420707 [16:54:32] !log ryankemper@deploy1003 Finished deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host (duration: 00m 04s) [16:54:36] !log ryankemper@deploy1003 Started deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host [16:54:40] !log ryankemper@deploy1003 Finished deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host (duration: 00m 04s) [16:54:46] !log ryankemper@deploy1003 Started deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host [16:54:51] !log ryankemper@deploy1003 Finished deploy [wdqs/wdqs@cf64fe9]: T420707: deploy to fresh wdqs host (duration: 00m 04s) [16:55:11] (03CR) 10Ladsgroup: [C:03+1] "why we even have nodejs here (later-me problem though). I'm ooo, if you want to, feel free to merge and deploy." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1345920 (owner: 10Muehlenhoff) [16:58:58] !log brett@puppetserver1001 conftool action : set/pooled=yes; selector: name=cp5028.* [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1700) [17:00:56] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards [17:00:59] T420707: Configure wdqs1033-1035 - https://phabricator.wikimedia.org/T420707 [17:01:13] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards [17:01:42] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T420707, T420707: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards [17:02:57] (03CR) 10Dduvall: "> does passing --no-cache work when using a remote buildkitd, or do we need to prune all the old image layers from it? We had discussed th" [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958) (owner: 10Dduvall) [17:16:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:17:55] (03CR) 10Dzahn: [C:03+1] "I confirmed the vrts processes also have systemd units. Yea, this is good to go." [puppet] - 10https://gerrit.wikimedia.org/r/1344226 (https://phabricator.wikimedia.org/T357099) (owner: 10Hnowlan) [17:19:21] (03CR) 10Dzahn: [C:03+2] collab-services: remove check_procs alerts in favour of SystemdUnitFailed [puppet] - 10https://gerrit.wikimedia.org/r/1344226 (https://phabricator.wikimedia.org/T357099) (owner: 10Hnowlan) [17:19:29] (03PS2) 10Hnowlan: collab-services: remove check_procs alerts in favour of SystemdUnitFailed [puppet] - 10https://gerrit.wikimedia.org/r/1344226 (https://phabricator.wikimedia.org/T357099) [17:19:33] (03CR) 10RLazarus: [C:03+1] upgrade rsyslog to 8.2608.0-4 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345655 (https://phabricator.wikimedia.org/T438375) (owner: 10Herron) [17:20:32] (03CR) 10Dzahn: [C:03+2] collab-services: remove check_procs alerts in favour of SystemdUnitFailed [puppet] - 10https://gerrit.wikimedia.org/r/1344226 (https://phabricator.wikimedia.org/T357099) (owner: 10Hnowlan) [17:24:02] (03PS1) 10Cklimas: InitialiseSettings: Enable return-to-app banner on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346087 (https://phabricator.wikimedia.org/T434258) [17:26:20] (03CR) 10Herron: [V:03+2 C:03+2] upgrade rsyslog to 8.2608.0-4 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345655 (https://phabricator.wikimedia.org/T438375) (owner: 10Herron) [17:27:12] (03CR) 10Jdlrobson: [C:04-1] InitialiseSettings: Enable return-to-app banner on testwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346087 (https://phabricator.wikimedia.org/T434258) (owner: 10Cklimas) [17:29:01] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [17:29:30] (03PS2) 10Cklimas: InitialiseSettings: Enable return-to-app banner on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346087 (https://phabricator.wikimedia.org/T434258) [17:32:56] (03CR) 10Cklimas: InitialiseSettings: Enable return-to-app banner on testwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346087 (https://phabricator.wikimedia.org/T434258) (owner: 10Cklimas) [17:33:03] (03CR) 10Kamila Součková: [C:03+1] "I will refrain from mentioning that all of this would come for free with [click.File](https://click.palletsprojects.com/en/stable/api/#cli" [puppet] - 10https://gerrit.wikimedia.org/r/1343171 (https://phabricator.wikimedia.org/T415835) (owner: 10RLazarus) [17:33:56] PROBLEM - Ensure trafficserver_exporter is running for instance backend on cp7003 is CRITICAL: PROCS CRITICAL: 2 processes with args /usr/bin/python3 /usr/bin/prometheus-trafficserver-exporter --no-procstats --no-ssl-verification --endpoint http://127.0.0.1:3128/_stats --port 9122 https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [17:34:56] RECOVERY - Ensure trafficserver_exporter is running for instance backend on cp7003 is OK: PROCS OK: 1 process with args /usr/bin/python3 /usr/bin/prometheus-trafficserver-exporter --no-procstats --no-ssl-verification --endpoint http://127.0.0.1:3128/_stats --port 9122 https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [17:39:11] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [17:41:05] RECOVERY - OSPF status on cr2-eqiad is OK: OSPFv2: 9/9 UP : OSPFv3: 9/9 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [17:41:21] (03CR) 10Jdlrobson: [C:03+1] "This makes more sense. Need help deploying?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346087 (https://phabricator.wikimedia.org/T434258) (owner: 10Cklimas) [17:41:43] RECOVERY - OSPF status on cr2-codfw is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [17:41:45] RECOVERY - OSPF status on cr4-ulsfo is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [17:42:06] 10SRE-SLO, 06SRE Observability: Support SLOs based on Test Kitchen instruments - https://phabricator.wikimedia.org/T438274#12376031 (10Ottomata) Related: - {T390323} - {T390328} From a [[ https://wikimedia.slack.com/archives/C01DFMX6QLB/p1790700747353419?thread_ts=1790693883.238179&cid=C01DFMX6QLB | slack thr... [17:42:39] FIRING: [5x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [17:43:10] FIRING: [4x] BFDdown: BFD session down between cr2-codfw and 198.35.26.157 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:44:02] !log Deployed refinery-source using jenkins as part of weekly deployment train [17:44:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:44:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [17:45:29] (03CR) 10Cklimas: "I think I've got it, but I'll ping you on Slack if I have questions/aren't sure!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346087 (https://phabricator.wikimedia.org/T434258) (owner: 10Cklimas) [17:45:55] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [17:46:03] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [17:46:33] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346087 (https://phabricator.wikimedia.org/T434258) (owner: 10Cklimas) [17:47:39] FIRING: [5x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [17:48:10] RESOLVED: [4x] BFDdown: BFD session down between cr2-codfw and 198.35.26.157 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:49:01] FIRING: SystemdUnitFailed: wmf_auto_restart_krb5-admin-server.service on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:50:16] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [17:51:12] (03PS1) 10Jdlrobson: Enable ReadingList CTA on Minerva for all reading list wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346053 (https://phabricator.wikimedia.org/T438779) [17:51:14] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir5004.eqsin.wmnet with OS trixie [17:51:37] (03PS2) 10Jdlrobson: Enable ReadingList CTA on Minerva for all reading list wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346053 (https://phabricator.wikimedia.org/T438779) [17:51:45] (03CR) 10Jdlrobson: "Done!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346053 (https://phabricator.wikimedia.org/T438779) (owner: 10Jdlrobson) [17:51:53] !log ebysans@deploy1003 Started deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] [17:52:40] !log ebysans@deploy1003 Finished deploy [analytics/refinery@613ef4d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@613ef4d9] (duration: 00m 46s) [17:53:09] !log Deploying Refinery as part of weekly deployment train [17:53:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:53:28] (03PS1) 10Jforrester: wikifunctions: Drop support for unused template block [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346098 [17:53:28] (03PS1) 10Jforrester: wikifunctions: Log at INFO level in the orchestrator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346099 (https://phabricator.wikimedia.org/T439592) [17:53:34] !log ebysans@deploy1003 Started deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] [17:54:25] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [17:54:27] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [17:55:10] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [17:57:17] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [17:58:28] !log ebysans@deploy1003 Finished deploy [analytics/refinery@613ef4d]: Regular analytics weekly train [analytics/refinery@613ef4d9] (duration: 04m 54s) [17:59:18] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T420707, T420707: initial data for new host) xfer scholarly_articles from wdqs2023.codfw.wmnet -> wdqs1035.eqiad.wmnet, repooling source-only afterwards [17:59:21] T420707: Configure wdqs1033-1035 - https://phabricator.wikimedia.org/T420707 [18:00:04] brennen and jeena: It is that lovely time of the day again! You are hereby commanded to deploy MediaWiki train - Utc-7 Version. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T1800). [18:00:20] o/ [18:01:32] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2013.codfw.wmnet -> wdqs1034.eqiad.wmnet, repooling source-only afterwards [18:02:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [18:05:26] !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_eqsin - 7.1.1-2~bpo13+wmf3 () [18:05:28] !log Deployed Refinery as part of weekly deployment train [18:05:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:06:17] !log 1.47.0-wmf.22 train status (T438218) - no current blockers, rolling to group0 [18:06:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:06:20] T438218: 1.47.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T438218 [18:06:58] (03PS1) 10TrainBranchBot: group0 to 1.47.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346101 (https://phabricator.wikimedia.org/T438218) [18:07:00] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by brennen@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346101 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [18:08:46] (03Merged) 10jenkins-bot: group0 to 1.47.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346101 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [18:08:48] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T420707, T420707: initial data for new host) xfer wdqs-all from wdqs2021.codfw.wmnet -> wdqs1033.eqiad.wmnet, repooling source-only afterwards [18:08:50] T420707: Configure wdqs1033-1035 - https://phabricator.wikimedia.org/T420707 [18:15:10] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [18:15:11] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [18:15:36] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [18:18:09] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [18:18:16] !log brennen@deploy1003 rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs T438218 [18:18:19] T438218: 1.47.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T438218 [18:19:31] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [18:19:37] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [18:19:53] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [18:22:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [18:23:26] (03CR) 10DLynch: [C:03+1] InitialiseSettings: Enable return-to-app banner on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346087 (https://phabricator.wikimedia.org/T434258) (owner: 10Cklimas) [18:23:55] (03CR) 10Atsuko: [C:03+2] airflow3: fixes for OAuth [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345956 (https://phabricator.wikimedia.org/T433383) (owner: 10Atsuko) [18:24:19] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [18:25:17] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [18:26:00] !log ryankemper@cumin2003 conftool action : set/weight=10:pooled=yes; selector: name=wdqs1035.eqiad.wmnet [18:26:05] !log ryankemper@cumin2003 conftool action : set/weight=10:pooled=yes; selector: name=wdqs1034.eqiad.wmnet [18:26:11] !log ryankemper@cumin2003 conftool action : set/weight=10:pooled=yes; selector: name=wdqs1033.eqiad.wmnet [18:27:07] (03Merged) 10jenkins-bot: airflow3: fixes for OAuth [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345956 (https://phabricator.wikimedia.org/T433383) (owner: 10Atsuko) [18:27:28] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [18:29:26] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [18:30:14] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [18:30:46] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [18:31:30] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004" [18:34:33] btullis@cumin1004 netbox (PID 1643573) is awaiting input [18:34:55] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1002 and ml-cache1003 - btullis@cumin1004" [18:34:56] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [18:35:07] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [18:37:04] (03CR) 10RLazarus: [C:03+2] "Haha thanks! Will consider for a followup for sure." [puppet] - 10https://gerrit.wikimedia.org/r/1343171 (https://phabricator.wikimedia.org/T415835) (owner: 10RLazarus) [18:39:45] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" [18:41:31] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" [18:41:32] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [18:45:24] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [18:49:20] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" [18:49:48] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache1001 network - btullis@cumin1004" [18:49:48] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [18:50:01] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [18:55:22] btullis@cumin1004 netbox (PID 1647491) is awaiting input [18:56:49] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm [18:56:52] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004" [18:56:55] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache servers - btullis@cumin1004" [18:56:55] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [18:57:43] !log cdobbins@cumin1004 conftool action : set/pooled=yes; selector: name=ncredir5004.* [18:58:02] (03CR) 10Ayounsi: [C:03+2] ulsfo/eqsin - remove dhcp_relay_ra [homer/public] - 10https://gerrit.wikimedia.org/r/1346028 (owner: 10Ayounsi) [19:01:29] (03PS2) 10SBassett: Remove foundationwiki's CSP report-only header [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345649 (https://phabricator.wikimedia.org/T423691) [19:01:39] (03PS5) 10SBassett: Content Security Policy: add stricter report-only policy [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344774 (https://phabricator.wikimedia.org/T419612) [19:02:08] !log btullis@cumin1004 START - Cookbook sre.dns.wipe-cache ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors [19:02:12] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-cache1001.eqiad.wmnet ml-cache1003.eqiad.wmnet on all recursors [19:03:25] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm [19:03:40] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm [19:04:03] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm [19:05:17] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:05:47] !log btullis@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1002.eqiad.wmnet with OS bookworm [19:06:19] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:06:41] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:07:18] !log btullis@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ml-cache1003.eqiad.wmnet with OS bookworm [19:08:03] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:08:18] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:08:22] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:08:25] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:09:01] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [19:09:07] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [19:09:13] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:10:19] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm [19:10:31] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache1001.eqiad.wmnet with OS bookworm [19:10:51] (03Merged) 10jenkins-bot: ulsfo/eqsin - remove dhcp_relay_ra [homer/public] - 10https://gerrit.wikimedia.org/r/1346028 (owner: 10Ayounsi) [19:13:11] (03PS1) 10Lerickson: Allow ingress on the qlever port, too. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346112 (https://phabricator.wikimedia.org/T439449) [19:14:39] (03CR) 10DLynch: [C:03+2] editcheck-headless: Declare the TLS port as gRPC [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345648 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:14:43] (03CR) 10DLynch: [C:03+2] python-webapp: Update mesh.service to 1.2.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345647 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:14:47] (03CR) 10DLynch: [C:03+2] mesh.service: Add public_port_app_protocol [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345646 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:14:51] (03CR) 10DLynch: [C:03+2] mesh: Copy service 1.2.1 to 1.2.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345645 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:15:09] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:15:29] (03CR) 10CI reject: [V:04-1] Allow ingress on the qlever port, too. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346112 (https://phabricator.wikimedia.org/T439449) (owner: 10Lerickson) [19:15:58] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1001.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:17:32] (03Merged) 10jenkins-bot: mesh: Copy service 1.2.1 to 1.2.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345645 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:17:42] (03Merged) 10jenkins-bot: mesh.service: Add public_port_app_protocol [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345646 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:17:46] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache1003.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:18:29] (03Merged) 10jenkins-bot: python-webapp: Update mesh.service to 1.2.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345647 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:18:31] (03Merged) 10jenkins-bot: editcheck-headless: Declare the TLS port as gRPC [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345648 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:18:37] PROBLEM - Check unit status of statograph_post on alert1002 is CRITICAL: CRITICAL: Status of the systemd unit statograph_post https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [19:18:51] PROBLEM - Host titan1002 is DOWN: PING CRITICAL - Packet loss = 100% [19:19:03] btullis@cumin1004 provision (PID 1651656) is awaiting input [19:19:10] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host ml-cache1003.eqiad.wmnet with OS bookworm [19:22:19] RECOVERY - Host titan1002 is UP: PING OK - Packet loss = 0%, RTA = 0.18 ms [19:22:27] (03PS2) 10Lerickson: Allow ingress on the qlever port, too. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346112 (https://phabricator.wikimedia.org/T439449) [19:22:40] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host ml-cache1001.eqiad.wmnet with OS bookworm [19:23:41] !log kemayo@deploy1003 helmfile [staging] START helmfile.d/services/editcheck-headless: apply [19:23:42] btullis@cumin1004 provision (PID 1651656) is awaiting input [19:24:01] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [19:24:03] !log kemayo@deploy1003 helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply [19:26:09] (03PS1) 10Xcollazo: analytics: refinery: Keep File Exports for 170 days [puppet] - 10https://gerrit.wikimedia.org/r/1346119 (https://phabricator.wikimedia.org/T437463) [19:27:54] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [19:28:37] RECOVERY - Check unit status of statograph_post on alert1002 is OK: OK: Status of the systemd unit statograph_post https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [19:28:57] (03CR) 10DLynch: Update mesh.{configuration,networkpolicy} and dependents in all charts (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338030 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [19:29:07] (03CR) 10CI reject: [V:04-1] analytics: refinery: Keep File Exports for 170 days [puppet] - 10https://gerrit.wikimedia.org/r/1346119 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [19:29:15] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [19:30:17] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [19:31:57] (03PS3) 10Lerickson: Allow ingress on the qlever port, too. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346112 (https://phabricator.wikimedia.org/T439449) [19:33:24] (03CR) 10Btullis: analytics: refinery: Keep File Exports for 170 days (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1346119 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [19:33:31] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage [19:33:52] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host ml-cache1002.eqiad.wmnet with OS bookworm [19:34:01] 06SRE, 10DNS, 06Fundraising-Backlog, 07fr-combowiki, and 2 others: Investigate having combowiki on donate.wikipedia.org - https://phabricator.wikimedia.org/T433108#12376605 (10Ejegg) [19:37:04] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1003.eqiad.wmnet with reason: host reimage [19:38:01] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage [19:38:41] (03CR) 10Btullis: dumps: web: Add a brownout for v1 meta-current and meta-history (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1346007 (https://phabricator.wikimedia.org/T437461) (owner: 10Btullis) [19:39:53] 06SRE, 10DNS, 06Fundraising-Backlog, 10fundraising-tech-ops, and 3 others: Investigate having combowiki on donate.wikipedia.org - https://phabricator.wikimedia.org/T433108#12376635 (10Ejegg) [19:39:59] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [19:40:15] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [19:40:36] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1001.eqiad.wmnet with reason: host reimage [19:44:25] (03PS7) 10Dduvall: buildx: New driver for building images via Buildx/BuildKit [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958) [19:44:48] (03CR) 10Btullis: [C:03+2] dse-k8s-eqiad: Remove the old Pod IP range [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345973 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [19:47:38] (03PS1) 10Btullis: dse-k8s-eqiad: Add dse-k8s-worker104[2-4] to site.pp and preseed.yaml [puppet] - 10https://gerrit.wikimedia.org/r/1346130 (https://phabricator.wikimedia.org/T438058) [19:48:38] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage [19:50:38] 10SRE-SLO, 06SRE Observability: Support SLOs based on Test Kitchen instruments - https://phabricator.wikimedia.org/T438274#12376686 (10mpopov) > (Side note: Test Kitchen is designed for collecting data about //experiments//, and as such requires every instrument to have a pre-planned end date, whereas SLOs are... [19:51:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [19:52:05] (03CR) 10DLynch: "Since the editcheck-headless is actually deployed now." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327632 (https://phabricator.wikimedia.org/T431499) (owner: 10Eevans) [19:52:42] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" [19:53:20] !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_eqsin - 7.1.1-2~bpo13+wmf3 () [19:53:34] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache1002.eqiad.wmnet with reason: host reimage [19:53:50] (03CR) 10Dduvall: "There was quite a bit of cruft from the other patch still, but it should be good now. I also implemented a return structure for `build_all" [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958) (owner: 10Dduvall) [19:54:46] (03PS1) 10Dzahn: zuul: turn physical machine zuul1005 into an executor [puppet] - 10https://gerrit.wikimedia.org/r/1346136 (https://phabricator.wikimedia.org/T427353) [19:55:13] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" [19:55:13] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1003.eqiad.wmnet with OS bookworm [19:55:31] (03Merged) 10jenkins-bot: dse-k8s-eqiad: Remove the old Pod IP range [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345973 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [19:55:58] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" [19:56:07] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [19:56:08] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [19:56:16] (03PS8) 10Dduvall: buildx: New driver for building images via Buildx/BuildKit [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958) [19:56:17] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [19:56:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [19:57:06] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [19:58:58] (03PS1) 10Dzahn: zuul: decom VM zuul1001 as a main zuul, keep zuul2001 as is [puppet] - 10https://gerrit.wikimedia.org/r/1346139 (https://phabricator.wikimedia.org/T427353) [19:59:01] btullis@cumin1004 reimage (PID 1656392) is awaiting input [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: gettimeofday() says it's time for UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T2000) [20:00:05] sbassett, aranyap, sbassett, Southparkfan, and cklimas: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:19] o/ [20:00:49] o/ [20:01:52] (03PS1) 10Eevans: linked-artifacts: provision edit_suggestions_counts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346140 (https://phabricator.wikimedia.org/T431499) [20:01:57] Hi [20:02:07] (03PS1) 10Aghirelli: git::clone: add optional sparse-checkout support [puppet] - 10https://gerrit.wikimedia.org/r/1343101 [20:02:13] (03PS1) 10Aghirelli: eventschemas: sparse-checkout the mediawiki repo to its mediawiki/ dir [puppet] - 10https://gerrit.wikimedia.org/r/1343109 [20:03:03] happy to go after everyone else--I'll probably be a little slow with my deploy. [20:03:28] ok, i should have a quick config deploy... [20:04:08] (03CR) 10SBassett: [C:03+1] Content Security Policy: add stricter report-only policy [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344774 (https://phabricator.wikimedia.org/T419612) (owner: 10SBassett) [20:04:12] (03CR) 10SBassett: [C:03+1] Remove foundationwiki's CSP report-only header [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345649 (https://phabricator.wikimedia.org/T423691) (owner: 10SBassett) [20:04:17] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" [20:04:17] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1001.eqiad.wmnet with OS bookworm [20:04:32] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbassett@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344774 (https://phabricator.wikimedia.org/T419612) (owner: 10SBassett) [20:04:32] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbassett@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345649 (https://phabricator.wikimedia.org/T423691) (owner: 10SBassett) [20:05:49] (03Merged) 10jenkins-bot: Content Security Policy: add stricter report-only policy [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344774 (https://phabricator.wikimedia.org/T419612) (owner: 10SBassett) [20:07:19] (03CR) 10DLynch: "See this merge request: https://gitlab.wikimedia.org/repos/sre/hoarde/-/merge_requests/14" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327632 (https://phabricator.wikimedia.org/T431499) (owner: 10Eevans) [20:08:24] !log sbassett@deploy1003 Started scap sync-world: Backport for [[gerrit:1344774|Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649|Remove foundationwiki's CSP report-only header (T423691)]] [20:08:34] T431144: CSP: Unlock loading data and block loading scripts for execution - https://phabricator.wikimedia.org/T431144 [20:08:35] T327588: Consider adding more CSP directives to MediaWIki core - https://phabricator.wikimedia.org/T327588 [20:08:35] T245424: Consider adding base-uri 'none' to mediawiki csp policy - https://phabricator.wikimedia.org/T245424 [20:08:35] T423691: Should foundationwiki's custom (report-only) content-security-policy be removed? - https://phabricator.wikimedia.org/T423691 [20:09:27] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" [20:10:38] PROBLEM - MariaDB Replica Lag: s8 on db2198 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 621.95 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:10:52] PROBLEM - MariaDB Replica Lag: s1 on db2250 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 635.30 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:10:56] PROBLEM - MariaDB Replica Lag: s8 on db1285 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 641.11 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:11:26] stuff is seeming slow... [20:11:28] seeing a bunch of JsonConfig/jsontransform errors in logstash, but that should not have anything to do with what I’m deploying… [20:11:51] spike in response time since 14:50 UTC? [20:11:58] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" [20:11:58] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache1002.eqiad.wmnet with OS bookworm [20:12:11] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346130 (https://phabricator.wikimedia.org/T438058) (owner: 10Btullis) [20:12:32] !log sbassett@deploy1003 sbassett: Backport for [[gerrit:1344774|Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649|Remove foundationwiki's CSP report-only header (T423691)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:12:50] I can revery my config patches and re-deploy but I’m a bit shocked they would have this effect... [20:13:02] not sure it's you [20:13:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-int releases routed via main at eqiad: 16.2% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-int&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:13:28] https://grafana.wikimedia.org/goto/sq9v4p?orgId=default [20:13:36] that's sus [20:14:05] my config patches were 1) add a new csp report-only header to the top 10 wikis 2) remove foundationwiki’s csp report-only header [20:14:29] but right now I can barely load diffs [20:14:34] like 10+ seconds [20:14:53] (03CR) 10Ottomata: [C:03+1] linked-artifacts: provision edit_suggestions_counts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346140 (https://phabricator.wikimedia.org/T431499) (owner: 10Eevans) [20:15:03] oh, I have access control check errors in my console?? [20:15:12] maybe unrelated [20:15:52] PROBLEM - MariaDB Replica Lag: s1 on db1240 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 936.08 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:17:01] (03CR) 10Bking: [C:03+2] dse-k8s-eqiad: Add dse-k8s-worker104[2-4] to site.pp and preseed.yaml [puppet] - 10https://gerrit.wikimedia.org/r/1346130 (https://phabricator.wikimedia.org/T438058) (owner: 10Btullis) [20:19:16] !log sbassett@deploy1003 sbassett: Rolling back deployment [20:19:26] I’m rolling back just to be safe [20:20:16] (03PS1) 10SBassett: Revert "Content Security Policy: add stricter report-only policy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346149 [20:20:31] (03PS1) 10SBassett: Revert "Remove foundationwiki's CSP report-only header" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346150 [20:21:24] FIRING: [3x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:21:27] Yeah idk. I'm seeing the updated CSP header, but the access control errors seem to have only occurred on two loads for me. Diffs are still loading very slowly though. (20+ seconds on one) [20:22:19] I rolled back and the error rate has declined. Which makes no sense to me, since my config didn’t even touch euwiki, where all of the errors occurred. [20:22:31] !log sbassett@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344774|Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)]], [[gerrit:1345649|Remove foundationwiki's CSP report-only header (T423691)]] (duration: 14m 07s) [20:22:40] T431144: CSP: Unlock loading data and block loading scripts for execution - https://phabricator.wikimedia.org/T431144 [20:22:40] (03CR) 10SBassett: [C:03+2] Revert "Content Security Policy: add stricter report-only policy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346149 (owner: 10SBassett) [20:22:40] T327588: Consider adding more CSP directives to MediaWIki core - https://phabricator.wikimedia.org/T327588 [20:22:41] T245424: Consider adding base-uri 'none' to mediawiki csp policy - https://phabricator.wikimedia.org/T245424 [20:22:41] T423691: Should foundationwiki's custom (report-only) content-security-policy be removed? - https://phabricator.wikimedia.org/T423691 [20:22:44] (03CR) 10SBassett: [C:03+2] Revert "Remove foundationwiki's CSP report-only header" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346150 (owner: 10SBassett) [20:23:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-int releases routed via main at eqiad: 13.11% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-int&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:23:35] (03Merged) 10jenkins-bot: Revert "Content Security Policy: add stricter report-only policy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346149 (owner: 10SBassett) [20:23:43] (03Merged) 10jenkins-bot: Revert "Remove foundationwiki's CSP report-only header" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346150 (owner: 10SBassett) [20:24:35] I landed reverts of the config patches as well, so this doesn’t happen on the next deployment. [20:25:31] Part of me thinks this could all be coincidence with something transient but I've seen CSP break things in weirder ways... [20:25:41] So… I guess I’m done for now, since I’ll have to investigate why this might have happened. [20:26:17] We’ve set report-only policies on more production wikis in the past, and didn’t really see an issue like this. [20:26:48] ah, word [20:27:22] (03Abandoned) 10Eevans: linked-artifacts: add edit_suggestions_counts config (staging) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327632 (https://phabricator.wikimedia.org/T431499) (owner: 10Eevans) [20:27:50] cklimas: et al, feel free to deploy :) [20:28:26] Southparkfan: any pref on who goes next? [20:28:52] (I also have a config change today so I won't feel quite as cursed if mine triggers errors also...) [20:29:36] sbassett: can you peek at request 3fddf606-9e5a-40a0-b76c-da738519b90a? 20 seconds to load a diff, albeit for a large page, definitely doesn't seem right [20:29:44] I don't have a preference, I can wait [20:29:52] ok, I'll start mine. [20:29:57] I just need somebody to deploy my patch, because I do not have deployment access [20:30:02] Go ahead [20:30:32] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cklimas@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346087 (https://phabricator.wikimedia.org/T434258) (owner: 10Cklimas) [20:31:32] (03Merged) 10jenkins-bot: InitialiseSettings: Enable return-to-app banner on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346087 (https://phabricator.wikimedia.org/T434258) (owner: 10Cklimas) [20:32:21] perryprog: nothing obviously weird with that reqId. my working theory right now is that my csp report-only config change exhausted the action API in some strange ways. [20:32:25] sbassett: hmm, saw this message: "There were unexpected commits pulled from origin for /srv/mediawiki-staging." and it lists your reverts. I'm guessing this is normal, but just want to confirm? [20:32:47] !log bking@cumin2003 START - Cookbook sre.hosts.rename from ml-cache1001 to dse-k8s-worker1042 [20:33:02] !log bking@cumin2003 START - Cookbook sre.dns.netbox [20:33:12] cklimas: yeah, i think so. since it was the wmf-config repo, I needed to revert those. so IS.php and CS.php should be back to where they were before my csp config changes. [20:33:19] cool, just making sure :) [20:33:50] !log cklimas@deploy1003 Started scap sync-world: Backport for [[gerrit:1346087|InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] [20:33:53] T434258: [REQUEST] Add a blue banner to visual editor after publishing or abandoning an app edit - https://phabricator.wikimedia.org/T434258 [20:38:02] !log cklimas@deploy1003 cklimas: Backport for [[gerrit:1346087|InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:38:31] bking@cumin2003 rename (PID 1799595) is awaiting input [20:42:02] not seeing an increase in error rates, so going to proceed... [20:42:36] (03CR) 10Dzahn: [C:03+2] zuul: turn physical machine zuul1005 into an executor [puppet] - 10https://gerrit.wikimedia.org/r/1346136 (https://phabricator.wikimedia.org/T427353) (owner: 10Dzahn) [20:42:53] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003" [20:43:02] !log cklimas@deploy1003 cklimas: Continuing with deployment [20:44:37] (03CR) 10Ryan Kemper: [C:03+2] aptrepo: import kafka-configurator from apt-staging [puppet] - 10https://gerrit.wikimedia.org/r/1345852 (https://phabricator.wikimedia.org/T435282) (owner: 10Ryan Kemper) [20:45:20] 10ops-eqiad, 06SRE, 06DC-Ops, 06Machine-Learning-Team, and 2 others: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12376938 (10RobH) a:05RobH→03None [20:45:35] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1001 to dse-k8s-worker1042 - bking@cumin2003" [20:45:35] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [20:45:35] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache dse-k8s-worker1042 on all recursors [20:45:39] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1042 on all recursors [20:45:40] !log bking@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1042 [20:46:42] !log bking@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1042 [20:47:19] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1001 to dse-k8s-worker1042 [20:48:22] !log cklimas@deploy1003 Finished scap sync-world: Backport for [[gerrit:1346087|InitialiseSettings: Enable return-to-app banner on testwiki (T434258)]] (duration: 14m 32s) [20:48:24] T434258: [REQUEST] Add a blue banner to visual editor after publishing or abandoning an app edit - https://phabricator.wikimedia.org/T434258 [20:48:37] OK, all done here. [20:50:00] Could you deploy my patch as well? [20:51:11] I am pretty new at deploying so I'm probably not your best bet here, but if there's no one else available, I can take a stab at it. [20:51:27] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm [20:53:35] OK, it looks like I'm your guy. I'll go ahead and start it. [20:53:57] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cklimas@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344405 (https://phabricator.wikimedia.org/T436480) (owner: 10Southparkfan) [20:54:53] (03Merged) 10jenkins-bot: Beta Cluster: Use different Redis instance for LockManager [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344405 (https://phabricator.wikimedia.org/T436480) (owner: 10Southparkfan) [20:55:17] !log cklimas@deploy1003 Started scap sync-world: Backport for [[gerrit:1344405|Beta Cluster: Use different Redis instance for LockManager (T436480)]] [20:55:20] T436480: Replace deployment-rdb01 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436480 [20:55:23] Very grateful :) [20:59:36] !log cklimas@deploy1003 cklimas, southparkfan: Backport for [[gerrit:1344405|Beta Cluster: Use different Redis instance for LockManager (T436480)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:59:54] Alright, I'll take a look [21:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260929T2100) [21:01:59] (03PS1) 10JHathaway: cumin: update deprecated stdlib function name [puppet] - 10https://gerrit.wikimedia.org/r/1346160 [21:01:59] To the extent I can see, looks fine to me on k8s-mwdebug [21:02:14] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346160 (owner: 10JHathaway) [21:02:19] OK, so you're comfortable with proceeding? [21:02:35] Yes, I am [21:02:39] !log cklimas@deploy1003 cklimas, southparkfan: Continuing with deployment [21:02:41] (03PS2) 10Eevans: linked-artifacts: provision edit_suggestions_counts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346140 (https://phabricator.wikimedia.org/T431499) [21:03:37] !log bking@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] [21:03:40] !log bking@cumin2003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] [21:03:57] !log bking@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] [21:04:00] !log bking@cumin2003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1002.eqiad.wmnet'] [21:06:43] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage [21:06:55] !log bking@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet'] [21:06:58] !log bking@cumin2003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ml-cache1003.eqiad.wmnet'] [21:07:45] (03PS1) 10Dzahn: zuul: fix resource ordering so the first puppet run succeeds [puppet] - 10https://gerrit.wikimedia.org/r/1346161 (https://phabricator.wikimedia.org/T395938) [21:08:02] !log cklimas@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344405|Beta Cluster: Use different Redis instance for LockManager (T436480)]] (duration: 12m 45s) [21:08:05] T436480: Replace deployment-rdb01 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436480 [21:08:21] Southparkfan: believe it's done now. [21:09:03] Thank you! [21:09:59] (03PS1) 10Eevans: linked-artifacts: upgrade image to v1.5.1 (staging) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346162 (https://phabricator.wikimedia.org/T431499) [21:10:24] (03CR) 10CI reject: [V:04-1] zuul: fix resource ordering so the first puppet run succeeds [puppet] - 10https://gerrit.wikimedia.org/r/1346161 (https://phabricator.wikimedia.org/T395938) (owner: 10Dzahn) [21:11:30] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1042.eqiad.wmnet with reason: host reimage [21:13:24] (03CR) 10Eevans: [C:03+2] linked-artifacts: upgrade image to v1.5.1 (staging) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346162 (https://phabricator.wikimedia.org/T431499) (owner: 10Eevans) [21:14:23] (03PS3) 10Eevans: linked-artifacts: provision edit_suggestions_counts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346140 (https://phabricator.wikimedia.org/T431499) [21:14:55] RECOVERY - MariaDB Replica Lag: s8 on db1285 is OK: OK slave_sql_lag Replication lag: 48.97 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:15:39] RECOVERY - MariaDB Replica Lag: s8 on db2198 is OK: OK slave_sql_lag Replication lag: 0.15 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:15:47] (03PS2) 10Dzahn: zuul: fix resource ordering so the first puppet run succeeds [puppet] - 10https://gerrit.wikimedia.org/r/1346161 (https://phabricator.wikimedia.org/T395938) [21:16:04] !log eevans@deploy1003 helmfile [staging] START helmfile.d/services/linked-artifacts: apply [21:16:20] !log eevans@deploy1003 helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply [21:17:29] (03CR) 10Eevans: [C:03+2] linked-artifacts: provision edit_suggestions_counts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346140 (https://phabricator.wikimedia.org/T431499) (owner: 10Eevans) [21:19:08] (03Merged) 10jenkins-bot: linked-artifacts: provision edit_suggestions_counts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346140 (https://phabricator.wikimedia.org/T431499) (owner: 10Eevans) [21:20:23] !log eevans@deploy1003 helmfile [staging] START helmfile.d/services/linked-artifacts: apply [21:20:34] !log eevans@deploy1003 helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply [21:22:03] (03PS2) 10JHathaway: cumin: update deprecated stdlib function name [puppet] - 10https://gerrit.wikimedia.org/r/1346160 [21:22:06] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346160 (owner: 10JHathaway) [21:22:49] (03CR) 10Ryan Kemper: [C:03+2] "my last patchset just changed code comments which removed the +1; merging this now" [puppet] - 10https://gerrit.wikimedia.org/r/1318976 (https://phabricator.wikimedia.org/T276088) (owner: 10Ryan Kemper) [21:24:15] (03PS1) 10Dzahn: zuul: decom VM zuul1002, replaced by zuul1005, keep zuul2001 [puppet] - 10https://gerrit.wikimedia.org/r/1346164 (https://phabricator.wikimedia.org/T427353) [21:24:32] (03PS2) 10Dzahn: zuul: decom VM zuul1002, replaced by zuul1005, keep zuul2002 [puppet] - 10https://gerrit.wikimedia.org/r/1346164 (https://phabricator.wikimedia.org/T427353) [21:25:39] PROBLEM - MariaDB Replica Lag: x3 on db2200 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 651.01 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:27:20] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet with OS bookworm [21:29:01] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [21:29:04] !log decom'ing VM zuul1002 - replaced by physical machine zuul1005 - T427353 T395938 [21:29:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:29:09] T427353: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353 [21:29:10] T395938: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938 [21:30:52] (03CR) 10JHathaway: [C:03+2] cumin: update deprecated stdlib function name [puppet] - 10https://gerrit.wikimedia.org/r/1346160 (owner: 10JHathaway) [21:35:44] (03PS1) 10JHathaway: puppet-agent: add option to send legacy facts [puppet] - 10https://gerrit.wikimedia.org/r/1346165 (https://phabricator.wikimedia.org/T372666) [21:35:58] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346165 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:40:39] RECOVERY - MariaDB Replica Lag: x3 on db2200 is OK: OK slave_sql_lag Replication lag: 0.08 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:41:51] (03CR) 10JHathaway: "ready for review, initial rollout should effectively be a noop, though we add the config option." [puppet] - 10https://gerrit.wikimedia.org/r/1346165 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:47:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [21:47:53] RECOVERY - MariaDB Replica Lag: s1 on db2250 is OK: OK slave_sql_lag Replication lag: 0.43 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:48:55] PROBLEM - MariaDB Replica Lag: x3 on db1216 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 609.11 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:49:01] FIRING: SystemdUnitFailed: wmf_auto_restart_krb5-admin-server.service on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:51:53] RECOVERY - MariaDB Replica Lag: s1 on db1240 is OK: OK slave_sql_lag Replication lag: 0.16 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:55:05] !log bking@cumin2003 START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts [21:55:30] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts [21:55:42] !log bking@cumin2003 START - Cookbook sre.hosts.bmc-user-mgmt for 1 hosts [21:56:08] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1 hosts [21:56:23] !log bking@cumin2003 START - Cookbook sre.hosts.rename from ml-cache1002 to dse-k8s-worker1043 [21:56:48] !log bking@cumin2003 START - Cookbook sre.dns.netbox [21:56:53] !log bking@cumin2003 START - Cookbook sre.hosts.rename from ml-cache1003 to dse-k8s-worker1044 [21:57:56] 06SRE, 10SRE-Access-Requests: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644 (10eniroomand) 03NEW [21:59:55] RECOVERY - MariaDB Replica Lag: x3 on db1216 is OK: OK slave_sql_lag Replication lag: 0.03 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:01:07] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003" [22:01:23] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1002 to dse-k8s-worker1043 - bking@cumin2003" [22:01:23] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [22:01:24] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache dse-k8s-worker1043 on all recursors [22:01:27] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1043 on all recursors [22:01:28] !log bking@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1043 [22:01:55] !log bking@cumin2003 START - Cookbook sre.dns.netbox [22:02:43] !log bking@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1043 [22:03:20] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1002 to dse-k8s-worker1043 [22:03:27] (03CR) 10BCornwall: [C:04-1] "Thanks! This is great work." [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) (owner: 10Daniel Kertesz) [22:03:56] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm [22:05:47] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003" [22:05:51] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache1003 to dse-k8s-worker1044 - bking@cumin2003" [22:05:51] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [22:05:52] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache dse-k8s-worker1044 on all recursors [22:05:55] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1044 on all recursors [22:05:56] !log bking@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1044 [22:06:12] !log bking@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1044 [22:06:49] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache1003 to dse-k8s-worker1044 [22:07:12] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm [22:14:28] PROBLEM - MariaDB Replica Lag: s4 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 611.44 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:17:41] 06SRE: Bing Webmaster Tools access request for Ash Han - https://phabricator.wikimedia.org/T439646 (10AHan-WMF) 03NEW [22:19:01] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage [22:21:46] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage [22:23:48] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1043.eqiad.wmnet with reason: host reimage [22:24:28] PROBLEM - MariaDB Replica Lag: s4 on db1265 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 636.88 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:26:18] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1044.eqiad.wmnet with reason: host reimage [22:34:01] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [22:39:29] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet with OS bookworm [22:42:19] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet with OS bookworm [23:22:00] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: Standardize management routers interfaces - https://phabricator.wikimedia.org/T421674#12377400 (10Papaul) [23:24:26] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [23:38:57] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1346186 [23:38:57] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1346186 (owner: 10TrainBranchBot) [23:43:23] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for Tsevener - https://phabricator.wikimedia.org/T439138#12377485 (10thcipriani) Reason for access makes sense for `deployment` group membership. Approved! --- @Tsevener some additional info for you: - For MediaWiki deploys, you'll want access t... [23:46:28] RECOVERY - MariaDB Replica Lag: s4 on db2239 is OK: OK slave_sql_lag Replication lag: 0.08 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [23:46:40] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1346186 (owner: 10TrainBranchBot) [23:48:44] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users and deployment for lwatson - https://phabricator.wikimedia.org/T438926#12377518 (10thcipriani) Reason for access makes sense for `deployment` group membership. Approved! --- @lwatson some additional info for you: - If you ever ne...