[00:24:15] PROBLEM - SSH on arclamp2001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring [00:26:05] RECOVERY - SSH on arclamp2001 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u7 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [00:40:43] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:45:15] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [00:46:15] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [00:49:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.73% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [00:50:43] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:09:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.35% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:12:14] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1310685 [01:12:14] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1310685 (owner: 10TrainBranchBot) [01:12:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.08% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:14:02] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [01:15:26] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [01:19:13] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1310685 (owner: 10TrainBranchBot) [01:27:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.31% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [01:40:43] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:45:43] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:50:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [01:50:43] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [01:54:25] FIRING: [12x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [02:00:22] !log mwpresync@deploy2003 Started scap build-images: Publishing wmf/next image [02:07:00] !log mwpresync@deploy2003 Finished scap build-images: Publishing wmf/next image (duration: 06m 37s) [02:10:43] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:15:43] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:29:45] FIRING: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@eqiad is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [02:39:45] RESOLVED: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@eqiad is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [02:46:45] FIRING: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@eqiad is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [02:56:45] RESOLVED: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@eqiad is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [03:31:45] FIRING: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@eqiad is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [03:41:45] RESOLVED: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@eqiad is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [03:50:43] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:51:11] (03PS4) 10Andrea Denisse: WIP [DNM] Add SQLite-backed incident store and analytics CLI [software/klaxon] - 10https://gerrit.wikimedia.org/r/1274026 (https://phabricator.wikimedia.org/T431101) (owner: 10CDanis) [03:52:32] (03CR) 10CI reject: [V:04-1] WIP [DNM] Add SQLite-backed incident store and analytics CLI [software/klaxon] - 10https://gerrit.wikimedia.org/r/1274026 (https://phabricator.wikimedia.org/T431101) (owner: 10CDanis) [04:17:17] (03PS5) 10BCornwall: varnish: Configure text for thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/1306896 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [04:17:52] (03CR) 10CI reject: [V:04-1] varnish: Configure text for thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/1306896 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [04:18:10] (03CR) 10BCornwall: [V:04-1] "Initial porting of tests done, but currently failing tests:" [puppet] - 10https://gerrit.wikimedia.org/r/1306896 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [05:45:43] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [05:50:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 17.08% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:50:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:54:25] FIRING: [12x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [06:00:04] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260715T0600) [06:07:22] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-eqord:xe-0/1/3 (Transport: cr3-ulsfo:xe-0/1/1 (Arelion, IC-313592 51ms 10Gbps wave) {#11372}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [06:28:33] (03CR) 10Ayounsi: [C:03+1] "that's great, thx!" [homer/public] - 10https://gerrit.wikimedia.org/r/1309707 (https://phabricator.wikimedia.org/T431849) (owner: 10Cathal Mooney) [06:28:45] FIRING: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@eqiad is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [06:38:45] RESOLVED: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@eqiad is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [06:45:19] 10SRE-swift-storage, 06MediaWiki-Media-Platform-Team, 10MediaWiki-Uploading, 07Wikimedia-production-error: MediaWiki\Upload\Exception\UploadChunkFileException: Error storing file in '{chunkPath}': backend-fail-internal; local-swift-codfw - https://phabricator.wikimedia.org/T430986#12122483 (10Samwilson) a:... [06:54:22] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 15 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [core] (wmf/1.47.0-wmf.11) - 10https://gerrit.wikimedia.org/r/1310470 (https://phabricator.wikimedia.org/T339291) (owner: 10Kosta Harlan) [07:00:04] Amir1, urbanecm, and awight: UTC morning backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260715T0700). Please do the needful. [07:00:04] kostajh: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:00:31] hi [07:00:45] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy2003 using scap backport" [core] (wmf/1.47.0-wmf.11) - 10https://gerrit.wikimedia.org/r/1310470 (https://phabricator.wikimedia.org/T339291) (owner: 10Kosta Harlan) [07:05:27] (03Merged) 10jenkins-bot: SiteStats: Add temporary account columns to site_stats [core] (wmf/1.47.0-wmf.11) - 10https://gerrit.wikimedia.org/r/1310470 (https://phabricator.wikimedia.org/T339291) (owner: 10Kosta Harlan) [07:06:06] !log kharlan@deploy2003 Started scap sync-world: Backport for [[gerrit:1310470|SiteStats: Add temporary account columns to site_stats (T339291)]] [07:06:10] T339291: Should temp users be counted as registered & active users on Special:Statistics? - https://phabricator.wikimedia.org/T339291 [07:08:11] !log kharlan@deploy2003 kharlan: Backport for [[gerrit:1310470|SiteStats: Add temporary account columns to site_stats (T339291)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:09:33] !log kharlan@deploy2003 kharlan: Continuing with deployment [07:13:54] !log kharlan@deploy2003 Finished scap sync-world: Backport for [[gerrit:1310470|SiteStats: Add temporary account columns to site_stats (T339291)]] (duration: 07m 48s) [07:13:58] T339291: Should temp users be counted as registered & active users on Special:Statistics? - https://phabricator.wikimedia.org/T339291 [07:19:09] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-fe2013.codfw.wmnet with OS trixie [07:19:22] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12122547 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-fe2013.codfw.wmnet with OS trixie [07:19:27] !log mvernon@cumin2003 START - Cookbook sre.hosts.move-vlan for host ms-fe2013 [07:19:45] !log mvernon@cumin2003 START - Cookbook sre.dns.netbox [07:20:20] !log mvernon@cumin1003 START - Cookbook sre.hosts.reimage for host ms-fe1013.eqiad.wmnet with OS trixie [07:20:34] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12122550 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-fe1013.eqiad.wmnet with OS trixie [07:20:59] (03CR) 10Ayounsi: [C:03+1] "Perfecto!" [homer/public] - 10https://gerrit.wikimedia.org/r/1309643 (https://phabricator.wikimedia.org/T423430) (owner: 10Cathal Mooney) [07:21:44] FIRING: KubernetesDeploymentUnavailableReplicas: ... [07:21:44] Deployment mw-web.eqiad.main in mw-web at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=mw-web&var-deployment=mw-web.eqiad.main - ... [07:21:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [07:24:22] !log mvernon@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2013 - mvernon@cumin2003" [07:24:27] !log mvernon@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2013 - mvernon@cumin2003" [07:24:27] !log mvernon@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [07:24:28] !log mvernon@cumin2003 START - Cookbook sre.dns.wipe-cache ms-fe2013.codfw.wmnet 87.0.192.10.in-addr.arpa 7.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [07:24:31] !log mvernon@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-fe2013.codfw.wmnet 87.0.192.10.in-addr.arpa 7.8.0.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [07:24:32] !log mvernon@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host ms-fe2013 [07:26:44] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [07:26:44] Deployment mw-web.eqiad.main in mw-web at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=mw-web&var-deployment=mw-web.eqiad.main - ... [07:26:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [07:27:36] mvernon@cumin2003 reimage (PID 1628573) is awaiting input [07:30:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.76% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:30:45] !log root@cumin1003 START - Cookbook sre.mysql.depool depool pc1021: Security updates [07:30:46] !log root@cumin1003 START - Cookbook sre.mysql.parsercache [07:30:53] (03CR) 10Ayounsi: Pometheus nodes: adjust message on server_depool key (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1310588 (https://phabricator.wikimedia.org/T327300) (owner: 10Cathal Mooney) [07:30:54] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [07:30:54] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1021: Security updates [07:31:28] !log mvernon@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-fe2013 [07:31:29] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-fe2013 [07:31:51] (03CR) 10Elukey: [V:03+2 C:03+2] redfish: add delete_account method [software/spicerack] - 10https://gerrit.wikimedia.org/r/1310131 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [07:34:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.9% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:35:16] 10SRE-swift-storage, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: RdfStreamingUpdaterSpaceUsageTooHigh - https://phabricator.wikimedia.org/T431506#12122574 (10MatthewVernon) [07:36:23] PROBLEM - MariaDB Replica IO: pc1 on pc2021 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@pc1021.eqiad.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on pc1021.eqiad.wmnet (111 Connection refused) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [07:36:44] !log mvernon@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1013.eqiad.wmnet with reason: host reimage [07:38:23] RECOVERY - MariaDB Replica IO: pc1 on pc2021 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [07:40:13] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1013.eqiad.wmnet with reason: host reimage [07:45:43] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:46:42] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host bast1004.wikimedia.org [07:47:13] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2013.codfw.wmnet with reason: host reimage [07:50:43] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:52:52] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1004.wikimedia.org [07:53:00] !log root@cumin1003 START - Cookbook sre.mysql.pool pool pc1021: Security updates [07:53:00] !log root@cumin1003 START - Cookbook sre.mysql.parsercache [07:53:13] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [07:53:13] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1021: Security updates [07:53:29] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2013.codfw.wmnet with reason: host reimage [07:55:43] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:58:44] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1013.eqiad.wmnet with OS trixie [07:58:50] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12122622 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-fe1013.eqiad.wmnet with OS trixie completed: - ms-fe1013 (**PASS*... [07:58:54] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host bast5005.wikimedia.org [08:00:04] jeena and hashar: Deploy window MediaWiki train - Utc-7+Utc-0 Version (secondary timeslot) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260715T0800) [08:04:10] (03CR) 10Ayounsi: [C:03+1] "one nit, but lgtm!" [puppet] - 10https://gerrit.wikimedia.org/r/1310566 (https://phabricator.wikimedia.org/T430930) (owner: 10Effie Mouzeli) [08:04:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.72% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:05:09] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast5005.wikimedia.org [08:06:12] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org [08:07:55] !log mvernon@cumin1003 START - Cookbook sre.hosts.reimage for host ms-fe1012.eqiad.wmnet with OS trixie [08:08:03] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12122648 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-fe1012.eqiad.wmnet with OS trixie [08:09:19] !log root@cumin1003 START - Cookbook sre.mysql.depool depool pc1022: Security updates [08:09:19] !log root@cumin1003 START - Cookbook sre.mysql.parsercache [08:09:27] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [08:09:27] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1022: Security updates [08:10:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.86% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:12:05] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host krb2002.codfw.wmnet [08:12:16] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org [08:13:24] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org [08:14:45] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2013.codfw.wmnet with OS trixie [08:14:54] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12122666 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-fe2013.codfw.wmnet with OS trixie completed: - ms-fe2013 (**WARN*... [08:19:29] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org [08:19:49] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb2002.codfw.wmnet [08:21:24] (03PS1) 10MVernon: thanos: enable an expirer host for the thanos-swift cluster [puppet] - 10https://gerrit.wikimedia.org/r/1310971 (https://phabricator.wikimedia.org/T431506) [08:21:59] (03CR) 10CI reject: [V:04-1] thanos: enable an expirer host for the thanos-swift cluster [puppet] - 10https://gerrit.wikimedia.org/r/1310971 (https://phabricator.wikimedia.org/T431506) (owner: 10MVernon) [08:22:00] !log mvernon@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1012.eqiad.wmnet with reason: host reimage [08:25:45] (03CR) 10Filippo Giunchedi: Pometheus nodes: adjust message on server_depool key (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1310588 (https://phabricator.wikimedia.org/T327300) (owner: 10Cathal Mooney) [08:25:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.59% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:25:47] (03PS2) 10MVernon: thanos: enable an expirer host for the thanos-swift cluster [puppet] - 10https://gerrit.wikimedia.org/r/1310971 (https://phabricator.wikimedia.org/T431506) [08:26:00] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host krb1002.eqiad.wmnet [08:26:29] (03CR) 10Elukey: [C:03+2] Revert "Failover url-dowloaders in eqiad and codfw" [dns] - 10https://gerrit.wikimedia.org/r/1310578 (owner: 10Elukey) [08:26:50] !log mvernon@cumin2003 START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw [08:26:58] 07sre-alert-triage, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 06Machine-Learning-Team (Q1 FY2026-27): Alert in need of triage: SmartNotHealthy (instance ml-serve1001:9100) - https://phabricator.wikimedia.org/T414969#12122700 (10Gehel) [08:26:59] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team: hw troubleshooting: failing disk in ml-serve1001.eqiad.wmnet - https://phabricator.wikimedia.org/T432105#12122701 (10Gehel) [08:27:37] !log elukey@dns1004 START - running authdns-update [08:28:25] !log root@cumin1003 START - Cookbook sre.mysql.depool depool pc1022: Security updates [08:28:25] !log root@cumin1003 START - Cookbook sre.mysql.parsercache [08:28:31] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [08:28:31] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1022: Security updates [08:29:26] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1012.eqiad.wmnet with reason: host reimage [08:30:04] !log elukey@dns1004 END - running authdns-update [08:30:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.52% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:31:19] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1002.eqiad.wmnet [08:35:01] !log mvernon@cumin2003 END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw [08:35:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.07% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:37:37] (03PS3) 10Cathal Mooney: Pometheus nodes: adjust message on server_depool key [puppet] - 10https://gerrit.wikimedia.org/r/1310588 (https://phabricator.wikimedia.org/T327300) [08:40:13] (03CR) 10Cathal Mooney: "Updated to link to the wikitech instructions as discussed on irc. Let me know what you think." [puppet] - 10https://gerrit.wikimedia.org/r/1310588 (https://phabricator.wikimedia.org/T327300) (owner: 10Cathal Mooney) [08:41:02] !log mvernon@cumin2003 conftool action : set/pooled=inactive; selector: name=ms-fe2013.codfw.wmnet [08:41:30] (03CR) 10MVernon: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1310971 (https://phabricator.wikimedia.org/T431506) (owner: 10MVernon) [08:42:11] !log mvernon@cumin2003 conftool action : set/pooled=yes; selector: name=ms-fe2013.codfw.wmnet [08:43:19] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-fe2012.codfw.wmnet with OS trixie [08:43:32] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12122747 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-fe2012.codfw.wmnet with OS trixie [08:44:00] !log mvernon@cumin2003 START - Cookbook sre.hosts.move-vlan for host ms-fe2012 [08:44:06] !log mvernon@cumin2003 START - Cookbook sre.dns.netbox [08:44:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.38% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:47:17] (03PS1) 10Ayounsi: pki::multirootca: clarify depool policy [puppet] - 10https://gerrit.wikimedia.org/r/1310973 (https://phabricator.wikimedia.org/T327300) [08:47:37] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1012.eqiad.wmnet with OS trixie [08:47:43] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12122768 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-fe1012.eqiad.wmnet with OS trixie completed: - ms-fe1012 (**PASS*... [08:49:03] (03CR) 10Brouberol: "Looks good! FWIW, I'm also seeing" [puppet] - 10https://gerrit.wikimedia.org/r/1310558 (https://phabricator.wikimedia.org/T432089) (owner: 10Elukey) [08:49:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.38% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:49:59] (03CR) 10Ayounsi: [C:03+1] "2 nits but lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1310588 (https://phabricator.wikimedia.org/T327300) (owner: 10Cathal Mooney) [08:50:06] !log mvernon@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2012 - mvernon@cumin2003" [08:50:11] !log mvernon@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2012 - mvernon@cumin2003" [08:50:11] !log mvernon@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [08:50:13] !log mvernon@cumin2003 START - Cookbook sre.dns.wipe-cache ms-fe2012.codfw.wmnet 44.48.192.10.in-addr.arpa 4.4.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [08:50:15] !log mvernon@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-fe2012.codfw.wmnet 44.48.192.10.in-addr.arpa 4.4.0.0.8.4.0.0.2.9.1.0.0.1.0.0.4.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [08:50:16] !log mvernon@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host ms-fe2012 [08:50:43] PROBLEM - Check unit status of statograph_post on alert1002 is CRITICAL: CRITICAL: Status of the systemd unit statograph_post https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [08:50:55] !log root@cumin1003 START - Cookbook sre.mysql.pool pool pc1022: Security updates [08:50:55] !log root@cumin1003 START - Cookbook sre.mysql.parsercache [08:50:57] !log mvernon@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-fe2012 [08:50:58] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-fe2012 [08:51:09] (03PS1) 10Filippo Giunchedi: dumps: do not remove legacy mountpoint directory on cloud vps [puppet] - 10https://gerrit.wikimedia.org/r/1310975 (https://phabricator.wikimedia.org/T411248) [08:51:09] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [08:51:09] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1022: Security updates [08:51:53] (03CR) 10Ayounsi: [C:03+2] Add profile::server_depool for centrallog and graphite [puppet] - 10https://gerrit.wikimedia.org/r/1310119 (https://phabricator.wikimedia.org/T327300) (owner: 10Ayounsi) [08:52:02] !log mvernon@cumin1003 START - Cookbook sre.hosts.reimage for host ms-fe1011.eqiad.wmnet with OS trixie [08:52:09] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12122799 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-fe1011.eqiad.wmnet with OS trixie [08:54:58] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [08:55:50] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [08:56:35] (03PS3) 10MVernon: thanos: enable an expirer host for the thanos-swift cluster [puppet] - 10https://gerrit.wikimedia.org/r/1310971 (https://phabricator.wikimedia.org/T431506) [08:57:05] (03CR) 10MVernon: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1310971 (https://phabricator.wikimedia.org/T431506) (owner: 10MVernon) [08:57:09] (03CR) 10CI reject: [V:04-1] thanos: enable an expirer host for the thanos-swift cluster [puppet] - 10https://gerrit.wikimedia.org/r/1310971 (https://phabricator.wikimedia.org/T431506) (owner: 10MVernon) [08:57:42] (03CR) 10Brouberol: [C:03+2] cache/text: enable caching for query-scholarly next.w.o [puppet] - 10https://gerrit.wikimedia.org/r/1310013 (https://phabricator.wikimedia.org/T429150) (owner: 10Brouberol) [08:58:52] (03PS4) 10MVernon: thanos: enable an expirer host for the thanos-swift cluster [puppet] - 10https://gerrit.wikimedia.org/r/1310971 (https://phabricator.wikimedia.org/T431506) [09:00:43] RECOVERY - Check unit status of statograph_post on alert1002 is OK: OK: Status of the systemd unit statograph_post https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [09:02:27] (03CR) 10Brouberol: [C:03+1] kafka_config: migrate functions to the 4.x API [puppet] - 10https://gerrit.wikimedia.org/r/1308760 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [09:04:43] (03PS3) 10Elukey: kafka: remove leftovers for Kafka 1.1 [puppet] - 10https://gerrit.wikimedia.org/r/1310558 (https://phabricator.wikimedia.org/T432089) [09:04:54] (03CR) 10Elukey: "Good one! I added their removal, very risky :(" [puppet] - 10https://gerrit.wikimedia.org/r/1310558 (https://phabricator.wikimedia.org/T432089) (owner: 10Elukey) [09:05:30] (03CR) 10MVernon: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1310971 (https://phabricator.wikimedia.org/T431506) (owner: 10MVernon) [09:05:54] !log mvernon@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1011.eqiad.wmnet with reason: host reimage [09:08:21] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2012.codfw.wmnet with reason: host reimage [09:08:45] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1011.eqiad.wmnet with reason: host reimage [09:09:03] (03CR) 10Hnowlan: [C:03+1] images: Move loading of memcached key to asyncio [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1310623 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup) [09:11:38] (03PS4) 10Cathal Mooney: Pometheus nodes: adjust message on server_depool key [puppet] - 10https://gerrit.wikimedia.org/r/1310588 (https://phabricator.wikimedia.org/T327300) [09:12:43] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2012.codfw.wmnet with reason: host reimage [09:18:08] (03PS5) 10Cathal Mooney: Prometheus nodes: adjust message on server_depool key [puppet] - 10https://gerrit.wikimedia.org/r/1310588 (https://phabricator.wikimedia.org/T327300) [09:20:00] (03CR) 10MVernon: "[I checked the diff, and the swift-object-expirer package is selected as "absent" except on the new expirer host, so I think this is corre" [puppet] - 10https://gerrit.wikimedia.org/r/1310971 (https://phabricator.wikimedia.org/T431506) (owner: 10MVernon) [09:21:21] !log root@cumin1003 START - Cookbook sre.mysql.depool depool pc1023: Security updates [09:21:21] !log root@cumin1003 START - Cookbook sre.mysql.parsercache [09:21:29] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [09:21:29] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1023: Security updates [09:21:44] jouncebot: nowandnext [09:21:44] For the next 0 hour(s) and 38 minute(s): MediaWiki train - Utc-7+Utc-0 Version (secondary timeslot) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260715T0800) [09:21:44] In 0 hour(s) and 38 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260715T1000) [09:27:26] (03PS4) 10Elukey: kafka: remove leftovers for Kafka 1.1 [puppet] - 10https://gerrit.wikimedia.org/r/1310558 (https://phabricator.wikimedia.org/T432089) [09:27:26] (03PS1) 10Elukey: kafka: remove Kafka mirror maker configs [puppet] - 10https://gerrit.wikimedia.org/r/1310986 (https://phabricator.wikimedia.org/T432089) [09:27:40] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1011.eqiad.wmnet with OS trixie [09:27:46] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12122973 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-fe1011.eqiad.wmnet with OS trixie completed: - ms-fe1011 (**PASS*... [09:28:58] (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1310986 (https://phabricator.wikimedia.org/T432089) (owner: 10Elukey) [09:28:58] (03CR) 10Cathal Mooney: Prometheus nodes: adjust message on server_depool key (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1310588 (https://phabricator.wikimedia.org/T327300) (owner: 10Cathal Mooney) [09:29:01] (03CR) 10Cathal Mooney: [C:03+2] Prometheus nodes: adjust message on server_depool key [puppet] - 10https://gerrit.wikimedia.org/r/1310588 (https://phabricator.wikimedia.org/T327300) (owner: 10Cathal Mooney) [09:29:44] (03PS2) 10Elukey: kafka: remove Kafka mirror maker configs [puppet] - 10https://gerrit.wikimedia.org/r/1310986 (https://phabricator.wikimedia.org/T432089) [09:30:17] !log mvernon@cumin1003 START - Cookbook sre.hosts.reimage for host ms-fe1010.eqiad.wmnet with OS trixie [09:30:29] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12122980 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-fe1010.eqiad.wmnet with OS trixie [09:31:00] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2012.codfw.wmnet with OS trixie [09:31:11] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12122984 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-fe2012.codfw.wmnet with OS trixie completed: - ms-fe2012 (**PASS*... [09:31:17] !log mvernon@cumin2003 START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw [09:31:21] !log elukey@cumin1003 conftool action : set/pooled=false; selector: dnsdisc=tegola-vector-tiles,name=eqiad [09:31:30] !log elukey@cumin1003 conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=eqiad [09:32:19] !log cgoubert@cumin1003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-worker-codfw [09:32:37] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2001-2002,2005-2006,2011-2016].codfw.wmnet [09:32:39] (03CR) 10Blake: [C:03+2] mw-pretrain: Basic Ingress configuration. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309641 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [09:33:20] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet [09:33:43] (03PS1) 10Cathal Mooney: pki: move server_depool instructions to correct file [puppet] - 10https://gerrit.wikimedia.org/r/1310991 (https://phabricator.wikimedia.org/T327300) [09:35:29] (03Merged) 10jenkins-bot: mw-pretrain: Basic Ingress configuration. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309641 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [09:36:03] FIRING: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:36:37] !log blake@deploy2003 helmfile [codfw] START helmfile.d/services/mw-pretrain: apply [09:36:54] !log blake@deploy2003 helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply [09:37:17] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet [09:38:53] (03CR) 10Hnowlan: [C:04-1] swift: Migrate storage of results to asyncio (032 comments) [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1310618 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup) [09:39:55] (03PS2) 10Ayounsi: pki::multirootca: clarify depool policy [puppet] - 10https://gerrit.wikimedia.org/r/1310973 (https://phabricator.wikimedia.org/T327300) [09:40:21] (03CR) 10Cathal Mooney: [C:03+1] "lgtm thanks :)" [puppet] - 10https://gerrit.wikimedia.org/r/1310973 (https://phabricator.wikimedia.org/T327300) (owner: 10Ayounsi) [09:40:43] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:40:45] !log mvernon@cumin2003 END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw [09:40:50] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host netflow1002.eqiad.wmnet [09:41:03] RESOLVED: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:41:45] (03PS1) 10Blake: mw-pretrain: Fix host typo. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310996 (https://phabricator.wikimedia.org/T427668) [09:42:14] (03CR) 10Cathal Mooney: [C:03+2] pki::multirootca: clarify depool policy [puppet] - 10https://gerrit.wikimedia.org/r/1310973 (https://phabricator.wikimedia.org/T327300) (owner: 10Ayounsi) [09:42:14] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2001-2002,2005-2006,2011-2016].codfw.wmnet [09:43:04] (03CR) 10Clément Goubert: [C:03+1] mw-pretrain: Fix host typo. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310996 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [09:43:21] !log root@cumin1003 START - Cookbook sre.mysql.pool pool pc1023: Security updates [09:43:22] !log root@cumin1003 START - Cookbook sre.mysql.parsercache [09:43:33] (03CR) 10Cathal Mooney: "Done" [homer/public] - 10https://gerrit.wikimedia.org/r/1309707 (https://phabricator.wikimedia.org/T431849) (owner: 10Cathal Mooney) [09:43:35] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [09:43:35] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1023: Security updates [09:43:36] (03CR) 10Cathal Mooney: [C:03+2] Adjust order BGP_outfilter is applied to routes announced [homer/public] - 10https://gerrit.wikimedia.org/r/1309707 (https://phabricator.wikimedia.org/T431849) (owner: 10Cathal Mooney) [09:43:43] !log mvernon@cumin2003 conftool action : set/pooled=inactive; selector: name=ms-fe2012.codfw.wmnet [09:44:10] !log mvernon@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1010.eqiad.wmnet with reason: host reimage [09:44:27] (03CR) 10Blake: [C:03+2] mw-pretrain: Fix host typo. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310996 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [09:44:36] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow1002.eqiad.wmnet [09:44:51] !log mvernon@cumin2003 conftool action : set/pooled=yes; selector: name=ms-fe2012.codfw.wmnet [09:44:58] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host netflow1003.eqiad.wmnet [09:46:34] (03Merged) 10jenkins-bot: mw-pretrain: Fix host typo. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310996 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [09:46:50] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-fe2011.codfw.wmnet with OS trixie [09:46:58] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12123093 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-fe2011.codfw.wmnet with OS trixie [09:47:20] !log mvernon@cumin2003 START - Cookbook sre.hosts.move-vlan for host ms-fe2011 [09:47:26] !log blake@deploy2003 helmfile [codfw] START helmfile.d/services/mw-pretrain: apply [09:47:28] !log mvernon@cumin2003 START - Cookbook sre.dns.netbox [09:47:36] !log blake@deploy2003 helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply [09:49:09] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1010.eqiad.wmnet with reason: host reimage [09:49:40] !log blake@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-pretrain: apply [09:49:48] (03Merged) 10jenkins-bot: Adjust order BGP_outfilter is applied to routes announced [homer/public] - 10https://gerrit.wikimedia.org/r/1309707 (https://phabricator.wikimedia.org/T431849) (owner: 10Cathal Mooney) [09:49:56] !log blake@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-pretrain: apply [09:50:15] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow1003.eqiad.wmnet [09:50:26] (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1310986 (https://phabricator.wikimedia.org/T432089) (owner: 10Elukey) [09:50:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [09:50:43] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [09:51:07] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host netflow2003.codfw.wmnet [09:51:53] !log mvernon@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2011 - mvernon@cumin2003" [09:51:57] !log mvernon@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2011 - mvernon@cumin2003" [09:51:58] !log mvernon@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [09:51:58] !log mvernon@cumin2003 START - Cookbook sre.dns.wipe-cache ms-fe2011.codfw.wmnet 36.32.192.10.in-addr.arpa 6.3.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [09:52:02] !log mvernon@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-fe2011.codfw.wmnet 36.32.192.10.in-addr.arpa 6.3.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [09:52:03] !log mvernon@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host ms-fe2011 [09:52:38] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2001-2002,2005-2006,2011-2016].codfw.wmnet [09:52:44] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2001-2002,2005-2006,2011-2016].codfw.wmnet [09:53:04] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2017-2018,2033-2039,2041].codfw.wmnet [09:54:24] !log mvernon@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-fe2011 [09:54:24] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-fe2011 [09:54:25] FIRING: [12x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:54:56] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2003.codfw.wmnet [09:59:20] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2017-2018,2033-2039,2041].codfw.wmnet [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260715T1000) [10:01:02] 06SRE, 06Product Safety and Integrity, 06ServiceOps new, 07Kubernetes: EtcdConfig failed to fetch data: (curl error: 28) Timeout was reached - https://phabricator.wikimedia.org/T432110#12123160 (10Dreamy_Jazz) [10:02:49] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host netflow2004.codfw.wmnet [10:04:21] !log push out config change to BGP_outfilter on core routers T431849 [10:04:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:04:28] (03CR) 10Hnowlan: [C:03+2] karma: automatically linkify Phab task IDs in messages [puppet] - 10https://gerrit.wikimedia.org/r/1310141 (https://phabricator.wikimedia.org/T431980) (owner: 10Hnowlan) [10:06:01] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet [10:07:22] !log cgoubert@deploy2003 Started deploy [restbase/deploy@06301bd]: Deploying 1306088 1308347 - T429944 T428279 [10:07:28] T429944: Add minwikiquote to RESTBase - https://phabricator.wikimedia.org/T429944 [10:07:28] T428279: Post-creation work for magwiki - https://phabricator.wikimedia.org/T428279 [10:08:08] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1010.eqiad.wmnet with OS trixie [10:08:09] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow2004.codfw.wmnet [10:08:15] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12123189 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-fe1010.eqiad.wmnet with OS trixie completed: - ms-fe1010 (**PASS*... [10:08:37] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-eqord:xe-0/1/3 (Transport: cr3-ulsfo:xe-0/1/1 (Arelion, IC-313592 51ms 10Gbps wave) {#11372}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [10:09:10] FIRING: [12x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:09:14] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host netflow3004.esams.wmnet [10:09:38] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2017-2018,2033-2039,2041].codfw.wmnet [10:09:44] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2017-2018,2033-2039,2041].codfw.wmnet [10:09:48] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet [10:10:04] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host maps1014.eqiad.wmnet [10:10:04] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2042,2044,2046,2049-2051,2055-2058].codfw.wmnet [10:11:16] (03PS1) 10Blake: service-catalog: add a new service for mw-pretrain. [puppet] - 10https://gerrit.wikimedia.org/r/1310999 (https://phabricator.wikimedia.org/T427668) [10:11:26] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2011.codfw.wmnet with reason: host reimage [10:13:22] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow3004.esams.wmnet [10:14:10] FIRING: [12x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:14:12] !log root@cumin1003 START - Cookbook sre.mysql.depool depool pc1017: Security updates [10:14:12] !log root@cumin1003 START - Cookbook sre.mysql.parsercache [10:14:21] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [10:14:21] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1017: Security updates [10:14:49] (03PS1) 10Brouberol: test-kitchen-next: temporarily disable the GB experiment validation feature [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311001 (https://phabricator.wikimedia.org/T431015) [10:15:19] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2011.codfw.wmnet with reason: host reimage [10:15:43] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:16:55] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2042,2044,2046,2049-2051,2055-2058].codfw.wmnet [10:17:06] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1014.eqiad.wmnet [10:17:31] FIRING: Not accepting/receiving prefixes from anycast BGP peer: Alert for device asw1-b4-magru.mgmt.magru.wmnet - Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [10:18:52] (03PS2) 10Brouberol: test-kitchen-next: temporarily disable the GB experiment validation feature [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311001 (https://phabricator.wikimedia.org/T431015) [10:19:32] (03PS1) 10Tiziano Fogli: thanos/recrules: wdqs update lag [puppet] - 10https://gerrit.wikimedia.org/r/1311002 [10:21:14] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host maps1013.eqiad.wmnet [10:21:55] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host netflow4003.ulsfo.wmnet [10:22:01] (03CR) 10Santiago Faci: [C:03+2] test-kitchen-next: temporarily disable the GB experiment validation feature [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311001 (https://phabricator.wikimedia.org/T431015) (owner: 10Brouberol) [10:23:26] (03CR) 10Brouberol: [V:03+2] test-kitchen-next: temporarily disable the GB experiment validation feature [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311001 (https://phabricator.wikimedia.org/T431015) (owner: 10Brouberol) [10:25:13] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply [10:25:23] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply [10:25:32] (03CR) 10Elukey: [C:03+1] thanos/recrules: wdqs update lag [puppet] - 10https://gerrit.wikimedia.org/r/1311002 (owner: 10Tiziano Fogli) [10:25:55] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow4003.ulsfo.wmnet [10:26:50] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2042,2044,2046,2049-2051,2055-2058].codfw.wmnet [10:26:56] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2042,2044,2046,2049-2051,2055-2058].codfw.wmnet [10:27:08] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host netflow5003.eqsin.wmnet [10:27:16] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2059-2062,2064-2065,2067-2070].codfw.wmnet [10:27:36] (03CR) 10Fabfur: [C:03+1] "I think there should be no issues with this!" [puppet] - 10https://gerrit.wikimedia.org/r/1310151 (https://phabricator.wikimedia.org/T431627) (owner: 10BCornwall) [10:28:10] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1013.eqiad.wmnet [10:28:25] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host maps1012.eqiad.wmnet [10:29:46] (03PS1) 10Tiziano Fogli: sloth: add rule to detect missing slo:sli_error:ratio_rate metric [alerts] - 10https://gerrit.wikimedia.org/r/1310604 (https://phabricator.wikimedia.org/T432140) [10:30:01] !log mvernon@cumin1003 START - Cookbook sre.hosts.reimage for host ms-fe1009.eqiad.wmnet with OS trixie [10:30:09] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12123276 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-fe1009.eqiad.wmnet with OS trixie [10:32:31] (03PS2) 10Tiziano Fogli: sloth: add rule to detect missing slo:sli_error:ratio_rate metric [alerts] - 10https://gerrit.wikimedia.org/r/1310604 (https://phabricator.wikimedia.org/T432140) [10:33:30] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2059-2062,2064-2065,2067-2070].codfw.wmnet [10:34:26] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2011.codfw.wmnet with OS trixie [10:34:40] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12123296 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-fe2011.codfw.wmnet with OS trixie completed: - ms-fe2011 (**PASS*... [10:34:49] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow5003.eqsin.wmnet [10:35:06] !log mvernon@cumin2003 START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw [10:35:38] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1012.eqiad.wmnet [10:35:53] (03PS3) 10Btullis: cloudnative-pg-cluster: omit empty backup encryption from Cluster manifests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310602 (https://phabricator.wikimedia.org/T424493) [10:35:56] !log cgoubert@deploy2003 Finished deploy [restbase/deploy@06301bd]: Deploying 1306088 1308347 - T429944 T428279 (duration: 28m 34s) [10:36:01] T429944: Add minwikiquote to RESTBase - https://phabricator.wikimedia.org/T429944 [10:36:01] T428279: Post-creation work for magwiki - https://phabricator.wikimedia.org/T428279 [10:36:57] RECOVERY - Dell PowerEdge or Supermicro Broadcom RAID Controller on backup1019 is OK: communication: 0 OK : controller: 0 OK : physical_disk: 0 OK : virtual_disk: 0 OK : bbu: 0 OK : enclosure: 0 OK https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [10:38:26] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host maps1011.eqiad.wmnet [10:39:07] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host netflow6001.drmrs.wmnet [10:39:20] (03CR) 10Tiziano Fogli: [C:03+2] thanos/recrules: wdqs update lag [puppet] - 10https://gerrit.wikimedia.org/r/1311002 (owner: 10Tiziano Fogli) [10:39:46] !log root@cumin1003 START - Cookbook sre.mysql.pool pool pc1017: Security updates [10:39:46] !log root@cumin1003 START - Cookbook sre.mysql.parsercache [10:39:59] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [10:39:59] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1017: Security updates [10:40:34] (03CR) 10Cathal Mooney: [C:03+2] Nokia SR-Linux: add export policy for unicast IBGP clusters [homer/public] - 10https://gerrit.wikimedia.org/r/1309643 (https://phabricator.wikimedia.org/T423430) (owner: 10Cathal Mooney) [10:41:58] (03Merged) 10jenkins-bot: Nokia SR-Linux: add export policy for unicast IBGP clusters [homer/public] - 10https://gerrit.wikimedia.org/r/1309643 (https://phabricator.wikimedia.org/T423430) (owner: 10Cathal Mooney) [10:43:08] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow6001.drmrs.wmnet [10:43:55] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2059-2062,2064-2065,2067-2070].codfw.wmnet [10:44:02] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2059-2062,2064-2065,2067-2070].codfw.wmnet [10:44:13] !log mvernon@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe1009.eqiad.wmnet with reason: host reimage [10:44:17] !log mvernon@cumin2003 END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw [10:44:35] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2071-2078,2087-2088].codfw.wmnet [10:44:57] !log mvernon@cumin2003 conftool action : set/pooled=inactive; selector: name=ms-fe2011.codfw.wmnet [10:45:02] (03CR) 10Brouberol: [C:03+1] "Nice, thanks!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310602 (https://phabricator.wikimedia.org/T424493) (owner: 10Btullis) [10:45:14] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1011.eqiad.wmnet [10:46:04] !log mvernon@cumin2003 conftool action : set/pooled=yes; selector: name=ms-fe2011.codfw.wmnet [10:49:00] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-fe2010.codfw.wmnet with OS trixie [10:49:08] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12123363 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-fe2010.codfw.wmnet with OS trixie [10:49:28] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe1009.eqiad.wmnet with reason: host reimage [10:49:29] !log mvernon@cumin2003 START - Cookbook sre.hosts.move-vlan for host ms-fe2010 [10:49:38] !log mvernon@cumin2003 START - Cookbook sre.dns.netbox [10:50:25] FIRING: SystemdUnitFailed: fetch-rings-eqiad.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:51:34] (03CR) 10Btullis: [C:03+2] cloudnative-pg-cluster: omit empty backup encryption from Cluster manifests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310602 (https://phabricator.wikimedia.org/T424493) (owner: 10Btullis) [10:52:24] (03PS4) 10Btullis: cloudnative-pg-cluster: omit empty backup encryption from Cluster manifests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310602 (https://phabricator.wikimedia.org/T424493) [10:52:24] (03PS2) 10Btullis: istio: expose a TLS passthrough port for PostgreSQL on dse-k8s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310601 (https://phabricator.wikimedia.org/T432104) [10:52:24] (03PS3) 10Btullis: cloudnative-pg-cluster: add SNI-based TLS passthrough ingress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310603 (https://phabricator.wikimedia.org/T432104) [10:54:31] !log mvernon@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2010 - mvernon@cumin2003" [10:54:36] !log mvernon@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2010 - mvernon@cumin2003" [10:54:37] !log mvernon@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [10:54:37] !log mvernon@cumin2003 START - Cookbook sre.dns.wipe-cache ms-fe2010.codfw.wmnet 76.16.192.10.in-addr.arpa 6.7.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [10:54:40] !log mvernon@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-fe2010.codfw.wmnet 76.16.192.10.in-addr.arpa 6.7.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [10:54:41] !log mvernon@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host ms-fe2010 [10:54:44] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2071-2078,2087-2088].codfw.wmnet [10:54:45] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/webrequest-page-view-next: apply [10:54:52] !log mvernon@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-fe2010 [10:54:52] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-fe2010 [10:54:56] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/webrequest-page-view-next: apply [11:00:02] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host netflow7002.magru.wmnet [11:00:02] (03PS5) 10Btullis: cloudnative-pg-cluster: omit empty backup encryption from Cluster manifests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310602 (https://phabricator.wikimedia.org/T424493) [11:00:03] (03PS3) 10Btullis: istio: expose a TLS passthrough port for PostgreSQL on dse-k8s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310601 (https://phabricator.wikimedia.org/T432104) [11:00:03] (03PS4) 10Btullis: cloudnative-pg-cluster: add SNI-based TLS passthrough ingress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310603 (https://phabricator.wikimedia.org/T432104) [11:00:05] mvolz: How many deployers does it take to do Services – Citoid / Zotero deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260715T1100). [11:01:31] !log elukey@cumin1003 conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad [11:01:42] !log elukey@cumin1003 conftool action : set/pooled=true; selector: dnsdisc=tegola-vector-tiles,name=eqiad [11:02:16] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2071-2078,2087-2088].codfw.wmnet [11:02:22] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2071-2078,2087-2088].codfw.wmnet [11:02:46] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2089-2095,2102-2104].codfw.wmnet [11:03:25] (03CR) 10Btullis: [V:03+2 C:03+2] cloudnative-pg-cluster: omit empty backup encryption from Cluster manifests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310602 (https://phabricator.wikimedia.org/T424493) (owner: 10Btullis) [11:05:40] 06SRE, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Don't announce OSPF routes in unicast BGP on Nokia SR-Linux - https://phabricator.wikimedia.org/T423430#12123447 (10cmooney) Ok all merged and looking good. We now have no hidden routes in the table on the codfw spines ` cmooney@ssw1-e... [11:05:44] 06SRE, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Don't announce OSPF routes in unicast BGP on Nokia SR-Linux - https://phabricator.wikimedia.org/T423430#12123450 (10cmooney) 05Open→03Resolved [11:05:55] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netflow7002.magru.wmnet [11:06:03] (03Merged) 10jenkins-bot: cloudnative-pg-cluster: omit empty backup encryption from Cluster manifests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310602 (https://phabricator.wikimedia.org/T424493) (owner: 10Btullis) [11:07:54] (03CR) 10Btullis: [C:03+2] opensearch-cluster: render nodePool affinity through tpl [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309107 (https://phabricator.wikimedia.org/T402408) (owner: 10Btullis) [11:09:37] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe1009.eqiad.wmnet with OS trixie [11:09:51] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12123476 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-fe1009.eqiad.wmnet with OS trixie completed: - ms-fe1009 (**PASS*... [11:10:12] !log root@cumin1003 START - Cookbook sre.mysql.depool depool pc1018: Security updates [11:10:12] !log root@cumin1003 START - Cookbook sre.mysql.parsercache [11:10:20] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [11:10:20] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc1018: Security updates [11:10:25] RESOLVED: SystemdUnitFailed: fetch-rings-eqiad.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:10:44] 06SRE, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Don't announce OSPF routes in unicast BGP on Nokia SR-Linux - https://phabricator.wikimedia.org/T423430#12123477 (10cmooney) 05Resolved→03Open [11:11:22] (03Merged) 10jenkins-bot: opensearch-cluster: render nodePool affinity through tpl [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309107 (https://phabricator.wikimedia.org/T402408) (owner: 10Btullis) [11:11:59] (03CR) 10Btullis: "Thanks @gmodena@wikimedia.org for your work on this. However, I feel that we can abandon this patch now, as ephemeral volumes based on loc" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1305852 (https://phabricator.wikimedia.org/T428235) (owner: 10Gmodena) [11:12:19] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2010.codfw.wmnet with reason: host reimage [11:12:25] FIRING: SystemdUnitFailed: fetch-rings-eqiad.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:12:55] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2089-2095,2102-2104].codfw.wmnet [11:12:57] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12123487 (10MatthewVernon) [11:15:21] (03CR) 10Btullis: "As discussed on I6d2adec9bee13562f28e860efd301fee261f86de I think that we can abandon this patch too. Ephemeral local volumes using a vari" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1305854 (https://phabricator.wikimedia.org/T428235) (owner: 10Gmodena) [11:17:25] RESOLVED: SystemdUnitFailed: fetch-rings-eqiad.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:18:50] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2010.codfw.wmnet with reason: host reimage [11:18:56] (03PS1) 10Cathal Mooney: team-netops: include ibgp group in RejectedPrefixes alert [alerts] - 10https://gerrit.wikimedia.org/r/1311026 (https://phabricator.wikimedia.org/T423430) [11:19:12] (03PS4) 10Btullis: istio: expose a TLS passthrough port for PostgreSQL on dse-k8s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310601 (https://phabricator.wikimedia.org/T432104) [11:19:12] (03PS5) 10Btullis: cloudnative-pg-cluster: add SNI-based TLS passthrough ingress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310603 (https://phabricator.wikimedia.org/T432104) [11:19:56] (03PS1) 10Arthur taylor: wikidata-query-gui: Bump query-gui image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311027 (https://phabricator.wikimedia.org/T431803) [11:20:16] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2089-2095,2102-2104].codfw.wmnet [11:20:22] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2089-2095,2102-2104].codfw.wmnet [11:20:42] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2105-2114].codfw.wmnet [11:21:34] (03CR) 10Elukey: [C:03+1] "Totally missed this, let's try it!" [cookbooks] - 10https://gerrit.wikimedia.org/r/1307091 (owner: 10FNegri) [11:22:10] (03PS1) 10Krinkle: Switch math rendering for group0 from native to mathjax [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1224074 (https://phabricator.wikimedia.org/T413973) (owner: 10Physikerwelt) [11:22:10] (03PS1) 10Hnowlan: profile::alertmanager: extend default login session length [puppet] - 10https://gerrit.wikimedia.org/r/1311028 (https://phabricator.wikimedia.org/T432226) [11:22:31] (03PS5) 10Btullis: istio: expose a TLS passthrough port for PostgreSQL on dse-k8s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310601 (https://phabricator.wikimedia.org/T432104) [11:22:31] (03PS6) 10Btullis: cloudnative-pg-cluster: add SNI-based TLS passthrough ingress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310603 (https://phabricator.wikimedia.org/T432104) [11:26:22] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2105-2114].codfw.wmnet [11:27:22] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1224074 (https://phabricator.wikimedia.org/T413973) (owner: 10Physikerwelt) [11:28:55] (03Merged) 10jenkins-bot: Switch math rendering for group0 from native to mathjax [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1224074 (https://phabricator.wikimedia.org/T413973) (owner: 10Physikerwelt) [11:29:09] (03PS2) 10Hnowlan: karma: consistently order some labels first [puppet] - 10https://gerrit.wikimedia.org/r/1310519 (https://phabricator.wikimedia.org/T431980) [11:29:10] (03PS1) 10Hnowlan: karma: strip some useless labels from panel display [puppet] - 10https://gerrit.wikimedia.org/r/1311029 (https://phabricator.wikimedia.org/T431980) [11:29:12] !log krinkle@deploy2003 Started scap sync-world: Backport for [[gerrit:1224074|Switch math rendering for group0 from native to mathjax (T413973)]] [11:29:16] T413973: Change wmf-config for Math extension from Mathoid to MathML+MatJax on all wikis that have Parsoid and no known usage - https://phabricator.wikimedia.org/T413973 [11:31:06] !log krinkle@deploy2003 physikerwelt, krinkle: Backport for [[gerrit:1224074|Switch math rendering for group0 from native to mathjax (T413973)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:32:33] (03PS2) 10Hnowlan: archiva: clean up absented check [puppet] - 10https://gerrit.wikimedia.org/r/1307159 (https://phabricator.wikimedia.org/T407117) [11:35:49] (03CR) 10Hnowlan: [C:03+2] archiva: clean up absented check [puppet] - 10https://gerrit.wikimedia.org/r/1307159 (https://phabricator.wikimedia.org/T407117) (owner: 10Hnowlan) [11:36:08] !log root@cumin1003 START - Cookbook sre.mysql.pool pool pc1018: Security updates [11:36:08] !log root@cumin1003 START - Cookbook sre.mysql.parsercache [11:36:23] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [11:36:23] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc1018: Security updates [11:36:34] !log krinkle@deploy2003 physikerwelt, krinkle: Continuing with deployment [11:36:39] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2105-2114].codfw.wmnet [11:36:46] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2105-2114].codfw.wmnet [11:37:05] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2115,2124-2132].codfw.wmnet [11:37:56] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2010.codfw.wmnet with OS trixie [11:38:04] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12123619 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-fe2010.codfw.wmnet with OS trixie completed: - ms-fe2010 (**PASS*... [11:39:10] FIRING: [12x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:39:21] (03PS1) 10Mvolz: Update citoid [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311032 [11:40:51] !log krinkle@deploy2003 Finished scap sync-world: Backport for [[gerrit:1224074|Switch math rendering for group0 from native to mathjax (T413973)]] (duration: 11m 38s) [11:40:54] T413973: Rollout MathML+MathJax to group0, and wikis that use Parsoid and have no known usage - https://phabricator.wikimedia.org/T413973 [11:41:48] (03CR) 10Mvolz: [C:03+2] Update citoid [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311032 (owner: 10Mvolz) [11:42:53] (03CR) 10Clément Goubert: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1310589 (https://phabricator.wikimedia.org/T432108) (owner: 10Kamila Součková) [11:43:17] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2115,2124-2132].codfw.wmnet [11:43:54] (03Merged) 10jenkins-bot: Update citoid [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311032 (owner: 10Mvolz) [11:44:10] FIRING: [12x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:44:17] (03PS5) 10Jelto: Update calico-crds to calico v3.30.7 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306165 (https://phabricator.wikimedia.org/T427400) [11:44:46] (03CR) 10Jelto: "fixed in patchset 5" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306165 (https://phabricator.wikimedia.org/T427400) (owner: 10Jelto) [11:44:56] !log mvolz@deploy2003 helmfile [staging] START helmfile.d/services/citoid: apply [11:45:22] !log mvolz@deploy2003 helmfile [staging] DONE helmfile.d/services/citoid: apply [11:46:28] !log mvolz@deploy2003 helmfile [codfw] START helmfile.d/services/citoid: apply [11:46:53] !log mvolz@deploy2003 helmfile [codfw] DONE helmfile.d/services/citoid: apply [11:47:33] !log btullis@cumin1003 START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes [11:48:49] !log mvolz@deploy2003 helmfile [eqiad] START helmfile.d/services/citoid: apply [11:49:06] !log btullis@cumin1003 START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-codfw [11:49:22] !log mvolz@deploy2003 helmfile [eqiad] DONE helmfile.d/services/citoid: apply [11:50:06] jouncebot: now [11:50:06] For the next 0 hour(s) and 9 minute(s): Services – Citoid / Zotero (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260715T1100) [11:51:16] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2115,2124-2132].codfw.wmnet [11:51:22] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2115,2124-2132].codfw.wmnet [11:51:42] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2133-2142].codfw.wmnet [11:51:51] PROBLEM - BFD status on lsw1-a7-codfw.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [11:52:05] 06SRE, 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12123705 (10BTullis) [11:53:22] (03CR) 10Hnowlan: profile::puppetserver::volatile: add metrics for airflow webrequest (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1310118 (https://phabricator.wikimedia.org/T402512) (owner: 10Elukey) [11:54:10] FIRING: [14x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [11:55:43] (03PS1) 10Btullis: service: add k8s-ingress-dse-postgresql TLS passthrough service [puppet] - 10https://gerrit.wikimedia.org/r/1311036 (https://phabricator.wikimedia.org/T432104) [11:56:18] (03CR) 10CI reject: [V:04-1] service: add k8s-ingress-dse-postgresql TLS passthrough service [puppet] - 10https://gerrit.wikimedia.org/r/1311036 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [11:56:53] RECOVERY - BFD status on lsw1-a7-codfw.mgmt is OK: UP: 4 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [11:57:07] (03PS2) 10CWilliams: sanitize-wiki: use of --section is mutually exclusive with the use of --check [cookbooks] - 10https://gerrit.wikimedia.org/r/1308566 (https://phabricator.wikimedia.org/T431534) [11:57:24] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2133-2142].codfw.wmnet [11:59:10] FIRING: [14x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [12:02:08] !log mvernon@cumin2003 START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw [12:02:52] (03PS2) 10Btullis: service: add k8s-ingress-dse-postgresql TLS passthrough service [puppet] - 10https://gerrit.wikimedia.org/r/1311036 (https://phabricator.wikimedia.org/T432104) [12:03:29] (03CR) 10CI reject: [V:04-1] service: add k8s-ingress-dse-postgresql TLS passthrough service [puppet] - 10https://gerrit.wikimedia.org/r/1311036 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [12:07:13] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2133-2142].codfw.wmnet [12:07:19] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2133-2142].codfw.wmnet [12:07:43] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2143-2152].codfw.wmnet [12:08:05] PROBLEM - BFD status on lsw1-c2-codfw.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [12:08:08] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] wikidata-query-gui: Bump query-gui image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311027 (https://phabricator.wikimedia.org/T431803) (owner: 10Arthur taylor) [12:08:42] me and Thiemo are going to bump the version of the wikidata-query-gui service in a sec [12:09:47] (03CR) 10Arthur taylor: [C:03+2] wikidata-query-gui: Bump query-gui image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311027 (https://phabricator.wikimedia.org/T431803) (owner: 10Arthur taylor) [12:09:47] (03PS1) 10Krinkle: Enable MathML+MathJax on wikis with no known math use as of June 2026 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311041 (https://phabricator.wikimedia.org/T413973) [12:10:24] !log mvernon@cumin2003 END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw [12:10:39] (03CR) 10CI reject: [V:04-1] Enable MathML+MathJax on wikis with no known math use as of June 2026 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311041 (https://phabricator.wikimedia.org/T413973) (owner: 10Krinkle) [12:11:07] (03PS1) 10JavierMonton: stream: pageview-trending-relative [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311043 (https://phabricator.wikimedia.org/T432204) [12:11:28] !log mvernon@cumin2003 conftool action : set/pooled=inactive; selector: name=ms-fe2010.codfw.wmnet [12:12:05] RECOVERY - BFD status on lsw1-c2-codfw.mgmt is OK: UP: 4 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [12:12:07] (03PS3) 10Btullis: service: add k8s-ingress-dse-postgresql TLS passthrough service [puppet] - 10https://gerrit.wikimedia.org/r/1311036 (https://phabricator.wikimedia.org/T432104) [12:12:13] (03Merged) 10jenkins-bot: wikidata-query-gui: Bump query-gui image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311027 (https://phabricator.wikimedia.org/T431803) (owner: 10Arthur taylor) [12:12:36] !log mvernon@cumin2003 conftool action : set/pooled=yes; selector: name=ms-fe2010.codfw.wmnet [12:13:17] (03Abandoned) 10Nikerabbit: Enable AG experiment on phase 2 batch 2 wikis: ar, bn [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1293139 (https://phabricator.wikimedia.org/T426871) (owner: 10Sbisson) [12:13:46] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311036 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [12:13:52] !log arthurtaylor@deploy2003 helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply [12:14:07] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2143-2152].codfw.wmnet [12:14:39] !log arthurtaylor@deploy2003 helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply [12:15:02] !log arthurtaylor@deploy2003 helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply [12:15:07] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-fe2009.codfw.wmnet with OS trixie [12:15:15] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12123797 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-fe2009.codfw.wmnet with OS trixie [12:15:23] !log arthurtaylor@deploy2003 helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply [12:15:36] !log mvernon@cumin2003 START - Cookbook sre.hosts.move-vlan for host ms-fe2009 [12:15:39] !log arthurtaylor@deploy2003 helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply [12:15:56] !log arthurtaylor@deploy2003 helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply [12:16:00] !log mvernon@cumin2003 START - Cookbook sre.dns.netbox [12:17:22] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-eqord:xe-0/1/3 (Transport: cr3-ulsfo:xe-0/1/1 (Arelion, IC-313592 51ms 10Gbps wave) {#11372}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [12:17:40] (03CR) 10Physikerwelt: Enable MathML+MathJax on wikis with no known math use as of June 2026 (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311041 (https://phabricator.wikimedia.org/T413973) (owner: 10Krinkle) [12:22:02] !log mvernon@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2009 - mvernon@cumin2003" [12:22:06] !log mvernon@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host ms-fe2009 - mvernon@cumin2003" [12:22:07] !log mvernon@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:22:07] !log mvernon@cumin2003 START - Cookbook sre.dns.wipe-cache ms-fe2009.codfw.wmnet 139.0.192.10.in-addr.arpa 9.3.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [12:22:11] !log mvernon@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ms-fe2009.codfw.wmnet 139.0.192.10.in-addr.arpa 9.3.1.0.0.0.0.0.2.9.1.0.0.1.0.0.1.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [12:22:12] !log mvernon@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host ms-fe2009 [12:22:50] !log mvernon@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ms-fe2009 [12:22:51] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host ms-fe2009 [12:22:53] PROBLEM - BFD status on lsw1-d2-codfw.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [12:24:59] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2143-2152].codfw.wmnet [12:25:05] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2143-2152].codfw.wmnet [12:25:25] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2153-2162].codfw.wmnet [12:26:53] RECOVERY - BFD status on lsw1-d2-codfw.mgmt is OK: UP: 4 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [12:29:34] (03PS3) 10Aqu: airflow: add gcs-token-search-console secret [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309204 (https://phabricator.wikimedia.org/T427457) [12:29:35] !log btullis@cumin1003 END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-codfw [12:29:53] (03PS2) 10Krinkle: MathML+MathJax rollout to phase 2 (not Wikibooks/Wikisource/Wikipedia) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311041 (https://phabricator.wikimedia.org/T271001) [12:30:41] (03CR) 10Nikerabbit: [V:03+2] Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1310074 (owner: 10L10n-bot) [12:31:04] (03PS1) 10Klausman: profiles/roles: mount CephFS on /home on ml-lab1002 [puppet] - 10https://gerrit.wikimedia.org/r/1311046 (https://phabricator.wikimedia.org/T380279) [12:31:05] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2153-2162].codfw.wmnet [12:31:37] (03CR) 10CI reject: [V:04-1] MathML+MathJax rollout to phase 2 (not Wikibooks/Wikisource/Wikipedia) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311041 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [12:32:25] FIRING: SystemdUnitFailed: fetch-rings-codfw.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:35:56] (03CR) 10Brouberol: [C:03+1] "Nice!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310601 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [12:36:30] (03CR) 10Brouberol: [C:03+1] service: add k8s-ingress-dse-postgresql TLS passthrough service [puppet] - 10https://gerrit.wikimedia.org/r/1311036 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [12:36:50] (03CR) 10Krinkle: MathML+MathJax rollout to phase 2 (not Wikibooks/Wikisource/Wikipedia) (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311041 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [12:37:40] (03CR) 10Brouberol: [C:03+1] "Beautifully done!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310603 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [12:40:09] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-fe2009.codfw.wmnet with reason: host reimage [12:40:57] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2153-2162].codfw.wmnet [12:41:03] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2153-2162].codfw.wmnet [12:41:32] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2163-2172].codfw.wmnet [12:43:48] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-fe2009.codfw.wmnet with reason: host reimage [12:47:14] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2163-2172].codfw.wmnet [12:54:27] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2163-2172].codfw.wmnet [12:54:33] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2163-2172].codfw.wmnet [12:54:58] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2173-2179,2184-2186].codfw.wmnet [12:56:56] 10SRE-swift-storage, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: RdfStreamingUpdaterSpaceUsageTooHigh - https://phabricator.wikimedia.org/T431506#12124055 (10bking) a:05bking→03RKemper [12:57:25] !log btullis@cumin1003 END (PASS) - Cookbook sre.druid.reboot-workers (exit_code=0) for Druid analytics cluster: Reboot Druid nodes [12:57:29] (03CR) 10Tiziano Fogli: "Looks good to me, but I don't have a ton of experience with Swift. If you'd prefer to get another review from someone else, I won't mind." [puppet] - 10https://gerrit.wikimedia.org/r/1310971 (https://phabricator.wikimedia.org/T431506) (owner: 10MVernon) [13:00:05] urbanecm and TheresNoTime: #bothumor My software never has bugs. It just develops random features. Rise for UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260715T1300). [13:00:05] No Gerrit patches in the queue for this window AFAICS. [13:00:43] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2173-2179,2184-2186].codfw.wmnet [13:01:00] !log bking@deploy2003 Started deploy [wdqs/wdqs@e8fb00c] (wcqs): T430879 [13:01:04] T430879: Migrate WCQS to Bookworm or later - https://phabricator.wikimedia.org/T430879 [13:01:07] !log bking@deploy2003 Finished deploy [wdqs/wdqs@e8fb00c] (wcqs): T430879 (duration: 00m 15s) [13:02:39] FIRING: CoreBGPDown: Core BGP session down between cr2-eqord and cr1-eqiad (208.80.154.196) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=cr2-eqord:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:03:21] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T430879, restore data on newly-reimaged host) xfer commons from wcqs1001.eqiad.wmnet -> wcqs1002.eqiad.wmnet, repooling source-only afterwards [13:03:21] (03CR) 10FNegri: [C:03+2] tox.ini: allow running a single test [cookbooks] - 10https://gerrit.wikimedia.org/r/1307091 (owner: 10FNegri) [13:03:28] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [13:03:34] (03PS5) 10MVernon: thanos: enable an expirer host for the thanos-swift cluster [puppet] - 10https://gerrit.wikimedia.org/r/1310971 (https://phabricator.wikimedia.org/T431506) [13:03:36] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-fe2009.codfw.wmnet with OS trixie [13:03:44] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12124108 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-fe2009.codfw.wmnet with OS trixie completed: - ms-fe2009 (**PASS*... [13:04:14] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [13:04:33] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [13:04:34] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [13:05:17] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm [13:05:45] !log bking@cumin2003 START - Cookbook sre.hosts.move-vlan for host wcqs2001 [13:05:47] !log bking@cumin2003 END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T430879, restore data on newly-reimaged host) xfer commons from wcqs1001.eqiad.wmnet -> wcqs1002.eqiad.wmnet, repooling source-only afterwards [13:06:29] !log mvernon@cumin2003 START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw [13:06:43] (03CR) 10MVernon: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1310971 (https://phabricator.wikimedia.org/T431506) (owner: 10MVernon) [13:06:46] (03CR) 10Tiziano Fogli: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1307405 (https://phabricator.wikimedia.org/T407117) (owner: 10Hnowlan) [13:07:21] (03Merged) 10jenkins-bot: tox.ini: allow running a single test [cookbooks] - 10https://gerrit.wikimedia.org/r/1307091 (owner: 10FNegri) [13:08:15] 10ops-eqiad, 06SRE, 06cloud-services-team, 10Cloud-VPS, and 2 others: New thermal paste for cloudvirt1071.eqiad.wmnet - https://phabricator.wikimedia.org/T431429#12124136 (10Andrew) Thanks! I have put this host back in service and we'll keep an eye on it. [13:08:33] 10ops-eqiad, 06SRE, 06cloud-services-team, 10Cloud-VPS, and 2 others: New thermal paste for cloudvirt1071.eqiad.wmnet - https://phabricator.wikimedia.org/T431429#12124137 (10Andrew) a:05VRiley-WMF→03Andrew [13:08:48] bking@cumin2003 reimage (PID 1709096) is awaiting input [13:11:11] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2173-2179,2184-2186].codfw.wmnet [13:11:17] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2173-2179,2184-2186].codfw.wmnet [13:11:37] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2187-2196].codfw.wmnet [13:12:15] !log sukhe@cumin1003 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_ulsfo [13:12:25] RESOLVED: SystemdUnitFailed: fetch-rings-codfw.service on puppetserver1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:12:33] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [13:12:36] !log sukhe@cumin1003 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_ulsfo [13:12:39] FIRING: [3x] CoreBGPDown: Core BGP session down between cr2-eqiad and cr2-eqord (208.80.154.198) - group Confed_eqord - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [13:12:42] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [13:13:01] (03CR) 10MVernon: "Hi," [puppet] - 10https://gerrit.wikimedia.org/r/1310971 (https://phabricator.wikimedia.org/T431506) (owner: 10MVernon) [13:13:01] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [13:13:07] !log sukhe@cumin1003 START - Cookbook sre.dns.roll-reboot rolling reboot on A:dnsbox and not (A:ulsfo or A:magru) and (A:dnsbox) [13:13:07] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot begin reboot of dns1004.wikimedia.org [13:13:07] !log brouberol@cumin1003 START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad [13:13:13] !log brouberol@cumin1003 START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-jumbo-eqiad [13:13:14] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [13:13:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 13.91% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:13:20] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [13:13:39] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [13:13:40] !log bking@cumin2003 START - Cookbook sre.dns.netbox [13:15:14] !log mvernon@cumin2003 END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw [13:15:43] FIRING: [2x] JobUnavailable: Reduced availability for job jmx_wcqs_blazegraph in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:16:39] FIRING: [14x] RejectingBGPPrefixes: Rejecting 1 BGP prefixes - ssw1-e1-eqiad (10.64.128.1 - group EVPN_IBGP) - https://wikitech.wikimedia.org/wiki/Network_monitoring#RejectingBGPPrefixes - https://alerts.wikimedia.org/?q=alertname%3DRejectingBGPPrefixes [13:17:09] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [13:17:15] !log cmooney@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 15 hosts with reason: switch upgrade and line card install [13:17:16] !log sukhe@cumin1003 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_drmrs [13:17:20] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2187-2196].codfw.wmnet [13:17:27] !log sukhe@cumin1003 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_drmrs [13:17:42] !log mvernon@cumin2003 conftool action : set/pooled=inactive; selector: name=ms-fe2009.codfw.wmnet [13:17:51] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2001 - bking@cumin2003" [13:17:54] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12124189 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=121d96b4-7ab1-4cc6-9b24-75a50e15a132) set by... [13:17:56] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wcqs2001 - bking@cumin2003" [13:17:56] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:17:56] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache wcqs2001.codfw.wmnet 26.16.192.10.in-addr.arpa 6.2.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [13:18:00] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wcqs2001.codfw.wmnet 26.16.192.10.in-addr.arpa 6.2.0.0.6.1.0.0.2.9.1.0.0.1.0.0.2.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [13:18:01] !log bking@cumin2003 START - Cookbook sre.network.configure-switch-interfaces for host wcqs2001 [13:18:50] !log mvernon@cumin2003 conftool action : set/pooled=yes; selector: name=ms-fe2009.codfw.wmnet [13:19:10] FIRING: [8x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:19:11] !log cmooney@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on lvs[1018-1020].eqiad.wmnet with reason: switch upgrade and line card install [13:19:31] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12124196 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=84179922-a401-44a7-95de-38b063d38fb1) set by... [13:20:15] FIRING: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-web releases routed via main (k8s) 1.208s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [13:20:43] FIRING: [2x] JobUnavailable: Reduced availability for job jmx_wcqs_blazegraph in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:21:04] bking@cumin2003 reimage (PID 1709096) is awaiting input [13:21:13] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12124202 (10MatthewVernon) [13:21:39] FIRING: [24x] RejectingBGPPrefixes: Rejecting 1 BGP prefixes - ssw1-e1-eqiad (10.64.128.1 - group EVPN_IBGP) - https://wikitech.wikimedia.org/wiki/Network_monitoring#RejectingBGPPrefixes - https://alerts.wikimedia.org/?q=alertname%3DRejectingBGPPrefixes [13:21:55] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot finished rebooting dns1004.wikimedia.org [13:22:22] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp4045.ulsfo.wmnet [13:22:39] !log bking@cumin2003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wcqs2001 [13:22:39] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wcqs2001 [13:23:03] (03CR) 10Eevans: [C:03+1] thanos: enable an expirer host for the thanos-swift cluster [puppet] - 10https://gerrit.wikimedia.org/r/1310971 (https://phabricator.wikimedia.org/T431506) (owner: 10MVernon) [13:23:48] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp4037.ulsfo.wmnet [13:23:56] (03CR) 10Tiziano Fogli: "It's just a type casting issue, but other than that it looks good to me. Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1307405 (https://phabricator.wikimedia.org/T407117) (owner: 10Hnowlan) [13:25:15] RESOLVED: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-web releases routed via main (k8s) 1.879s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [13:26:39] RESOLVED: [24x] RejectingBGPPrefixes: Rejecting 1 BGP prefixes - ssw1-e1-eqiad (10.64.128.1 - group EVPN_IBGP) - https://wikitech.wikimedia.org/wiki/Network_monitoring#RejectingBGPPrefixes - https://alerts.wikimedia.org/?q=alertname%3DRejectingBGPPrefixes [13:27:28] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wcqs1002.eqiad.wmnet with OS bookworm [13:28:01] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2187-2196].codfw.wmnet [13:28:07] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2187-2196].codfw.wmnet [13:28:08] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp6009.drmrs.wmnet [13:28:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.72% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [13:28:28] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp6001.drmrs.wmnet [13:28:32] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2197-2206].codfw.wmnet [13:29:49] FIRING: HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [13:30:44] (03PS1) 10Gmodena: WIP: wdqs: pin qlever memory request == limit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311052 [13:30:53] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [13:34:27] (03CR) 10Tiziano Fogli: "I agree with all the proposed labels, but I'd like to keep the Prometheus one if possible." [puppet] - 10https://gerrit.wikimedia.org/r/1311029 (https://phabricator.wikimedia.org/T431980) (owner: 10Hnowlan) [13:34:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [13:35:02] (03CR) 10MVernon: [C:03+2] thanos: enable an expirer host for the thanos-swift cluster [puppet] - 10https://gerrit.wikimedia.org/r/1310971 (https://phabricator.wikimedia.org/T431506) (owner: 10MVernon) [13:35:20] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2197-2206].codfw.wmnet [13:36:10] (03PS4) 10Elukey: profile::puppetserver::volatile: add metrics for airflow webrequest [puppet] - 10https://gerrit.wikimedia.org/r/1310118 (https://phabricator.wikimedia.org/T402512) [13:36:24] (03CR) 10Elukey: profile::puppetserver::volatile: add metrics for airflow webrequest (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1310118 (https://phabricator.wikimedia.org/T402512) (owner: 10Elukey) [13:36:55] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot begin reboot of dns1005.wikimedia.org [13:37:44] (03CR) 10Tiziano Fogli: karma: consistently order some labels first (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1310519 (https://phabricator.wikimedia.org/T431980) (owner: 10Hnowlan) [13:42:47] (03PS3) 10Bking: deployment-server: install python commandline parser [puppet] - 10https://gerrit.wikimedia.org/r/1310630 (https://phabricator.wikimedia.org/T431506) [13:43:31] (03CR) 10Bking: "ACK, done." [puppet] - 10https://gerrit.wikimedia.org/r/1310630 (https://phabricator.wikimedia.org/T431506) (owner: 10Bking) [13:43:49] (03CR) 10Brouberol: [C:03+1] "Nicely done!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309204 (https://phabricator.wikimedia.org/T427457) (owner: 10Aqu) [13:43:59] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2197-2206].codfw.wmnet [13:44:06] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2197-2206].codfw.wmnet [13:44:10] FIRING: [9x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:44:25] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2207-2215,2242].codfw.wmnet [13:44:58] (03CR) 10Hnowlan: [C:03+1] profile::puppetserver::volatile: add metrics for airflow webrequest (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1310118 (https://phabricator.wikimedia.org/T402512) (owner: 10Elukey) [13:49:28] !log brouberol@cumin1003 END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad [13:50:37] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2207-2215,2242].codfw.wmnet [13:50:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:51:05] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot finished rebooting dns1005.wikimedia.org [13:51:26] (03CR) 10TChin: [C:03+1] stream: pageview-trending-relative [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311043 (https://phabricator.wikimedia.org/T432204) (owner: 10JavierMonton) [13:52:32] (03PS1) 10Sbisson: Enable Article Guidance on Polish Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311054 (https://phabricator.wikimedia.org/T432137) [13:52:39] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1310630 (https://phabricator.wikimedia.org/T431506) (owner: 10Bking) [13:52:57] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, July 16 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311054 (https://phabricator.wikimedia.org/T432137) (owner: 10Sbisson) [13:53:22] !log switch routing-engine on cr2-eqiad resetting all interfaces T417873 [13:53:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:53:25] T417873: eqiad: upgrade routers (2026) - https://phabricator.wikimedia.org/T417873 [13:55:02] (03CR) 10Btullis: profiles/roles: mount CephFS on /home on ml-lab1002 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1311046 (https://phabricator.wikimedia.org/T380279) (owner: 10Klausman) [13:55:30] (03PS1) 10Jforrester: wikifunctions: Upgrade evaluators from 2026-07-08-113600 to 2026-07-14-153821 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311055 (https://phabricator.wikimedia.org/T423735) [13:55:39] (03PS1) 10Jforrester: wikifunctions: Upgrade orchestrator from 2026-07-07-185549 to 2026-07-14-211423 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311056 (https://phabricator.wikimedia.org/T402857) [13:55:44] !log btullis@cumin1003 START - Cookbook sre.druid.reboot-workers for Druid public cluster: Reboot Druid nodes [13:55:54] !log btullis@cumin1003 START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad [13:56:50] (03PS1) 10Cathal Mooney: Transport circuits: enable OSPF on new cr3-ulsfo -> cr1-eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1311057 (https://phabricator.wikimedia.org/T424839) [13:57:16] (03PS2) 10Klausman: profiles/roles: mount CephFS on /home on ml-lab1002 [puppet] - 10https://gerrit.wikimedia.org/r/1311046 (https://phabricator.wikimedia.org/T380279) [13:57:20] (03CR) 10Klausman: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311046 (https://phabricator.wikimedia.org/T380279) (owner: 10Klausman) [13:57:22] FIRING: [4x] CoreRouterInterfaceDown: Core router interface down - cr1-eqiad:xe-1/0/1:1 (Core: cr2-eqiad:xe-1/0/1:1 {#180180823000240:1}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [13:57:29] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [13:57:38] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [13:58:13] 10SRE-swift-storage, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: RdfStreamingUpdaterSpaceUsageTooHigh - https://phabricator.wikimedia.org/T431506#12124404 (10MatthewVernon) There is now an object-expirer service running on the thanos-swift cluster. [13:58:45] btullis@cumin1003 reboot-workers (PID 137459) is awaiting input [13:58:52] PROBLEM - VRRP status on cr1-eqiad is CRITICAL: VRRP CRITICAL - 6 misconfigured interfaces, 0 inconsistent interfaces https://wikitech.wikimedia.org/wiki/Network_monitoring%23VRRP_status [13:59:10] PROBLEM - BFD status on lsw1-e1-eqiad.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [13:59:30] (03PS3) 10Krinkle: MathML+MathJax rollout to phase 2 (not Wikibooks/Wikisource/Wikipedia) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311041 (https://phabricator.wikimedia.org/T271001) [13:59:37] ^VRRP alert can be ignored that's me, forgot to include cr1 in the downtime doh [13:59:50] (03CR) 10CI reject: [V:04-1] profiles/roles: mount CephFS on /home on ml-lab1002 [puppet] - 10https://gerrit.wikimedia.org/r/1311046 (https://phabricator.wikimedia.org/T380279) (owner: 10Klausman) [13:59:55] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2207-2215,2242].codfw.wmnet [14:00:01] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2207-2215,2242].codfw.wmnet [14:00:05] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260715T1400) [14:00:12] !log cmooney@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on cr1-eqiad with reason: switch upgrade and line card install [14:00:19] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12124411 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=7bebb0cd-fd97-4261-8062-22ac9765f376) set by... [14:00:21] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2243,2248-2256].codfw.wmnet [14:01:33] (03PS3) 10Klausman: profiles/roles: mount CephFS on /home on ml-lab1002 [puppet] - 10https://gerrit.wikimedia.org/r/1311046 (https://phabricator.wikimedia.org/T380279) [14:01:40] (03CR) 10Klausman: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311046 (https://phabricator.wikimedia.org/T380279) (owner: 10Klausman) [14:02:07] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp4046.ulsfo.wmnet [14:02:12] RECOVERY - VRRP status on cr1-eqiad is OK: VRRP OK - 0 misconfigured interfaces, 0 inconsistent interfaces https://wikitech.wikimedia.org/wiki/Network_monitoring%23VRRP_status [14:02:39] RESOLVED: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr2-eqiad (208.80.154.197) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:03:10] RECOVERY - BFD status on lsw1-e1-eqiad.mgmt is OK: UP: 4 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [14:03:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.41% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:03:34] (03PS4) 10Krinkle: MathML+MathJax rollout to phase 2 (not Wikibooks/Wikisource/Wikipedia) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311041 (https://phabricator.wikimedia.org/T271001) [14:04:41] (03CR) 10Btullis: cloudnative-pg-cluster: add SNI-based TLS passthrough ingress (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310603 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [14:05:28] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp4038.ulsfo.wmnet [14:05:56] !log bking@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs2001.codfw.wmnet with OS bookworm [14:06:03] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2243,2248-2256].codfw.wmnet [14:06:05] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot begin reboot of dns1006.wikimedia.org [14:06:05] (03CR) 10Jforrester: [C:03+2] wikifunctions: Upgrade evaluators from 2026-07-08-113600 to 2026-07-14-153821 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311055 (https://phabricator.wikimedia.org/T423735) (owner: 10Jforrester) [14:08:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:08:31] (03Merged) 10jenkins-bot: wikifunctions: Upgrade evaluators from 2026-07-08-113600 to 2026-07-14-153821 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311055 (https://phabricator.wikimedia.org/T423735) (owner: 10Jforrester) [14:09:11] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp6010.drmrs.wmnet [14:09:26] !log jforrester@deploy2003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:09:31] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp6002.drmrs.wmnet [14:10:00] !log bking@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs1002.eqiad.wmnet with OS bookworm [14:10:16] (03CR) 10JavierMonton: [C:03+2] stream: pageview-trending-relative [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311043 (https://phabricator.wikimedia.org/T432204) (owner: 10JavierMonton) [14:10:17] !log jforrester@deploy2003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:10:31] !log jforrester@deploy2003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:10:53] (03CR) 10Btullis: [C:03+2] istio: expose a TLS passthrough port for PostgreSQL on dse-k8s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310601 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [14:10:59] !log jforrester@deploy2003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:11:04] !log jforrester@deploy2003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:11:29] !log jforrester@deploy2003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:11:54] (03CR) 10Jforrester: [C:03+2] wikifunctions: Upgrade orchestrator from 2026-07-07-185549 to 2026-07-14-211423 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311056 (https://phabricator.wikimedia.org/T402857) (owner: 10Jforrester) [14:12:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:12:52] (03Merged) 10jenkins-bot: stream: pageview-trending-relative [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311043 (https://phabricator.wikimedia.org/T432204) (owner: 10JavierMonton) [14:13:18] (03Merged) 10jenkins-bot: istio: expose a TLS passthrough port for PostgreSQL on dse-k8s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310601 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [14:13:46] (03CR) 10Ayounsi: [C:03+1] Transport circuits: enable OSPF on new cr3-ulsfo -> cr1-eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1311057 (https://phabricator.wikimedia.org/T424839) (owner: 10Cathal Mooney) [14:14:03] !log switch routing-engine on cr2-eqiad resetting all interfaces T417873 [14:14:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:14:08] T417873: eqiad: upgrade routers (2026) - https://phabricator.wikimedia.org/T417873 [14:15:39] (03Merged) 10jenkins-bot: wikifunctions: Upgrade orchestrator from 2026-07-07-185549 to 2026-07-14-211423 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311056 (https://phabricator.wikimedia.org/T402857) (owner: 10Jforrester) [14:15:39] !log btullis@cumin1003 START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling reboot on A:schema-eqiad [14:15:39] (03CR) 10Ayounsi: [C:03+1] rpkivalidator: migrate TCP check to blackbox check [puppet] - 10https://gerrit.wikimedia.org/r/1307182 (https://phabricator.wikimedia.org/T407117) (owner: 10Hnowlan) [14:15:39] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot finished rebooting dns1006.wikimedia.org [14:15:39] PROBLEM - BFD status on lsw1-e2-eqiad.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [14:15:52] !log mvernon@cumin1003 START - Cookbook sre.hosts.reimage for host ms-be1064.eqiad.wmnet with OS trixie [14:16:04] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12124467 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-be1064.eqiad.wmnet with OS trixie [14:16:22] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-be2062.codfw.wmnet with OS trixie [14:16:28] !log jforrester@deploy2003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:16:30] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2243,2248-2256].codfw.wmnet [14:16:35] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12124468 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-be2062.codfw.wmnet with OS trixie [14:16:36] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2243,2248-2256].codfw.wmnet [14:16:54] !log jforrester@deploy2003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:16:55] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2257-2266].codfw.wmnet [14:17:06] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [14:17:11] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [14:17:33] !log jforrester@deploy2003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:17:46] FIRING: Not accepting/receiving prefixes from anycast BGP peer: Alert for device asw1-b4-magru.mgmt.magru.wmnet - Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [14:18:15] !log jforrester@deploy2003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:18:25] !log jforrester@deploy2003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:18:30] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.76% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:19:04] !log jforrester@deploy2003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:19:11] RECOVERY - BFD status on lsw1-e2-eqiad.mgmt is OK: UP: 4 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [14:19:24] (03CR) 10Klausman: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311046 (https://phabricator.wikimedia.org/T380279) (owner: 10Klausman) [14:19:35] (03PS1) 10Krinkle: MathML+MathJax rollout to phase 3 (Wikibooks) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311058 (https://phabricator.wikimedia.org/T271001) [14:19:38] (03PS1) 10Krinkle: MathML+MathJax rollout to phase 4 (Wikisource) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311059 (https://phabricator.wikimedia.org/T271001) [14:19:41] (03PS1) 10Krinkle: MathML+MathJax rollout to phase 5 (Wikipedia sans large) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311060 (https://phabricator.wikimedia.org/T271001) [14:19:47] (03PS1) 10Krinkle: MathML+MathJax rollout to phase 6 (remaining large Wikipedias) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311061 (https://phabricator.wikimedia.org/T271001) [14:19:59] (03PS1) 10Krinkle: Remove Mathoid SVG option from wgMathValidModes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311062 (https://phabricator.wikimedia.org/T271001) [14:20:03] (03PS1) 10Krinkle: Simplify Math config, reduce mention of Mathoid, prepare for removal [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311063 (https://phabricator.wikimedia.org/T271001) [14:20:08] !log jforrester@deploy2003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:20:13] !log jforrester@deploy2003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:22:39] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2257-2266].codfw.wmnet [14:24:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.9% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:24:45] (03CR) 10Jforrester: "It'd be great to actually stop users being able to use Mathoid at some point, given we want to switch it off. It's already a continuous st" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311041 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [14:24:53] !log btullis@cumin1003 END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling reboot on A:schema-eqiad [14:25:01] 10SRE-swift-storage, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: RdfStreamingUpdaterSpaceUsageTooHigh - https://phabricator.wikimedia.org/T431506#12124554 (10bking) [14:26:27] (03CR) 10Klausman: [V:03+1] "PCC SUCCESS (NOOP 1 CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1311046 (https://phabricator.wikimedia.org/T380279) (owner: 10Klausman) [14:26:29] (03CR) 10Jforrester: "Do you want to drop the mathoid service definition from wmf-config/ProductionServices.php / wmf-config/LabsServices.php ahead of this, so " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311063 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [14:26:41] 06SRE, 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12124569 (10brouberol) [14:28:08] (03CR) 10Klausman: [V:03+1] profiles/roles: mount CephFS on /home on ml-lab1002 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1311046 (https://phabricator.wikimedia.org/T380279) (owner: 10Klausman) [14:29:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:29:23] (03CR) 10Btullis: profiles/roles: mount CephFS on /home on ml-lab1002 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1311046 (https://phabricator.wikimedia.org/T380279) (owner: 10Klausman) [14:29:32] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2257-2266].codfw.wmnet [14:29:39] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2257-2266].codfw.wmnet [14:29:59] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2267-2276].codfw.wmnet [14:30:00] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot begin reboot of dns2004.wikimedia.org [14:30:05] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260715T1400) [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260715T1430) [14:30:33] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [14:31:08] 06SRE, 06cloud-services-team, 06collaboration-services, 10Infrastructure Security, and 6 others: Reboot cirrussearch in CODFW - https://phabricator.wikimedia.org/T432119#12124591 (10hnowlan) [14:32:08] 06SRE, 06cloud-services-team, 06collaboration-services, 10Infrastructure Security, and 7 others: Reboot cirrussearch in CODFW - https://phabricator.wikimedia.org/T432119#12124605 (10hnowlan) Is there anything observability can help with on testing this? [14:32:14] PROBLEM - BFD status on lsw1-e3-eqiad.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [14:32:35] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [14:32:36] (03CR) 10Kamila Součková: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9010/co" [puppet] - 10https://gerrit.wikimedia.org/r/1310589 (https://phabricator.wikimedia.org/T432108) (owner: 10Kamila Součková) [14:32:43] (03CR) 10Ayounsi: [C:03+1] team-netops: include ibgp group in RejectedPrefixes alert [alerts] - 10https://gerrit.wikimedia.org/r/1311026 (https://phabricator.wikimedia.org/T423430) (owner: 10Cathal Mooney) [14:33:11] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [14:33:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.97% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:33:22] (03PS6) 10RLazarus: wikifunctions: Add mesh.restricted_listeners port to orchestrator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1296071 (https://phabricator.wikimedia.org/T427863) [14:33:23] !log javiermonton@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [14:33:33] (03PS6) 10RLazarus: function-evaluator: Add outgoing Envoy config and egress policy for callbacks [deployment-charts] - 10https://gerrit.wikimedia.org/r/1296072 (https://phabricator.wikimedia.org/T427863) [14:34:11] !log mvernon@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1064.eqiad.wmnet with reason: host reimage [14:34:18] (03PS4) 10Kamila Součková: modules/rsync: add pidfile to stunnel.conf [puppet] - 10https://gerrit.wikimedia.org/r/1310589 (https://phabricator.wikimedia.org/T432108) [14:34:24] (03CR) 10Kamila Součková: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1310589 (https://phabricator.wikimedia.org/T432108) (owner: 10Kamila Součková) [14:35:04] (03CR) 10Scott French: service-catalog: add a new service for mw-pretrain. (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1310999 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [14:35:14] RECOVERY - BFD status on lsw1-e3-eqiad.mgmt is OK: UP: 4 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [14:35:26] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm [14:35:26] !log klausman@cumin1003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{ml-serve1001.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) [14:35:29] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1001.eqiad.wmnet [14:35:33] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wcqs1002.eqiad.wmnet with OS bookworm [14:36:12] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2267-2276].codfw.wmnet [14:36:36] !log disconnect power on cr2-eqiad to shut down device for switch fabric replacement T426343 [14:36:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:36:38] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage [14:36:39] T426343: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343 [14:38:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.97% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:38:16] (03CR) 10Jforrester: [C:03+2] admin_ng: Restrict wikifunctions network access to DNS [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309293 (https://phabricator.wikimedia.org/T427864) (owner: 10RLazarus) [14:38:32] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1064.eqiad.wmnet with reason: host reimage [14:39:10] FIRING: [6x] BFDdown: BFD session down between cr1-codfw and 208.80.153.48 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:40:24] (03CR) 10Kamila Součková: [C:03+1] "Fair question. @jhathaway@wikimedia.org will you be merging it? how do you feel about splitting it or not? I'm fine with it either way (ha" [puppet] - 10https://gerrit.wikimedia.org/r/1305986 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [14:40:32] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1001.eqiad.wmnet [14:40:55] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: db2209 firmware upgrade after a crash - https://phabricator.wikimedia.org/T431952#12124663 (10Jhancock.wm) I looked into this and had a think. I don't think they're going to replace any parts cause it's the first time it's happened and there are no current errors. but I... [14:41:52] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp4047.ulsfo.wmnet [14:42:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr2-eqiad (208.80.154.197) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:43:08] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2062.codfw.wmnet with reason: host reimage [14:43:23] (03CR) 10Btullis: [C:03+1] "Looks good to me. Just a nit on a comment, but feel free to merge and try it out." [puppet] - 10https://gerrit.wikimedia.org/r/1311046 (https://phabricator.wikimedia.org/T380279) (owner: 10Klausman) [14:43:36] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2267-2276].codfw.wmnet [14:43:42] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2267-2276].codfw.wmnet [14:43:48] 06SRE, 10SRE-SLO, 06SRE Observability, 06Traffic: Page on ATS backend errors relative to traffic - https://phabricator.wikimedia.org/T400675#12124674 (10hnowlan) [14:44:02] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2277-2286].codfw.wmnet [14:44:14] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot finished rebooting dns2004.wikimedia.org [14:44:17] !log btullis@cumin1003 START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling reboot on A:schema-codfw [14:44:35] !log bking@cumin2003 START - Cookbook sre.hosts.reboot-single for host relforge1009.eqiad.wmnet [14:45:08] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp4039.ulsfo.wmnet [14:45:48] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1001.eqiad.wmnet [14:46:15] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [14:46:38] !log klausman@cumin1003 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host ml-serve1001.eqiad.wmnet [14:46:38] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{ml-serve1001.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) [14:47:43] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [14:47:45] (03Merged) 10jenkins-bot: admin_ng: Restrict wikifunctions network access to DNS [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309293 (https://phabricator.wikimedia.org/T427864) (owner: 10RLazarus) [14:48:14] PROBLEM - BFD status on lsw1-f1-eqiad.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [14:48:31] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: db2209 firmware upgrade after a crash - https://phabricator.wikimedia.org/T431952#12124687 (10Jhancock.wm) SR: 229101239 [14:49:43] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp6011.drmrs.wmnet [14:49:46] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2277-2286].codfw.wmnet [14:49:52] 06SRE, 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12124698 (10bking) [14:50:05] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host relforge1009.eqiad.wmnet [14:50:06] !log jforrester@deploy2003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:50:11] !log jforrester@deploy2003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:50:26] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp6003.drmrs.wmnet [14:50:27] 06SRE, 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12124703 (10brouberol) [14:51:11] (03CR) 10Jforrester: [C:03+2] "Ah, bother, I can't meaningfully deploy this, can I?" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309293 (https://phabricator.wikimedia.org/T427864) (owner: 10RLazarus) [14:51:35] 06SRE, 10SRE-SLO, 06SRE Observability, 06Traffic: Page on ATS backend errors relative to traffic - https://phabricator.wikimedia.org/T400675#12124708 (10hnowlan) For the most part I think @fgiunchedi's original proposal of a ratio based on requests makes the most sense here. We already do [[ https://gerrit... [14:52:04] (03CR) 10Bking: [C:03+2] cumin: Fix/simplify cirrussearch aliases [puppet] - 10https://gerrit.wikimedia.org/r/1309288 (https://phabricator.wikimedia.org/T431484) (owner: 10Bking) [14:52:14] RECOVERY - BFD status on lsw1-f1-eqiad.mgmt is OK: UP: 4 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [14:52:38] !log btullis@cumin1003 END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling reboot on A:schema-codfw [14:53:43] (03CR) 10Trueg: wdqs: codfw: deploy test qlever image (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310533 (https://phabricator.wikimedia.org/T431271) (owner: 10Gmodena) [14:54:20] (03PS1) 10DCausse: cirrus: set savepoint.dispose-on-delete to true [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311066 (https://phabricator.wikimedia.org/T432249) [14:56:53] (03CR) 10CDobbins: [C:03+2] varnish: put testwiki back into enforce mode [puppet] - 10https://gerrit.wikimedia.org/r/1309263 (owner: 10CDobbins) [14:57:05] (03CR) 10Ebernhardson: [C:03+1] cirrus: set savepoint.dispose-on-delete to true [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311066 (https://phabricator.wikimedia.org/T432249) (owner: 10DCausse) [14:57:44] !log brouberol@cumin1003 END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-jumbo-eqiad [14:58:50] (03PS4) 10Btullis: service: add k8s-ingress-dse-postgresql TLS passthrough service [puppet] - 10https://gerrit.wikimedia.org/r/1311036 (https://phabricator.wikimedia.org/T432104) [14:59:14] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot begin reboot of dns2005.wikimedia.org [15:00:15] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2277-2286].codfw.wmnet [15:00:21] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2277-2286].codfw.wmnet [15:00:41] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2287-2296].codfw.wmnet [15:04:10] FIRING: [7x] BFDdown: BFD session down between cr1-codfw and 208.80.153.74 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:04:19] (03PS1) 10CDobbins: varnish: update CSP report-only header [puppet] - 10https://gerrit.wikimedia.org/r/1311069 [15:05:14] PROBLEM - BFD status on lsw1-f2-eqiad.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [15:05:38] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2062.codfw.wmnet with OS trixie [15:05:51] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12124825 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-be2062.codfw.wmnet with OS trixie completed: - ms-be2062 (**PASS*... [15:06:36] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1064.eqiad.wmnet with OS trixie [15:06:50] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12124829 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-be1064.eqiad.wmnet with OS trixie completed: - ms-be1064 (**PASS*... [15:08:14] RECOVERY - BFD status on lsw1-f2-eqiad.mgmt is OK: UP: 4 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [15:08:28] 06SRE, 06cloud-services-team, 10Infrastructure Security, 06Infrastructure-Foundations, and 5 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12124837 (10klausman) [15:09:10] FIRING: [7x] BFDdown: BFD session down between cr1-codfw and 208.80.153.74 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:09:58] (03PS1) 10Andrew Bogott: scan-and-shrink: use shutil.move instead of os.rename for directories [puppet] - 10https://gerrit.wikimedia.org/r/1311071 (https://phabricator.wikimedia.org/T429578) [15:11:35] !log btullis@cumin1003 END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-eqiad [15:11:56] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2287-2296].codfw.wmnet [15:12:17] (03CR) 10CI reject: [V:04-1] scan-and-shrink: use shutil.move instead of os.rename for directories [puppet] - 10https://gerrit.wikimedia.org/r/1311071 (https://phabricator.wikimedia.org/T429578) (owner: 10Andrew Bogott) [15:12:39] RESOLVED: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr2-eqiad (208.80.154.197) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:13:01] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot finished rebooting dns2005.wikimedia.org [15:13:45] (03PS2) 10Andrew Bogott: scan-and-shrink: use shutil.move instead of os.rename for directories [puppet] - 10https://gerrit.wikimedia.org/r/1311071 (https://phabricator.wikimedia.org/T429578) [15:14:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.69% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:14:22] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-be2063.codfw.wmnet with OS trixie [15:14:34] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12124853 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-be2063.codfw.wmnet with OS trixie [15:16:51] (03PS1) 10Bking: cirrus-streaming-updater: Add alerts for thanos-swift storage utilization [alerts] - 10https://gerrit.wikimedia.org/r/1311073 (https://phabricator.wikimedia.org/T432249) [15:17:24] (03CR) 10Effie Mouzeli: [C:03+1] "iirc, I would recommend prepping the DNS patch and add it as Depends-On" [puppet] - 10https://gerrit.wikimedia.org/r/1311036 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [15:17:26] !log bking@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs1002.eqiad.wmnet with OS bookworm [15:18:00] !log bking@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs2001.codfw.wmnet with OS bookworm [15:18:29] (03CR) 10Andrew Bogott: [C:03+2] scan-and-shrink: use shutil.move instead of os.rename for directories [puppet] - 10https://gerrit.wikimedia.org/r/1311071 (https://phabricator.wikimedia.org/T429578) (owner: 10Andrew Bogott) [15:19:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.97% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:19:34] (03CR) 10Effie Mouzeli: [C:03+1] service: add k8s-ingress-dse-postgresql TLS passthrough service (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1311036 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [15:20:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.35% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:20:25] !log btullis@cumin1003 END (PASS) - Cookbook sre.druid.reboot-workers (exit_code=0) for Druid public cluster: Reboot Druid nodes [15:20:47] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2287-2296].codfw.wmnet [15:20:54] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2287-2296].codfw.wmnet [15:21:13] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2297-2306].codfw.wmnet [15:21:23] !log btullis@cumin1003 START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. [15:21:49] (03PS5) 10Btullis: service: add k8s-ingress-dse-postgresql TLS passthrough service [puppet] - 10https://gerrit.wikimedia.org/r/1311036 (https://phabricator.wikimedia.org/T432104) [15:22:19] (03CR) 10CDobbins: "upload: 0 tests failed, 0 tests skipped, 21 tests passed" [puppet] - 10https://gerrit.wikimedia.org/r/1311069 (owner: 10CDobbins) [15:22:35] (03CR) 10Btullis: service: add k8s-ingress-dse-postgresql TLS passthrough service (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1311036 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [15:23:21] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp4048.ulsfo.wmnet [15:24:07] !log mvernon@cumin1003 START - Cookbook sre.hosts.reimage for host ms-be1065.eqiad.wmnet with OS trixie [15:24:23] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12124879 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-be1065.eqiad.wmnet with OS trixie [15:27:10] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp4040.ulsfo.wmnet [15:27:34] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2297-2306].codfw.wmnet [15:27:43] !log btullis@cumin1003 END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. [15:28:01] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot begin reboot of dns2006.wikimedia.org [15:30:24] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:30:46] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp6012.drmrs.wmnet [15:31:19] !log btullis@cumin1003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw [15:31:23] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet [15:31:24] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet [15:31:28] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp6004.drmrs.wmnet [15:32:18] (03PS2) 10Blake: wmnet: Add CNAME records for mw-pretrain. [dns] - 10https://gerrit.wikimedia.org/r/1311074 (https://phabricator.wikimedia.org/T427668) [15:33:53] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2063.codfw.wmnet with reason: host reimage [15:34:10] FIRING: [8x] BFDdown: BFD session down between cr1-codfw and 208.80.153.107 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:34:20] (03PS2) 10Blake: service-catalog: add a new service for mw-pretrain. [puppet] - 10https://gerrit.wikimedia.org/r/1310999 (https://phabricator.wikimedia.org/T427668) [15:36:45] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet [15:36:47] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet [15:36:53] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet [15:36:54] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet [15:37:13] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2297-2306].codfw.wmnet [15:37:19] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2297-2306].codfw.wmnet [15:37:26] (03CR) 10Blake: service-catalog: add a new service for mw-pretrain. (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1310999 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [15:37:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr2-eqiad (208.80.154.197) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:37:43] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2307-2316].codfw.wmnet [15:38:29] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2063.codfw.wmnet with reason: host reimage [15:39:10] FIRING: [8x] BFDdown: BFD session down between cr1-codfw and 208.80.153.107 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:41:39] FIRING: KubernetesAPILatency: High Kubernetes API latency (GET deployments) on k8s-dse@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s-dse&var-latency_percentile=0.95&var-verb=GET - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [15:42:08] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot finished rebooting dns2006.wikimedia.org [15:42:15] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet [15:42:17] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet [15:42:22] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet [15:42:53] !log mvernon@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1065.eqiad.wmnet with reason: host reimage [15:42:54] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet [15:43:26] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2307-2316].codfw.wmnet [15:44:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.52% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:44:23] (03PS2) 10CDobbins: varnish: update CSP report-only header [puppet] - 10https://gerrit.wikimedia.org/r/1311069 [15:46:06] (03CR) 10Krinkle: "Are those fatals in prod or beta? I'm guessing third-party traffic?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311041 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [15:46:39] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (GET deployments) on k8s-dse@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s-dse&var-latency_percentile=0.95&var-verb=GET - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [15:47:58] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be1065.eqiad.wmnet with reason: host reimage [15:48:04] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for mbsantos - https://phabricator.wikimedia.org/T432258 (10MSantos) 03NEW [15:48:28] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet [15:48:29] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet [15:48:35] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet [15:48:39] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for mbsantos - https://phabricator.wikimedia.org/T432258#12124942 (10MSantos) Requesting it as the KR owner for ST6.4 and ST6.6. [15:49:10] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet [15:49:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.65% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:50:46] (03CR) 10Jforrester: "This is prod; there is no WF beta. However, it might have been fixed, as I can't find any fatals for the past few months at least." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311041 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [15:51:24] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2307-2316].codfw.wmnet [15:51:30] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2307-2316].codfw.wmnet [15:51:50] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2317-2326].codfw.wmnet [15:54:35] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet [15:54:37] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet [15:54:42] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet [15:54:43] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet [15:57:08] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot begin reboot of dns3003.wikimedia.org [15:57:31] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2317-2326].codfw.wmnet [16:00:13] 06SRE, 10SRE-SLO, 06SRE Observability, 06Traffic: Page on ATS backend errors relative to traffic - https://phabricator.wikimedia.org/T400675#12125007 (10fgiunchedi) Thank you for the feedback @hnowlan @ssingh ! re: exclusion list I'm also expecting that moving to ratio-based paging will carry more signal (... [16:00:34] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2063.codfw.wmnet with OS trixie [16:00:45] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet [16:00:46] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet [16:00:48] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12125010 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-be2063.codfw.wmnet with OS trixie completed: - ms-be2063 (**PASS*... [16:00:50] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet [16:00:58] PROBLEM - BFD status on asw1-by27-esams.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [16:01:48] (03CR) 10Btullis: "Good thinking. Thanks." [puppet] - 10https://gerrit.wikimedia.org/r/1311036 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [16:01:50] (03Abandoned) 10Simon04: Replace w.wiki links with their target, use Special:MyLanguage [puppet] - 10https://gerrit.wikimedia.org/r/1306094 (owner: 10Simon04) [16:02:37] (03CR) 10Btullis: [C:03+2] cloudnative-pg-cluster: add SNI-based TLS passthrough ingress (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310603 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [16:02:58] RECOVERY - BFD status on asw1-by27-esams.mgmt is OK: UP: 2 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [16:03:08] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp4049.ulsfo.wmnet [16:03:59] (03CR) 10Physikerwelt: "I would love to switch off Mathoid as fast as possible. Usually, you can't even tell if the SVG image was generated server-side or client-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311041 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [16:04:35] (03CR) 10Scott French: [C:03+1] "Thanks, Blake!" [puppet] - 10https://gerrit.wikimedia.org/r/1310999 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [16:05:14] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-analytics-product: apply [16:05:21] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-analytics-product: apply [16:05:28] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-analytics-test: apply [16:05:35] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-analytics-test: apply [16:05:57] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet [16:06:06] (03Merged) 10jenkins-bot: cloudnative-pg-cluster: add SNI-based TLS passthrough ingress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310603 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [16:06:58] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2317-2326].codfw.wmnet [16:07:04] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2317-2326].codfw.wmnet [16:07:29] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2327-2336].codfw.wmnet [16:07:39] RESOLVED: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr2-eqiad (208.80.154.197) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [16:08:35] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot finished rebooting dns3003.wikimedia.org [16:08:58] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp4041.ulsfo.wmnet [16:09:51] (03CR) 10Gmodena: wdqs: codfw: deploy test qlever image (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1310533 (https://phabricator.wikimedia.org/T431271) (owner: 10Gmodena) [16:10:43] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:11:13] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12125056 (10MatthewVernon) [16:11:33] (03CR) 10RLazarus: "No, this one needs root. I can apply it today if that works for you." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309293 (https://phabricator.wikimedia.org/T427864) (owner: 10RLazarus) [16:11:50] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp6013.drmrs.wmnet [16:12:15] (03CR) 10Scott French: "Thanks, Blake!" [dns] - 10https://gerrit.wikimedia.org/r/1311074 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [16:12:31] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp6005.drmrs.wmnet [16:12:33] !log mvernon@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be1065.eqiad.wmnet with OS trixie [16:12:38] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet [16:12:39] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet [16:12:40] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12125062 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-be1065.eqiad.wmnet with OS trixie completed: - ms-be1065 (**PASS*... [16:12:45] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet [16:13:14] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2327-2336].codfw.wmnet [16:15:43] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:17:22] FIRING: [5x] CoreRouterInterfaceDown: Core router interface down - cr1-esams:xe-0/0/7 (Transport: cr2-eqiad:xe-3/2/1 (Colt, 445419311 80ms 10Gbps wave) {#30385}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [16:17:39] FIRING: [14x] CoreBGPDown: Core BGP session down between cr1-codfw and cr2-eqiad (208.80.154.197) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [16:17:57] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet [16:19:03] (03PS1) 10Pppery: Update translations [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1311078 [16:19:10] FIRING: [25x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [16:20:23] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for cr2-eqiad.wikimedia.org is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:22:54] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2327-2336].codfw.wmnet [16:23:00] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2327-2336].codfw.wmnet [16:23:15] (03CR) 10Jforrester: [C:03+2] "Sure, sorry for the noise." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1309293 (https://phabricator.wikimedia.org/T427864) (owner: 10RLazarus) [16:23:20] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2337-2346].codfw.wmnet [16:23:35] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot begin reboot of dns3004.wikimedia.org [16:24:39] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet [16:24:41] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet [16:24:46] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet [16:26:58] PROBLEM - BFD status on asw1-bw27-esams.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [16:29:00] RECOVERY - BFD status on asw1-bw27-esams.mgmt is OK: UP: 2 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [16:29:10] FIRING: [21x] BFDdown: BFD session down between asw1-bw27-esams and 185.15.59.2 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [16:29:40] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2337-2346].codfw.wmnet [16:29:59] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet [16:30:35] Howdy! Can anyone advise me on how to get approved for the ops mailing list? I put in a request about a month ago and also emailed the list owner two weeks ago. [16:30:53] (03PS2) 10Pppery: Drop "Variable types" spew from autogenerated documentation [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1309274 (https://phabricator.wikimedia.org/T428958) [16:30:54] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-dev: apply [16:32:09] (03PS2) 10Pppery: Move most punctuation out of PLURAL [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1309287 (https://phabricator.wikimedia.org/T431715) [16:32:35] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-dev: apply [16:32:40] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-fr-tech: apply [16:32:46] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-fr-tech: apply [16:32:51] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-main: apply [16:33:00] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-main: apply [16:33:35] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-ml: apply [16:33:41] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-ml: apply [16:34:10] FIRING: [21x] BFDdown: BFD session down between asw1-bw27-esams and 185.15.59.2 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [16:34:45] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot finished rebooting dns3004.wikimedia.org [16:35:19] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet [16:35:21] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet [16:35:21] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw [16:36:48] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2337-2346].codfw.wmnet [16:36:54] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2337-2346].codfw.wmnet [16:37:14] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2347-2356].codfw.wmnet [16:40:15] (03CR) 10Krinkle: "@James: Ack. That issue was fixed in [change 1133948](https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1133948/2/wmf-config/" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311041 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [16:42:17] (03PS1) 10Scott French: P:kubernetes::deployment_server::mediawiki: mediawiki-deployments v2 [puppet] - 10https://gerrit.wikimedia.org/r/1311081 [16:43:00] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2347-2356].codfw.wmnet [16:43:08] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp4050.ulsfo.wmnet [16:44:18] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-platform-eng: apply [16:44:24] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-platform-eng: apply [16:44:30] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-research: apply [16:44:37] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-research: apply [16:49:34] (03CR) 10Ahmon Dancy: [C:03+1] "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/1311081 (owner: 10Scott French) [16:49:45] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot begin reboot of dns5003.wikimedia.org [16:50:11] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2347-2356].codfw.wmnet [16:50:17] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2347-2356].codfw.wmnet [16:50:25] (03CR) 10BCornwall: [C:03+2] Enable auth rate_limiting_flags on upload [puppet] - 10https://gerrit.wikimedia.org/r/1310151 (https://phabricator.wikimedia.org/T431627) (owner: 10BCornwall) [16:50:39] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp4042.ulsfo.wmnet [16:50:40] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2357-2365].codfw.wmnet [16:51:37] !log rzl@deploy2003 helmfile [staging-codfw] START helmfile.d/admin 'apply'. [16:52:17] !log rzl@deploy2003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [16:52:46] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp6014.drmrs.wmnet [16:52:55] !log rzl@deploy2003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [16:53:13] (03CR) 10BCornwall: [V:03+1 C:03+2] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9013/co" [puppet] - 10https://gerrit.wikimedia.org/r/1310151 (https://phabricator.wikimedia.org/T431627) (owner: 10BCornwall) [16:53:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 4.477% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:53:21] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp6006.drmrs.wmnet [16:54:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 3.753% idle #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:54:49] !incidents [16:54:50] 8200 (ACKED) PHPFPMTooBusy sre (mw-web main eqiad) [16:55:06] (03PS5) 10CDobbins: bird: ensure that the `bird` user owns `/etc/bird` [puppet] - 10https://gerrit.wikimedia.org/r/1310160 [16:55:51] !log rzl@deploy2003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [16:55:59] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2357-2365].codfw.wmnet [16:56:15] FIRING: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-web releases routed via main (k8s) 1.671s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [16:56:20] (stopping there, won't touch prod wikikube until that page is resolved) [16:57:23] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-search: apply [16:57:30] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-search: apply [16:57:37] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-sre: apply [16:57:43] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-sre: apply [16:59:10] FIRING: [23x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [16:59:28] (03CR) 10BCornwall: [V:03+2 C:03+2] "Tests pass" [puppet] - 10https://gerrit.wikimedia.org/r/1310151 (https://phabricator.wikimedia.org/T431627) (owner: 10BCornwall) [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260715T1700) [17:00:12] (03CR) 10Krinkle: "I was going to but then stumbled upon `$wgMathMathMLUrl`." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311063 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [17:01:22] (03CR) 10Btullis: "Although I don't think that this is a hard dependency, because it's only in the `service_setup` state right now. It will need a backend po" [puppet] - 10https://gerrit.wikimedia.org/r/1311036 (https://phabricator.wikimedia.org/T432104) (owner: 10Btullis) [17:02:22] FIRING: [10x] CoreRouterInterfaceDown: Core router interface down - cr2-eqiad:et-0/0/1 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [17:03:03] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply [17:03:04] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2357-2365].codfw.wmnet [17:03:10] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-test-k8s: apply [17:03:10] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2357-2365].codfw.wmnet [17:03:15] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-wikidata: apply [17:03:22] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-wikidata: apply [17:03:27] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-wmde: apply [17:03:29] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2366-2374].codfw.wmnet [17:03:34] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-wmde: apply [17:03:39] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply [17:03:45] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply [17:03:48] !log cwilliams@cumin1003 START - Cookbook sre.mysql.depool depool db2241: Security update [17:04:10] FIRING: [28x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:04:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 4.545% idle #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:04:42] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2241: Security update [17:06:03] !log sukhe@cumin1003 cookbooks.sre.dns.roll-reboot finished rebooting dns5003.wikimedia.org [17:06:08] !log sukhe@cumin1003 END (ERROR) - Cookbook sre.dns.roll-reboot (exit_code=97) rolling reboot on A:dnsbox and not (A:ulsfo or A:magru) and (A:dnsbox) [17:06:19] !log sre.dns.roll-reboot to resume later [17:06:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:08:48] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2366-2374].codfw.wmnet [17:10:01] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply [17:10:07] !log btullis@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply [17:11:15] RESOLVED: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-web releases routed via main (k8s) 1.097s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [17:11:45] FIRING: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-web releases routed via main (k8s) 1.19s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [17:11:45] (03CR) 10CDobbins: [C:03+2] bird: ensure that the `bird` user owns `/etc/bird` [puppet] - 10https://gerrit.wikimedia.org/r/1310160 (owner: 10CDobbins) [17:15:35] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[2366-2374].codfw.wmnet [17:15:40] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[2366-2374].codfw.wmnet [17:15:44] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker-exp2001.codfw.wmnet [17:16:17] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker-exp2001.codfw.wmnet [17:17:18] !log cwilliams@cumin1003 START - Cookbook sre.hosts.remove-downtime for db2241.codfw.wmnet [17:17:19] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db2241.codfw.wmnet [17:19:28] PROBLEM - Check if ntpsec.service has been restarted after /etc/ntpsec/ntp.conf was changed on dns7002 is CRITICAL: CRITICAL: Service ntpsec.service has not been restarted after /etc/ntpsec/ntp.conf was changed (gt 2h). https://wikitech.wikimedia.org/wiki/NTP%23Monitoring [17:19:53] !log cwilliams@cumin1003 START - Cookbook sre.mysql.pool pool db2241: Security update [17:21:30] RESOLVED: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-web releases routed via main (k8s) 1.003s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [17:22:01] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [17:22:15] !log cgoubert@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker-exp2001.codfw.wmnet [17:22:16] !log lerickson@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [17:22:17] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker-exp2001.codfw.wmnet [17:22:17] !log cgoubert@cumin1003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-worker-codfw [17:22:23] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [17:25:50] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [17:25:52] (03PS1) 10Hnowlan: changeprop-jobqueue: drop concurrency for problem jobs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311087 [17:27:43] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [17:28:16] (03CR) 10RLazarus: [C:03+1] changeprop-jobqueue: drop concurrency for problem jobs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311087 (owner: 10Hnowlan) [17:28:32] (03CR) 10Scott French: [C:03+1] changeprop-jobqueue: drop concurrency for problem jobs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311087 (owner: 10Hnowlan) [17:28:48] (03CR) 10Hnowlan: [C:03+2] changeprop-jobqueue: drop concurrency for problem jobs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311087 (owner: 10Hnowlan) [17:30:20] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp4043.ulsfo.wmnet [17:30:57] (03Merged) 10jenkins-bot: changeprop-jobqueue: drop concurrency for problem jobs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311087 (owner: 10Hnowlan) [17:31:28] !log hnowlan@deploy1003 helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [17:31:30] FIRING: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-web releases routed via main (k8s) 1.037s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [17:32:34] !log hnowlan@deploy1003 helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [17:32:45] FIRING: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@codfw is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [17:33:39] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp6015.drmrs.wmnet [17:34:14] PROBLEM - haproxy process on cp4051 is CRITICAL: PROCS CRITICAL: 0 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy [17:34:16] PROBLEM - HAProxy HTTPS upload.wikimedia.org ECDSA on cp4051 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [17:34:16] PROBLEM - HAProxy HTTPS measure-eqiad.wikimedia.org ECDSA on cp4051 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [17:34:34] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp6007.drmrs.wmnet [17:37:20] FIRING: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 772.0467249999998 times in the last 10 minutes due to memory usage (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [17:39:53] !log hnowlan@deploy1003 helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [17:40:14] RECOVERY - haproxy process on cp4051 is OK: PROCS OK: 2 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy [17:40:16] RECOVERY - HAProxy HTTPS measure-eqiad.wikimedia.org ECDSA on cp4051 is OK: SSL OK - Certificate measure-eqiad.wikimedia.org contains all required SANs:Certificate measure-eqiad.wikimedia.org (ECDSA) valid until 2026-10-03 14:54:24 +0000 (expires in 79 days) https://wikitech.wikimedia.org/wiki/HTTPS [17:40:16] RECOVERY - HAProxy HTTPS upload.wikimedia.org ECDSA on cp4051 is OK: SSL OK - Certificate upload.wikimedia.org contains all required SANs:Certificate upload.wikimedia.org (ECDSA) valid until 2026-09-10 05:24:57 +0000 (expires in 56 days) https://wikitech.wikimedia.org/wiki/HTTPS [17:40:27] !log hnowlan@deploy1003 helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [17:41:49] FIRING: HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [17:42:20] RESOLVED: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 133.33333333333331 times in the last 10 minutes due to memory usage (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [17:42:24] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp4051.ulsfo.wmnet [17:42:45] RESOLVED: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@codfw is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [17:44:04] (03CR) 10Jforrester: "So we'd drop both later? OK." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311063 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [17:47:27] !log cdobbins@cumin2003 START - Cookbook sre.hosts.reimage for host dns7002.wikimedia.org with OS trixie [17:50:50] 06SRE, 10SRE-Access-Requests: Requesting access to Superset for jniren-ctr - https://phabricator.wikimedia.org/T432273 (10soworu) 03NEW [17:51:04] PROBLEM - BFD status on asw1-b4-magru.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [17:51:30] RESOLVED: MediaWikiLatencyExceeded: p75 latency high: eqiad mw-web releases routed via main (k8s) 801.3ms - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [17:51:44] FIRING: KubernetesDeploymentUnavailableReplicas: ... [17:51:44] Deployment mw-web.eqiad.main in mw-web at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=mw-web&var-deployment=mw-web.eqiad.main - ... [17:51:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [17:52:31] RESOLVED: Not accepting/receiving prefixes from anycast BGP peer: Device asw1-b4-magru.mgmt.magru.wmnet recovered from Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [17:52:39] RESOLVED: [2x] CoreBGPDown: Core BGP session down between cr2-eqord and cr1-eqiad (208.80.154.196) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [17:54:10] FIRING: [26x] BFDdown: BFD session down between asw1-b4-magru and 195.200.68.37 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [17:54:48] yes that's fine [17:54:50] reimaging [17:55:43] FIRING: [3x] JobUnavailable: Reduced availability for job haproxy in ops@magru - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:56:37] !log start draining traffic on cr1-eqiad ahead of line card installation T426343 [17:56:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:56:41] T426343: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343 [17:59:27] 06SRE, 10SRE-Access-Requests: Requesting access to Superset for jniren-ctr - https://phabricator.wikimedia.org/T432273#12125593 (10AAlikhan) This is approved on my end. [18:00:05] jeena and hashar: Deploy window MediaWiki train - Utc-7+Utc-0 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260715T1800) [18:02:05] jeena, hashar: hold the train for just a sec please [18:05:22] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2241: Security update [18:05:32] Rzl will do [18:05:43] FIRING: [3x] JobUnavailable: Reduced availability for job haproxy in ops@magru - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:08:58] sukhe@cumin1003 roll-reboot (PID 130298) is awaiting input [18:10:10] !log sukhe@cumin1003 END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-upload_drmrs [18:12:01] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp4044.ulsfo.wmnet [18:12:01] !log sukhe@cumin1003 END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_ulsfo [18:14:02] jeena: feel free to go ahead and roll the train [18:14:30] we'll keep an eye on how things respond, which may have implications for the upcoming backport window [18:14:43] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp6016.drmrs.wmnet [18:14:43] !log sukhe@cumin1003 END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_drmrs [18:15:35] !log cdobbins@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on dns7002.wikimedia.org with reason: host reimage [18:16:44] Thank you, I will roll forward in about 15 min or so [18:16:52] * swfrench-wmf thumbs up [18:19:49] !log cdobbins@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns7002.wikimedia.org with reason: host reimage [18:22:11] !log sukhe@cumin1003 cookbooks.sre.cdn.roll-reboot finished rebooting cp4052.ulsfo.wmnet [18:22:11] !log sukhe@cumin1003 END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_ulsfo [18:23:37] 06SRE, 06Infrastructure-Foundations, 10netops: GSHUT (and other?) community matching/actions not working on SR-Linux - https://phabricator.wikimedia.org/T430810#12125726 (10cmooney) Going to close this one, with 'graceful-shutdown sender' configured on cr1-eqiad I can see that the community is matches on ssw... [18:23:49] 06SRE, 06Infrastructure-Foundations, 10netops: GSHUT (and other?) community matching/actions not working on SR-Linux - https://phabricator.wikimedia.org/T430810#12125727 (10cmooney) 05Open→03Resolved a:03cmooney [18:26:27] PROBLEM - Recursive DNS on 195.200.68.37 is CRITICAL: DNS_QUERY CRITICAL - query timed out https://wikitech.wikimedia.org/wiki/DNS [18:27:32] !log cmooney@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 14 hosts with reason: linecard install in cr1-eqad [18:27:42] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12125735 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=b3a498bf-e4dc-461c-9cbe-4c3255e0019c) set by... [18:29:16] !log cmooney@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs[1018-1020].eqiad.wmnet with reason: line card install in cr1-eqiad [18:29:26] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12125749 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=944ebf34-ac03-4b88-9838-2666b665680e) set by... [18:29:36] !log pull power on cr1-eqiad to install new switch-control boards T426343 [18:29:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:29:39] T426343: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343 [18:32:11] log messages you don't expect to see often: "pull power on cr" [18:32:42] now we can officially blame Cathal for the network, with receipts [18:33:07] mutante: haha yeah [18:35:43] FIRING: [3x] JobUnavailable: Reduced availability for job haproxy in ops@magru - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:37:23] PROBLEM - OSPF status on cr2-eqsin is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:38:37] (03PS1) 10TrainBranchBot: group1 to 1.47.0-wmf.11 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311098 (https://phabricator.wikimedia.org/T430830) [18:38:40] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by jhuneidi@deploy2003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311098 (https://phabricator.wikimedia.org/T430830) (owner: 10TrainBranchBot) [18:39:46] (03Merged) 10jenkins-bot: group1 to 1.47.0-wmf.11 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311098 (https://phabricator.wikimedia.org/T430830) (owner: 10TrainBranchBot) [18:39:54] ^^ OSPF is me, I forgot we had that enabled to Singapore on that one, it's not in use [18:40:43] FIRING: [3x] JobUnavailable: Reduced availability for job haproxy in ops@magru - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:42:06] (03PS1) 10Dzahn: gerrit: add other gerrit server names to ssh host aliases [puppet] - 10https://gerrit.wikimedia.org/r/1311100 (https://phabricator.wikimedia.org/T398401) [18:42:14] !log sukhe@cumin1003 START - Cookbook sre.hosts.reboot-single for host ncmonitor1001.eqiad.wmnet [18:43:39] FIRING: [3x] CoreBGPDown: Core BGP session down between cr2-codfw and cr1-eqiad (208.80.154.196) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [18:44:00] !log kamila@cumin1003 START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1262.eqiad.wmnet [18:44:04] !log kamila@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1262.eqiad.wmnet [18:45:12] !log kamila@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1262.eqiad.wmnet [18:45:29] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/admin 'apply'. [18:45:34] !log kamila@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-worker1262.eqiad.wmnet with OS trixie [18:46:00] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/admin 'apply'. [18:46:02] !log sukhe@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ncmonitor1001.eqiad.wmnet [18:46:09] !log jhuneidi@deploy2003 rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.11 refs T430830 [18:46:13] T430830: 1.47.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T430830 [18:46:14] !log kamila@cumin1003 START - Cookbook sre.hosts.move-vlan for host wikikube-worker1262 [18:46:47] !log kamila@cumin1003 START - Cookbook sre.dns.netbox [18:46:54] (03CR) 10Majavah: "why is this using `ipresolve()`? even in the example, you're connecting to the hostname, not the IP address?" [puppet] - 10https://gerrit.wikimedia.org/r/1311100 (https://phabricator.wikimedia.org/T398401) (owner: 10Dzahn) [18:50:01] (03CR) 10DCausse: [C:03+2] cirrus: set savepoint.dispose-on-delete to true [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311066 (https://phabricator.wikimedia.org/T432249) (owner: 10DCausse) [18:50:25] RECOVERY - Recursive DNS on 195.200.68.37 is OK: DNS_QUERY OK - Success https://wikitech.wikimedia.org/wiki/DNS [18:51:36] !log kamila@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1262 - kamila@cumin1003" [18:51:40] !log kamila@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1262 - kamila@cumin1003" [18:51:40] !log kamila@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [18:51:40] !log kamila@cumin1003 START - Cookbook sre.dns.wipe-cache wikikube-worker1262.eqiad.wmnet 72.32.64.10.in-addr.arpa 2.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [18:51:44] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [18:51:44] Deployment mw-web.eqiad.main in mw-web at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=mw-web&var-deployment=mw-web.eqiad.main - ... [18:51:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [18:51:44] !log kamila@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1262.eqiad.wmnet 72.32.64.10.in-addr.arpa 2.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [18:51:45] !log kamila@cumin1003 START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1262 [18:52:19] (03Merged) 10jenkins-bot: cirrus: set savepoint.dispose-on-delete to true [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311066 (https://phabricator.wikimedia.org/T432249) (owner: 10DCausse) [18:52:27] !log kamila@cumin1003 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1262 [18:52:27] !log kamila@cumin1003 END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1262 [18:53:03] RECOVERY - BFD status on asw1-b4-magru.mgmt is OK: UP: 2 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [18:53:54] !log dcausse@deploy2003 helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [18:54:10] !log dcausse@deploy2003 helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [18:54:10] FIRING: [9x] BFDdown: BFD session down between asw1-b4-magru and 195.200.68.37 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [18:54:18] this should be resolved now [18:54:27] well, though the BGP session is still down as expected [18:55:43] FIRING: [2x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:57:23] RECOVERY - OSPF status on cr2-eqsin is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:58:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [18:59:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [18:59:53] !log cdobbins@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dns7002.wikimedia.org with OS trixie [19:04:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:07:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.52% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:08:55] (03PS1) 10Jforrester: wikifunctions: Configure wgWikiLambdaClientRepoSiteId so RC entries point correctly [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311106 [19:11:41] !log kamila@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1262.eqiad.wmnet with reason: host reimage [19:12:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.52% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:17:19] !log kamila@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1262.eqiad.wmnet with reason: host reimage [19:20:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:21:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-jobrunner - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [19:21:39] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12125906 (10cmooney) We have successfully installed the new switch-control boards and and MPC10E in cr2-eqiad. All is wor... [19:24:10] (03CR) 10Bking: [C:04-1] "To avoid noisy alerts, we should wait for the output of `swift_account_stats_bytes_total{account="AUTH_search-update-pipeline"}` to drop b" [alerts] - 10https://gerrit.wikimedia.org/r/1311073 (https://phabricator.wikimedia.org/T432249) (owner: 10Bking) [19:25:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.07% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:26:15] RESOLVED: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-jobrunner - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [19:28:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 21.51% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:31:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [19:32:30] (03PS2) 10Dzahn: gerrit: add other gerrit server names to ssh host aliases [puppet] - 10https://gerrit.wikimedia.org/r/1311100 (https://phabricator.wikimedia.org/T398401) [19:33:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.97% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:34:05] (03CR) 10CDobbins: "@sbassett@wikimedia.org sorry to bug you again, but could you look this over? This patch supersedes both of the prior ones (1308772 and 13" [puppet] - 10https://gerrit.wikimedia.org/r/1311069 (owner: 10CDobbins) [19:35:09] (03Abandoned) 10CDobbins: varnish: update CSP report-only header [puppet] - 10https://gerrit.wikimedia.org/r/1308772 (owner: 10CDobbins) [19:35:16] (03Abandoned) 10CDobbins: varnish: put testwiki back into enforce mode [puppet] - 10https://gerrit.wikimedia.org/r/1309263 (owner: 10CDobbins) [19:38:43] !log kamila@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1262.eqiad.wmnet with OS trixie [19:42:04] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@eqiad in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [19:43:58] (03CR) 10Cathal Mooney: [C:03+2] team-netops: include ibgp group in RejectedPrefixes alert [alerts] - 10https://gerrit.wikimedia.org/r/1311026 (https://phabricator.wikimedia.org/T423430) (owner: 10Cathal Mooney) [19:44:14] !log bking@cumin2003 START - Cookbook sre.hosts.reboot-single for host relforge1010.eqiad.wmnet [19:44:32] (03CR) 10Cathal Mooney: [C:03+2] Transport circuits: enable OSPF on new cr3-ulsfo -> cr1-eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1311057 (https://phabricator.wikimedia.org/T424839) (owner: 10Cathal Mooney) [19:45:27] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm [19:45:28] !log bking@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs2001.codfw.wmnet with OS bookworm [19:45:54] (03Merged) 10jenkins-bot: Transport circuits: enable OSPF on new cr3-ulsfo -> cr1-eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1311057 (https://phabricator.wikimedia.org/T424839) (owner: 10Cathal Mooney) [19:46:10] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm [19:47:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.62% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:47:49] !log arlolra@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [19:48:20] !log arlolra@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [19:48:21] !log arlolra@deploy2003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [19:48:46] !log arlolra@deploy2003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [19:52:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.38% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [19:52:38] !log dcausse@deploy2003 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply [19:52:48] !log dcausse@deploy2003 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply [19:55:09] !log bking@cumin2003 END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host relforge1010.eqiad.wmnet [19:57:31] FIRING: Not accepting/receiving prefixes from anycast BGP peer: Alert for device asw1-b4-magru.mgmt.magru.wmnet - Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [20:01:40] !log disabling puppet fleet wide to roll out kafka patch [20:01:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:02:15] !log dcausse@deploy2003 helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply [20:02:20] !log dcausse@deploy2003 helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply [20:03:29] !log bking@cumin2003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wcqs2001.codfw.wmnet with OS bookworm [20:03:49] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wcqs1002.eqiad.wmnet with OS bookworm [20:04:10] FIRING: [7x] BFDdown: BFD session down between cr2-codfw and 2620:0:860:1:208:80:153:6 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [20:04:28] !log bking@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wcqs2001'] [20:05:08] !log dcausse@deploy2003 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [20:05:16] !log dcausse@deploy2003 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [20:08:48] !log dcausse@deploy2003 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [20:09:00] !log dcausse@deploy2003 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [20:09:03] (03CR) 10JHathaway: [C:03+2] kafka_config: migrate functions to the 4.x API [puppet] - 10https://gerrit.wikimedia.org/r/1308760 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [20:09:10] FIRING: [7x] BFDdown: BFD session down between cr2-codfw and 2620:0:860:1:208:80:153:6 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [20:11:28] !log bking@cumin2003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wcqs2001'] [20:12:17] !log bking@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wcqs2001'] [20:17:25] !log bking@cumin2003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wcqs2001'] [20:18:04] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm [20:19:10] RESOLVED: [7x] BFDdown: BFD session down between cr2-codfw and 2620:0:860:1:208:80:153:6 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [20:20:23] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [20:21:24] !puppet is re-enabled, have fun, but not too much fun! [20:21:24] Sorry, you are not authorized to perform this [20:21:35] !log puppet is re-enabled, have fun, but not too much fun! [20:21:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:24:18] PROBLEM - NTP peers and stratum check on dns7002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/NTP [20:26:39] FIRING: RejectingBGPPrefixes: Rejecting 1 BGP prefixes - dns2005 (208.80.153.74 - group Anycast4) - https://wikitech.wikimedia.org/wiki/Network_monitoring#RejectingBGPPrefixes - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr2-codfw:9804&var-bgp_group=Anycast4&var-bgp_neighbor=dns2005 - https://alerts.wikimedia.org/?q=alertname%3DRejectingBGPPrefixes [20:29:10] FIRING: [20x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:38 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [20:31:39] FIRING: [2x] RejectingBGPPrefixes: Rejecting 1 BGP prefixes - dns2005 (208.80.153.74 - group Anycast4) - https://wikitech.wikimedia.org/wiki/Network_monitoring#RejectingBGPPrefixes - https://alerts.wikimedia.org/?q=alertname%3DRejectingBGPPrefixes [20:34:16] (03CR) 10RLazarus: [C:03+1] P:kubernetes::deployment_server::mediawiki: mediawiki-deployments v2 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1311081 (owner: 10Scott French) [20:35:40] (03PS2) 10Ryan Kemper: cirrus-streaming-updater: Add alerts for thanos-swift storage utilization [alerts] - 10https://gerrit.wikimedia.org/r/1311073 (https://phabricator.wikimedia.org/T432249) (owner: 10Bking) [20:36:15] (03PS6) 10Bking: stat hosts: Set up S3 config for wikidata platform team. [puppet] - 10https://gerrit.wikimedia.org/r/1310643 (https://phabricator.wikimedia.org/T432167) [20:39:59] (03PS6) 10BCornwall: varnish: Configure text for thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/1306896 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [20:39:59] (03PS1) 10BCornwall: varnish: Split out media functions from upload [puppet] - 10https://gerrit.wikimedia.org/r/1311110 (https://phabricator.wikimedia.org/T427465) [20:40:41] (03CR) 10CI reject: [V:04-1] varnish: Configure text for thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/1306896 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [20:40:49] (03CR) 10CI reject: [V:04-1] varnish: Split out media functions from upload [puppet] - 10https://gerrit.wikimedia.org/r/1311110 (https://phabricator.wikimedia.org/T427465) (owner: 10BCornwall) [20:42:41] (03PS2) 10BCornwall: varnish: Split out media functions from upload [puppet] - 10https://gerrit.wikimedia.org/r/1311110 (https://phabricator.wikimedia.org/T427465) [20:42:41] (03PS7) 10BCornwall: varnish: Configure text for thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/1306896 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [20:42:43] (03CR) 10Ryan Kemper: "Pushed PS2 to correct the dashboard URL" [alerts] - 10https://gerrit.wikimedia.org/r/1311073 (https://phabricator.wikimedia.org/T432249) (owner: 10Bking) [20:43:38] (03CR) 10CI reject: [V:04-1] varnish: Configure text for thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/1306896 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [20:45:43] !log bking@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs1002.eqiad.wmnet with OS bookworm [20:46:22] (03PS3) 10BCornwall: varnish: Split out media functions from upload [puppet] - 10https://gerrit.wikimedia.org/r/1311110 (https://phabricator.wikimedia.org/T427465) [20:46:22] (03PS8) 10BCornwall: varnish: Configure text for thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/1306896 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [20:46:39] RESOLVED: [2x] RejectingBGPPrefixes: Rejecting 1 BGP prefixes - dns2005 (208.80.153.74 - group Anycast4) - https://wikitech.wikimedia.org/wiki/Network_monitoring#RejectingBGPPrefixes - https://alerts.wikimedia.org/?q=alertname%3DRejectingBGPPrefixes [20:47:19] (03CR) 10CI reject: [V:04-1] varnish: Configure text for thumbnails [puppet] - 10https://gerrit.wikimedia.org/r/1306896 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [20:50:02] (03CR) 10Ryan Kemper: [C:03+1] deployment-server: install python commandline parser [puppet] - 10https://gerrit.wikimedia.org/r/1310630 (https://phabricator.wikimedia.org/T431506) (owner: 10Bking) [20:55:10] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/admin 'apply'. [20:55:52] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/admin 'apply'. [20:59:36] !log btullis@cumin1003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad [20:59:40] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet [20:59:41] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet [21:00:38] !log bking@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs2001.codfw.wmnet with OS bookworm [21:01:35] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm [21:01:41] !log bking@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wcqs2001.codfw.wmnet with OS bookworm [21:02:22] FIRING: [9x] CoreRouterInterfaceDown: Core router interface down - cr2-eqiad:et-0/0/1 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [21:02:22] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm [21:04:39] !log btullis@cumin1003 START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-druid-public cluster: Roll restart of jvm daemons. [21:05:02] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet [21:05:03] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet [21:05:09] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet [21:05:44] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet [21:08:33] !log bking@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wcqs1002'] [21:11:02] !log btullis@cumin1003 END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-druid-public cluster: Roll restart of jvm daemons. [21:11:07] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet [21:11:08] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet [21:11:14] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet [21:11:30] (03CR) 10Scott French: "Thank you both for the reviews!" [puppet] - 10https://gerrit.wikimedia.org/r/1311081 (owner: 10Scott French) [21:11:45] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet [21:14:56] (03PS4) 10BCornwall: varnish: Split out media functions from upload [puppet] - 10https://gerrit.wikimedia.org/r/1311110 (https://phabricator.wikimedia.org/T427465) [21:15:44] !log bking@cumin2003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wcqs1002'] [21:16:15] !log bking@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wcqs1002'] [21:17:03] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet [21:17:04] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet [21:17:09] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet [21:17:10] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet [21:17:38] (03PS2) 10Krinkle: Set $wgMathInternalRestbaseURL explicitly [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1310596 (https://phabricator.wikimedia.org/T349582) [21:18:04] !log reprepro include php8.3_8.3.32-1+wmf11u2 into component/php83 for bullseye-wikimedia [21:18:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:20:51] !log bking@cumin2003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wcqs1002'] [21:22:03] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wcqs1002.eqiad.wmnet with OS bookworm [21:23:23] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet [21:23:24] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet [21:23:28] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet [21:24:14] PROBLEM - Check if ntpsec.service has been restarted after /etc/ntpsec/ntp.conf was changed on dns7002 is CRITICAL: CRITICAL: Service ntpsec.service has not been restarted after /etc/ntpsec/ntp.conf was changed (gt 2h). https://wikitech.wikimedia.org/wiki/NTP%23Monitoring [21:28:26] !log bking@cumin2003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wcqs2001.codfw.wmnet with OS bookworm [21:28:53] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm [21:29:26] !log bking@cumin2003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wcqs1002.eqiad.wmnet with OS bookworm [21:29:44] !log bking@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wcqs1002'] [21:29:48] !log bking@cumin2003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wcqs1002'] [21:29:57] !log lerickson@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [21:30:11] !log bking@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wcqs1002'] [21:30:24] !log bking@cumin2003 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['wcqs1002'] [21:30:43] !log bking@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wcqs1002'] [21:31:19] RESOLVED: HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@eqiad in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [21:38:07] !log bking@cumin2003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wcqs1002'] [21:41:19] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [21:42:29] (03PS1) 10Scott French: Revert "changeprop-jobqueue: drop concurrency for problem jobs" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311113 [21:43:28] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wcqs1002.eqiad.wmnet with OS bookworm [21:44:24] (03CR) 10RLazarus: [C:03+1] Revert "changeprop-jobqueue: drop concurrency for problem jobs" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311113 (owner: 10Scott French) [21:45:35] (03CR) 10Scott French: [C:03+2] Revert "changeprop-jobqueue: drop concurrency for problem jobs" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311113 (owner: 10Scott French) [21:47:27] (03CR) 10SBassett: [C:03+1] "One very minor nit, other LGTM." [puppet] - 10https://gerrit.wikimedia.org/r/1311069 (owner: 10CDobbins) [21:47:43] (03Merged) 10jenkins-bot: Revert "changeprop-jobqueue: drop concurrency for problem jobs" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311113 (owner: 10Scott French) [21:48:20] (03CR) 10SBassett: [C:03+1] "\* _otherwise_ LGTM." [puppet] - 10https://gerrit.wikimedia.org/r/1311069 (owner: 10CDobbins) [21:50:06] !log bking@cumin2003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wcqs1002.eqiad.wmnet with OS bookworm [21:50:26] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wcqs1002.eqiad.wmnet with OS trixie [21:50:29] !log swfrench@deploy2003 helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [21:52:14] !log swfrench@deploy2003 helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [21:53:42] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet [21:56:12] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1305986 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [22:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260715T2200) [22:00:32] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet [22:00:33] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet [22:00:41] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet [22:01:15] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet [22:01:29] !log bking@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on datahubsearch[1001-1003].eqiad.wmnet with reason: Using datahubsearch1001 to test bookworm reimages [22:02:55] !log bking@cumin2003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wcqs2001.codfw.wmnet with OS bookworm [22:03:25] (03CR) 10JHathaway: "I think this is ready to merge, unless you folks want me to break it up more?" [puppet] - 10https://gerrit.wikimedia.org/r/1305988 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [22:05:15] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs1002.eqiad.wmnet with reason: host reimage [22:05:19] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host datahubsearch1001.eqiad.wmnet with OS bookworm [22:06:19] RESOLVED: HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [22:08:02] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet [22:08:03] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet [22:08:10] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet [22:08:41] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet [22:09:34] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs1002.eqiad.wmnet with reason: host reimage [22:14:19] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [22:15:00] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet [22:15:01] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet [22:15:06] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet [22:15:29] PROBLEM - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [22:16:06] !log bking@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on datahubsearch[1002-1003].eqiad.wmnet with reason: Using datahubsearch1001 to test bookworm reimages [22:16:24] ^ httpbb_kubernetes_mw-web_hourly looks like a transient 503 on one request, rerunning [22:16:42] (wouldn't normally even bother, just happen to be paying close attention right now because of the jobqueue-related work) [22:17:52] yep, passed on the retry, alert will clear momentarily [22:21:31] thank you :) [22:24:40] (03CR) 10Bking: [C:03+2] deployment-server: install python commandline parser [puppet] - 10https://gerrit.wikimedia.org/r/1310630 (https://phabricator.wikimedia.org/T431506) (owner: 10Bking) [22:25:30] RECOVERY - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [22:34:02] !log bking@cumin2003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host datahubsearch1001.eqiad.wmnet with OS bookworm [22:34:51] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host datahubsearch1001.eqiad.wmnet with OS bookworm [22:44:00] !log bking@cumin2003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wcqs1002.eqiad.wmnet with OS trixie [22:44:46] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wcqs1002.eqiad.wmnet with OS bookworm [22:45:08] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet [22:51:34] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet [22:51:35] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet [22:51:41] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet [22:55:58] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [22:57:30] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host wcqs2001.codfw.wmnet with OS bookworm [23:00:10] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs1002.eqiad.wmnet with reason: host reimage [23:04:01] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs1002.eqiad.wmnet with reason: host reimage [23:15:15] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on wcqs2001.codfw.wmnet with reason: host reimage [23:15:40] !log bking@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host datahubsearch1001.eqiad.wmnet with OS bookworm [23:18:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 19.15% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [23:20:17] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wcqs2001.codfw.wmnet with reason: host reimage [23:21:46] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet [23:23:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.31% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [23:28:50] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs1002.eqiad.wmnet with OS bookworm [23:28:55] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet [23:28:56] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet [23:29:02] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet [23:29:33] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet [23:29:39] (03CR) 10BryanDavis: [C:03+1] "rzl, would you have a couple of minutes to merge this for us? It is Beta Cluster affecting only and actively cherry-picked on the Puppetse" [puppet] - 10https://gerrit.wikimedia.org/r/1305770 (https://phabricator.wikimedia.org/T256168) (owner: 10Ahmon Dancy) [23:32:49] (03CR) 10RLazarus: [C:03+2] "Path is modules/beta so +2 is yes :)" [puppet] - 10https://gerrit.wikimedia.org/r/1305770 (https://phabricator.wikimedia.org/T256168) (owner: 10Ahmon Dancy) [23:36:04] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet [23:36:05] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet [23:36:12] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet [23:36:44] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet [23:42:20] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1311132 [23:42:20] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1311132 (owner: 10TrainBranchBot) [23:43:23] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet [23:43:25] !log btullis@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet [23:43:30] !log btullis@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet [23:44:05] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wcqs2001.codfw.wmnet with OS bookworm [23:50:34] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1311132 (owner: 10TrainBranchBot) [23:57:46] FIRING: Not accepting/receiving prefixes from anycast BGP peer: Alert for device asw1-b4-magru.mgmt.magru.wmnet - Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer