[00:00:25] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1318796 (owner: 10TrainBranchBot) [00:01:00] FIRING: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node ml-serve2004 has a BGP session which is not in the 'established' state. [00:12:10] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 193306640 and 30 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [00:12:40] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:14:10] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 13344 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [00:17:25] (03CR) 10Scott French: "Thanks, Blake!" [software/httpbb] - 10https://gerrit.wikimedia.org/r/1318682 (https://phabricator.wikimedia.org/T428972) (owner: 10Blake) [00:25:26] (03CR) 10Scott French: [C:03+1] "Ah, interesting!" [puppet] - 10https://gerrit.wikimedia.org/r/1318302 (https://phabricator.wikimedia.org/T263872) (owner: 10Ahmon Dancy) [00:31:47] (03CR) 10Scott French: [C:03+1] "Wow, that was quick! Thank very much for moving this into integration/config. This LGTM. Please poke me tomorrow and I'll get this merged." [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1318767 (https://phabricator.wikimedia.org/T432636) (owner: 10Ahmon Dancy) [00:32:09] (03CR) 10Scott French: [V:03+2 C:03+1] "Trivially builds (there is nothing to build)." [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1318767 (https://phabricator.wikimedia.org/T432636) (owner: 10Ahmon Dancy) [00:49:27] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:50:26] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [00:52:57] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-it" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1306988 (https://phabricator.wikimedia.org/T433314) (owner: 10BPirkle) [00:53:25] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-it" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1312694 (https://phabricator.wikimedia.org/T422405) (owner: 10Aaron Schulz) [00:58:12] (03CR) 10Ottomata: [C:03+1] [eventstreams] Bump to v0.20.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318784 (https://phabricator.wikimedia.org/T432824) (owner: 10TChin) [01:12:28] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1318807 [01:12:28] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1318807 (owner: 10TrainBranchBot) [01:27:45] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1318807 (owner: 10TrainBranchBot) [01:48:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:08:58] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:12:25] RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:13:58] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:14:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:14:27] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [02:14:37] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:47:11] (03CR) 10Ryan Kemper: [C:03+2] pontoon: add rkemper-kafka stack [puppet] - 10https://gerrit.wikimedia.org/r/1318761 (https://phabricator.wikimedia.org/T276088) (owner: 10Ryan Kemper) [03:55:05] FIRING: KubernetesCalicoDown: ml-serve2004.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s-mlserve&var-instance=ml-serve2004.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [03:58:58] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5) {#122601}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [03:59:04] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [04:01:15] FIRING: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node ml-serve2004 has a BGP session which is not in the 'established' state. [04:30:07] (03PS1) 10Samwilson: Disallow all audio formats in the format filter [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1318810 (https://phabricator.wikimedia.org/T431671) [04:43:58] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5) {#122601}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [04:50:26] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [05:06:39] FIRING: [3x] CoreBGPDown: Core BGP session down between cr2-eqord and cr3-ulsfo (198.35.26.128) - group Confed_ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [05:08:58] FIRING: [7x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-3/1/2 (Transport: Hurricane Electric (dc4841.dal5) {#122601}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [05:25:21] (03PS1) 10Ryan Kemper: Transferer: log the reason a copy failed [software/transferpy] - 10https://gerrit.wikimedia.org/r/1318972 (https://phabricator.wikimedia.org/T430880) [05:36:30] (03CR) 10Ryan Kemper: "Small correction because it's bothering me — my commit message said it unwinds the downtime, but it actually (correctly) leaves the downti" [cookbooks] - 10https://gerrit.wikimedia.org/r/1317127 (https://phabricator.wikimedia.org/T430880) (owner: 10Ryan Kemper) [05:46:53] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1164.eqiad.wmnet with reason: Reimage [05:47:40] !log marostegui@cumin1003 START - Cookbook sre.hosts.reimage for host db1164.eqiad.wmnet with OS trixie [05:48:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:00:04] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T0600) [06:01:23] !log marostegui@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on db1164.eqiad.wmnet with reason: host reimage [06:03:55] (03PS1) 10Marostegui: check_private_data_report: Add clouddb1033 [puppet] - 10https://gerrit.wikimedia.org/r/1318973 (https://phabricator.wikimedia.org/T409557) [06:04:57] (03CR) 10Marostegui: [C:03+2] check_private_data_report: Add clouddb1033 [puppet] - 10https://gerrit.wikimedia.org/r/1318973 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [06:05:11] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1164.eqiad.wmnet with reason: host reimage [06:07:55] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309675 (https://phabricator.wikimedia.org/T431830) (owner: 10Dreamrimmer) [06:14:25] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:14:27] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [06:14:37] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:26:57] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1164.eqiad.wmnet with OS trixie [06:31:29] (03PS1) 10Slyngshede: data.yaml sync WMF group members with LDAP [puppet] - 10https://gerrit.wikimedia.org/r/1318975 [06:39:43] (03CR) 10Slyngshede: [C:03+1] data.yaml sync WMF group members with LDAP [puppet] - 10https://gerrit.wikimedia.org/r/1318975 (owner: 10Slyngshede) [06:39:45] (03CR) 10Slyngshede: [C:03+2] data.yaml sync WMF group members with LDAP [puppet] - 10https://gerrit.wikimedia.org/r/1318975 (owner: 10Slyngshede) [06:44:13] 10SRE-swift-storage, 06Commons, 06Data-Persistence, 10MediaWiki-File-management, 07Wikimedia-production-error: DBTransactionSizeError when trying to move file - https://phabricator.wikimedia.org/T431616#12165936 (10Marostegui) [06:55:19] (03CR) 10Marostegui: "The thing is, that the new master is very unlikely to have a pt-heartbeat running at that time as it will be a replica of the primary mast" [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [06:57:30] (03PS1) 10Ryan Kemper: kafka: converge topic config from hieradata [puppet] - 10https://gerrit.wikimedia.org/r/1318976 (https://phabricator.wikimedia.org/T276088) [06:57:51] (03PS1) 10Marostegui: db2232: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1318977 (https://phabricator.wikimedia.org/T433463) [06:58:10] (03CR) 10Ryan Kemper: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1318976 (https://phabricator.wikimedia.org/T276088) (owner: 10Ryan Kemper) [06:58:14] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db[2160,2232].codfw.wmnet with reason: Reimage [06:58:44] (03CR) 10Marostegui: [C:03+2] db2232: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1318977 (https://phabricator.wikimedia.org/T433463) (owner: 10Marostegui) [06:58:58] !log marostegui@cumin1003 START - Cookbook sre.hosts.reimage for host db2232.codfw.wmnet with OS trixie [07:00:04] Amir1, urbanecm, and awight: How many deployers does it take to do UTC morning backport window deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T0700). [07:00:04] DreamRimmer: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:15:40] !log marostegui@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on db2232.codfw.wmnet with reason: host reimage [07:19:30] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2232.codfw.wmnet with reason: host reimage [07:23:54] (03CR) 10Brouberol: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1311441 (https://phabricator.wikimedia.org/T324335) (owner: 10Btullis) [07:31:08] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [07:33:03] (03PS1) 10Marostegui: Revert "db2232: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1318984 [07:33:58] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [07:34:09] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [07:34:26] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [07:36:13] I'm going to reload bird on ganeti2034, only netops VMs are running there, it might just make a bit of noise [07:37:51] (done) [07:42:12] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2232.codfw.wmnet with OS trixie [07:42:53] !log ayounsi@cumin1003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet [07:43:03] (03CR) 10Marostegui: [C:03+2] Revert "db2232: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1318984 (owner: 10Marostegui) [07:46:35] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet [07:46:45] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1163.eqiad.wmnet with reason: Maintenance [07:46:53] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1163 (T431660)', diff saved to https://phabricator.wikimedia.org/P95441 and previous config saved to /var/cache/conftool/dbconfig/20260729-074652-cwilliams.json [07:48:03] (03PS1) 10Btullis: hadoop-test: Switch to using the new namenodes [puppet] - 10https://gerrit.wikimedia.org/r/1318988 (https://phabricator.wikimedia.org/T406746) [07:48:28] (03PS1) 10Marostegui: eqiad.yaml: Add clouddb1033 [puppet] - 10https://gerrit.wikimedia.org/r/1318989 (https://phabricator.wikimedia.org/T409557) [07:48:49] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1156.eqiad.wmnet with reason: Maintenance [07:48:49] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [07:48:59] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance [07:49:06] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance [07:49:07] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1156 (T431660)', diff saved to https://phabricator.wikimedia.org/P95442 and previous config saved to /var/cache/conftool/dbconfig/20260729-074906-cwilliams.json [07:49:14] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1157 (T431660)', diff saved to https://phabricator.wikimedia.org/P95443 and previous config saved to /var/cache/conftool/dbconfig/20260729-074914-cwilliams.json [07:49:23] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance [07:49:31] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1190 (T431660)', diff saved to https://phabricator.wikimedia.org/P95444 and previous config saved to /var/cache/conftool/dbconfig/20260729-074930-cwilliams.json [07:50:10] FIRING: [4x] GanetiBGPNoInPrefixes: ganeti2033 is not sending any prefix to lsw1-b7-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [07:50:18] RECOVERY - Host ml-serve2004 is UP: PING OK - Packet loss = 0%, RTA = 34.46 ms [07:53:22] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1163: Maintenance [07:54:50] RESOLVED: KubernetesCalicoDown: ml-serve2004.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s-mlserve&var-instance=ml-serve2004.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:55:34] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1157: Maintenance [07:55:38] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1156: Maintenance [07:56:00] RESOLVED: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node ml-serve2004 has a BGP session which is not in the 'established' state. [07:57:11] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1190: Maintenance [07:58:31] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-be2072.codfw.wmnet with OS trixie [07:58:46] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12166088 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-be2072.codfw.wmnet with OS trixie [07:58:49] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [08:02:16] (03PS1) 10Bartosz Wójtowicz: ml-services: Update gpt-oss-safeguard-20b docker image. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319026 (https://phabricator.wikimedia.org/T432933) [08:03:03] (03PS3) 10Trueg: WDQSv2: Scheduling based on resource requests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318733 (https://phabricator.wikimedia.org/T431395) [08:04:13] (03PS30) 10Federico Ceratto: mysql: update replication source [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) [08:04:50] 10ops-codfw, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team: hw troubleshooting: Memory errors for ml-serve2002.codfw.wmnet - https://phabricator.wikimedia.org/T433476 (10klausman) 03NEW [08:05:09] (03CR) 10Federico Ceratto: "ok, I added the CLI option and updated the tests" [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [08:06:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [08:07:18] (03PS1) 10Btullis: hadoop-test: Replace an-test-master namenodes with refreshed hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319027 (https://phabricator.wikimedia.org/T406746) [08:08:24] 10ops-codfw, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team: hw troubleshooting: memory errors on ml-serve2004.codfw.wmnet - https://phabricator.wikimedia.org/T433478 (10klausman) 03NEW [08:08:49] 06SRE, 10SRE-swift-storage, 05Goal, 06Machine-Learning-Team (Q1 FY2026-27), 07OKR-Work: Provision and wire object storage for TTS v1 audio artifacts: S3 sink, Swift bucket, and credentials - https://phabricator.wikimedia.org/T432944#12166214 (10MatthewVernon) [08:10:28] 06SRE, 10Infrastructure Security, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), and 3 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12166229 (10klausman) As is tradition during reboots, I also found two machines with... [08:11:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [08:12:07] (03CR) 10Brouberol: [C:03+1] hadoop-test: Replace an-test-master namenodes with refreshed hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319027 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [08:12:34] (03CR) 10Brouberol: [C:03+1] hadoop-test: Switch to using the new namenodes [puppet] - 10https://gerrit.wikimedia.org/r/1318988 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [08:12:53] (03CR) 10Marostegui: mysql: update replication source (033 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [08:13:49] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [08:14:14] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: eqiad row A&B host migration details request for Data Persistence - https://phabricator.wikimedia.org/T432644#12166236 (10MatthewVernon) I've done this for thanos and ms-{fe,be} hosts. Can I check that this is just physical moving, no renumbering? [08:19:40] !log mvernon@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be2072.codfw.wmnet with reason: host reimage [08:20:20] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 1098 hosts [08:23:46] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2072.codfw.wmnet with reason: host reimage [08:23:49] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [08:24:23] (03PS4) 10Trueg: WDQSv2: Scheduling based on resource requests [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318733 (https://phabricator.wikimedia.org/T431395) [08:30:28] !log btullis@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 11 hosts with reason: Replacing the namenodes [08:30:29] (03CR) 10Filippo Giunchedi: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1318228 (https://phabricator.wikimedia.org/T401818) (owner: 10Majavah) [08:34:37] (03CR) 10Marostegui: [C:03+2] eqiad.yaml: Add clouddb1033 [puppet] - 10https://gerrit.wikimedia.org/r/1318989 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [08:37:24] (03CR) 10Kevin Bazira: [C:03+1] ml-services: Update gpt-oss-safeguard-20b docker image. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319026 (https://phabricator.wikimedia.org/T432933) (owner: 10Bartosz Wójtowicz) [08:38:34] (03PS1) 10JMeybohm: Update maintainer fields of core components to the k8s SIG [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1319032 (https://phabricator.wikimedia.org/T373526) [08:41:20] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1163: Maintenance [08:41:40] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance [08:41:48] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1169 (T431660)', diff saved to https://phabricator.wikimedia.org/P95458 and previous config saved to /var/cache/conftool/dbconfig/20260729-084147-cwilliams.json [08:43:32] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1157: Maintenance [08:43:52] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance [08:44:01] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1166 (T431660)', diff saved to https://phabricator.wikimedia.org/P95460 and previous config saved to /var/cache/conftool/dbconfig/20260729-084400-cwilliams.json [08:44:08] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1156: Maintenance [08:44:28] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance [08:44:37] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1182 (T431660)', diff saved to https://phabricator.wikimedia.org/P95462 and previous config saved to /var/cache/conftool/dbconfig/20260729-084436-cwilliams.json [08:45:08] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1190: Maintenance [08:45:27] !log ayounsi@cumin1003 START - Cookbook sre.dns.admin DNS admin: depool magru [reason: router upgrade, T431750] [08:45:27] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance [08:45:29] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: router upgrade, T431750] [08:45:31] T431750: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750 [08:45:35] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1199 (T431660)', diff saved to https://phabricator.wikimedia.org/P95464 and previous config saved to /var/cache/conftool/dbconfig/20260729-084534-cwilliams.json [08:45:41] !log mvernon@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ms-be2072.codfw.wmnet with OS trixie [08:45:48] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12166327 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2003 for host ms-be2072.codfw.wmnet with OS trixie completed: - ms-be2072 (**PASS*... [08:46:12] !log ayounsi@cumin1003 DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1:00:00 on cr1-magru,cr1-magru IPv6,cr1-magru.mgmt with reason: router upgrade [08:47:19] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1169: Maintenance [08:48:26] (03PS1) 10Ayounsi: Add magru core routers mgmt interface monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1319034 [08:50:19] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1166: Maintenance [08:50:52] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1182: Maintenance [08:51:44] (03CR) 10Cathal Mooney: [C:03+1] Add magru core routers mgmt interface monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1319034 (owner: 10Ayounsi) [08:52:28] (03PS1) 10Ayounsi: cr2-magru: enable BGP RIB sharding [homer/public] - 10https://gerrit.wikimedia.org/r/1319035 (https://phabricator.wikimedia.org/T320264) [08:52:43] (03CR) 10Btullis: [C:03+2] hadoop-test: Switch to using the new namenodes [puppet] - 10https://gerrit.wikimedia.org/r/1318988 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [08:52:48] (03CR) 10Ayounsi: [C:03+2] Add magru core routers mgmt interface monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1319034 (owner: 10Ayounsi) [08:53:10] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1199: Maintenance [08:53:28] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: Update gpt-oss-safeguard-20b docker image. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319026 (https://phabricator.wikimedia.org/T432933) (owner: 10Bartosz Wójtowicz) [08:53:49] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [08:53:57] (03CR) 10Cathal Mooney: [C:03+2] cr2-magru: enable BGP RIB sharding [homer/public] - 10https://gerrit.wikimedia.org/r/1319035 (https://phabricator.wikimedia.org/T320264) (owner: 10Ayounsi) [08:55:38] (03Merged) 10jenkins-bot: cr2-magru: enable BGP RIB sharding [homer/public] - 10https://gerrit.wikimedia.org/r/1319035 (https://phabricator.wikimedia.org/T320264) (owner: 10Ayounsi) [08:55:45] (03Merged) 10jenkins-bot: ml-services: Update gpt-oss-safeguard-20b docker image. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319026 (https://phabricator.wikimedia.org/T432933) (owner: 10Bartosz Wójtowicz) [08:56:31] (03CR) 10Btullis: [C:03+2] hadoop-test: Replace an-test-master namenodes with refreshed hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319027 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [08:58:39] (03PS1) 10Hnowlan: debian: update dependencies to add systemd [debs/pint] - 10https://gerrit.wikimedia.org/r/1319039 [08:58:52] (03Merged) 10jenkins-bot: hadoop-test: Replace an-test-master namenodes with refreshed hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319027 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [09:00:01] !log Dropping renamed tables T426341 [09:00:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:00:05] T426341: Drop DiscussionTools database tables from wikis which don't need them - https://phabricator.wikimedia.org/T426341 [09:00:14] !log ayounsi@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr1-magru,cr1-magru IPv6,cr1-magru.mgmt with reason: router upgrade [09:00:21] 06SRE, 06Infrastructure-Foundations, 10netops: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750#12166353 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a746a25c-706b-4c61-bdf4-214a1c78ff90) set by ayounsi@cumin1003 for 1:00:00 on 3 host(s) and their servi... [09:00:47] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host idp2005.wikimedia.org [09:02:49] (03PS4) 10Blake: mw-pretrain: Add a canary values.yaml. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318671 (https://phabricator.wikimedia.org/T427668) [09:03:49] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [09:03:55] (03CR) 10Blake: mw-pretrain: Add a canary values.yaml. (033 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318671 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [09:04:47] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp2005.wikimedia.org [09:07:17] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host idp1005.wikimedia.org [09:11:14] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp1005.wikimedia.org [09:11:40] (03PS1) 10JMeybohm: Update maintainer fields of core components to the k8s SIG [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319041 (https://phabricator.wikimedia.org/T373526) [09:11:46] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host idm1001.wikimedia.org [09:11:57] !log ayounsi@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-magru,cr2-magru IPv6,cr2-magru.mgmt with reason: router upgrade [09:12:04] 06SRE, 06Infrastructure-Foundations, 10netops: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750#12166372 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=cce20762-fa19-4ada-b944-1a93ea8ed59f) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their servi... [09:13:09] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [09:13:59] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [09:14:51] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply [09:15:38] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply [09:15:45] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm1001.wikimedia.org [09:17:17] !log drain cr1-magru - T431750 [09:17:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:17:21] T431750: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750 [09:18:49] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [09:21:06] !log reboot cr1-magru - T431750 [09:21:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:21:44] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [09:24:02] PROBLEM - OSPF status on cr2-eqiad is CRITICAL: OSPFv2: 10/11 UP : OSPFv3: 10/11 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:24:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b4-magru:et-0/0/48 (Core: cr1-magru:et-0/0/2 {#70128}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b4-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [09:26:39] FIRING: [2x] CoreBGPDown: Core BGP session down between asw1-b4-magru and cr1-magru (195.200.68.148) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=asw1-b4-magru:9804&var-bgp_group=core&var-bgp_neighbor=cr1-magru - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [09:28:17] (03CR) 10Klausman: [C:03+1] Add tts-section-generator CNAMEs to k8s-ingress-ml-serve [dns] - 10https://gerrit.wikimedia.org/r/1318685 (https://phabricator.wikimedia.org/T430536) (owner: 10Kevin Bazira) [09:28:37] (03CR) 10Elukey: [C:03+1] Update maintainer fields of core components to the k8s SIG [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1319032 (https://phabricator.wikimedia.org/T373526) (owner: 10JMeybohm) [09:29:16] (03CR) 10Elukey: [C:03+1] Update maintainer fields of core components to the k8s SIG [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319041 (https://phabricator.wikimedia.org/T373526) (owner: 10JMeybohm) [09:29:51] FIRING: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1 {#70094}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [09:30:10] FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:30:26] (03PS1) 10Bartosz Wójtowicz: ml-services: Request 1 GPU for gpt-oss-safeguard-20b. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319043 (https://phabricator.wikimedia.org/T432933) [09:30:55] (03PS1) 10Marostegui: eqiad.yaml: Fix hostname [puppet] - 10https://gerrit.wikimedia.org/r/1319044 (https://phabricator.wikimedia.org/T409557) [09:31:58] (03CR) 10JMeybohm: [C:03+2] Update maintainer fields of core components to the k8s SIG [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319041 (https://phabricator.wikimedia.org/T373526) (owner: 10JMeybohm) [09:32:02] RECOVERY - OSPF status on cr2-eqiad is OK: OSPFv2: 11/11 UP : OSPFv3: 11/11 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:32:06] (03CR) 10Marostegui: [C:03+2] eqiad.yaml: Fix hostname [puppet] - 10https://gerrit.wikimedia.org/r/1319044 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [09:32:13] (03CR) 10JMeybohm: [V:03+2 C:03+2] Update maintainer fields of core components to the k8s SIG [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1319032 (https://phabricator.wikimedia.org/T373526) (owner: 10JMeybohm) [09:33:05] all done with cr1-magru, moving on to cr2 [09:33:17] !log marostegui@cumin1003 conftool action : set/weight=100; selector: name=clouddb1033.eqiad.wmnet,service=s5 [09:33:21] !log marostegui@cumin1003 conftool action : set/weight=100; selector: name=clouddb1033.eqiad.wmnet,service=s8 [09:33:27] !log marostegui@cumin1003 conftool action : set/pooled=yes; selector: name=clouddb1033.eqiad.wmnet,service=s5 [09:33:32] !log marostegui@cumin1003 conftool action : set/pooled=yes; selector: name=clouddb1033.eqiad.wmnet,service=s8 [09:33:49] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [09:34:12] (03PS1) 10Marostegui: clouddb1033: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1319045 (https://phabricator.wikimedia.org/T409557) [09:34:26] (03PS1) 10Btullis: Move analytics-test-hive and analytics-test-presto to new host [dns] - 10https://gerrit.wikimedia.org/r/1319046 (https://phabricator.wikimedia.org/T406746) [09:34:51] RESOLVED: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/48 (Core: cr1-magru:et-0/0/1 {#70094}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [09:34:57] (03CR) 10Marostegui: [C:03+2] clouddb1033: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1319045 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [09:35:10] RESOLVED: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:35:16] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1169: Maintenance [09:35:17] (03CR) 10JMeybohm: "Right. And you also have to configure CI to use the new image after it has been build (https://gerrit.wikimedia.org/r/plugins/gitiles/inte" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318569 (https://phabricator.wikimedia.org/T388390) (owner: 10Jelto) [09:36:39] RESOLVED: [2x] CoreBGPDown: Core BGP session down between asw1-b4-magru and cr1-magru (195.200.68.148) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=magru&var-device=asw1-b4-magru:9804&var-bgp_group=core&var-bgp_neighbor=cr1-magru - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [09:36:39] (03CR) 10Brouberol: [C:03+1] Move analytics-test-hive and analytics-test-presto to new host [dns] - 10https://gerrit.wikimedia.org/r/1319046 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [09:37:50] (03CR) 10Hnowlan: "Mostly seems good, but a few pitfalls/concerns, and a few nits which can be ignored if needs be." [puppet] - 10https://gerrit.wikimedia.org/r/1306792 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [09:38:15] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1166: Maintenance [09:38:35] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance [09:38:43] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1175 (T431660)', diff saved to https://phabricator.wikimedia.org/P95479 and previous config saved to /var/cache/conftool/dbconfig/20260729-093842-cwilliams.json [09:38:49] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1182: Maintenance [09:39:10] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance [09:39:17] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1188 (T431660)', diff saved to https://phabricator.wikimedia.org/P95481 and previous config saved to /var/cache/conftool/dbconfig/20260729-093917-cwilliams.json [09:41:08] (03Merged) 10jenkins-bot: Update maintainer fields of core components to the k8s SIG [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319041 (https://phabricator.wikimedia.org/T373526) (owner: 10JMeybohm) [09:41:09] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1199: Maintenance [09:41:26] !log elukey@cumin1003 START - Cookbook sre.hosts.reboot-single for host zookeeper-test1002.eqiad.wmnet [09:41:29] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1221.eqiad.wmnet with reason: Maintenance [09:41:52] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance [09:42:00] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1221 (T431660)', diff saved to https://phabricator.wikimedia.org/P95483 and previous config saved to /var/cache/conftool/dbconfig/20260729-094200-cwilliams.json [09:42:28] (03CR) 10Btullis: [C:03+2] Move analytics-test-hive and analytics-test-presto to new host [dns] - 10https://gerrit.wikimedia.org/r/1319046 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [09:42:46] !log btullis@dns1004 START - running authdns-update [09:44:54] !log btullis@dns1004 END - running authdns-update [09:45:06] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1175: Maintenance [09:45:06] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1188: Maintenance [09:45:24] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zookeeper-test1002.eqiad.wmnet [09:46:21] (03PS1) 10CWilliams: profile::mariadb::replication_lag: remove the check from monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1319029 (https://phabricator.wikimedia.org/T433344) [09:46:52] (03CR) 10Marostegui: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319029 (https://phabricator.wikimedia.org/T433344) (owner: 10CWilliams) [09:48:21] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1221: Maintenance [09:48:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:50:19] 06SRE, 10Bitu, 06Infrastructure-Foundations: Move tracking of LDAP access to a new repository and manage it from Bitu - https://phabricator.wikimedia.org/T431111#12166505 (10LSobanski) p:05Triage→03High [09:50:36] 06SRE, 06Infrastructure-Foundations: Make airflow-wmde-ops managed in Bitu - https://phabricator.wikimedia.org/T431077#12166508 (10SLyngshede-WMF) p:05Triage→03Medium a:03SLyngshede-WMF [09:51:12] (03PS1) 10Btullis: hadoop-test: Update the database server used by hive and druid [puppet] - 10https://gerrit.wikimedia.org/r/1319049 (https://phabricator.wikimedia.org/T406746) [09:51:14] (03PS1) 10Bartosz Wójtowicz: ml-services: Remove ml-serve1012 affinity from smaller models. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319050 [09:52:08] 06SRE, 06Infrastructure-Foundations, 10WMDE Analytics: Make airflow-wmde-ops managed in Bitu - https://phabricator.wikimedia.org/T431077#12166512 (10AndrewTavis_WMDE) [09:52:39] !log drain cr2-magru - T431750 [09:52:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:52:43] T431750: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750 [09:53:31] !log reboot cr2-magru - T431750 [09:53:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:54:16] (03CR) 10Brouberol: [C:03+1] hadoop-test: Update the database server used by hive and druid [puppet] - 10https://gerrit.wikimedia.org/r/1319049 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [09:56:06] PROBLEM - OSPF status on cr2-eqdfw is CRITICAL: OSPFv2: 6/7 UP : OSPFv3: 6/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:56:23] (03CR) 10Btullis: [C:03+2] hadoop-test: Update the database server used by hive and druid [puppet] - 10https://gerrit.wikimedia.org/r/1319049 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [09:56:39] (03CR) 10Hnowlan: opensearch: add security-plugin specific configuration to opensearch.yml (035 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1318792 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [09:56:39] (03PS2) 10Blake: httpbb: Add a --request-header argument. [software/httpbb] - 10https://gerrit.wikimedia.org/r/1318682 (https://phabricator.wikimedia.org/T428972) [09:56:51] FIRING: SwitchCoreInterfaceDown: Switch core interface down - asw1-b4-magru:et-0/0/50 (Core: cr2-magru:et-0/0/2 {#70150}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=asw1-b4-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [09:57:10] FIRING: [2x] BFDdown: BFD session down between cr2-eqdfw and 195.200.68.153 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqdfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:57:16] (03CR) 10Kevin Bazira: "Thanks for the patch. The gpt-oss-safeguard-20b isvc requires 2 GPUs to meet its latency targets. We did extensive performance tuning on t" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319043 (https://phabricator.wikimedia.org/T432933) (owner: 10Bartosz Wójtowicz) [09:59:51] (03PS2) 10Lucas Werkmeister (WMDE): wikidata-query-gui: update staging custom-config.json [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318731 [10:00:04] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T1000) [10:00:18] PROBLEM - OSPF status on cr1-magru is CRITICAL: OSPFv2: 1/3 UP : OSPFv3: 1/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:00:32] (03PS1) 10Klausman: base: bump BPO Trixie kernel to 6.19 (only used by big ML GPU hosts atm) [puppet] - 10https://gerrit.wikimedia.org/r/1319051 (https://phabricator.wikimedia.org/T433485) [10:01:39] FIRING: [6x] CoreBGPDown: Core BGP session down between asw1-b4-magru and cr2-magru (195.200.68.144) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [10:01:51] FIRING: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/50 (Core: cr2-magru:et-0/0/1 {#70130}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [10:02:10] FIRING: [4x] BFDdown: BFD session down between cr1-magru and 195.200.68.129 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:03:15] (03CR) 10Lucas Werkmeister (WMDE): "Thanks, that seems to work better. (I didn’t know which order “staging” and “wikidata-query-gui” should be in the file name, so thanks for" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318731 (owner: 10Lucas Werkmeister (WMDE)) [10:03:38] (03CR) 10Bartosz Wójtowicz: "Acknowledged" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319043 (https://phabricator.wikimedia.org/T432933) (owner: 10Bartosz Wójtowicz) [10:03:58] (03Abandoned) 10Bartosz Wójtowicz: ml-services: Request 1 GPU for gpt-oss-safeguard-20b. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319043 (https://phabricator.wikimedia.org/T432933) (owner: 10Bartosz Wójtowicz) [10:05:06] RECOVERY - OSPF status on cr2-eqdfw is OK: OSPFv2: 7/7 UP : OSPFv3: 7/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:05:18] RECOVERY - OSPF status on cr1-magru is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:06:39] RESOLVED: [6x] CoreBGPDown: Core BGP session down between asw1-b4-magru and cr2-magru (195.200.68.144) - group core - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [10:06:51] FIRING: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/50 (Core: cr2-magru:et-0/0/1 {#70130}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [10:06:53] (03PS1) 10Filippo Giunchedi: hieradata: enable dumps-nfs.w.o usage [puppet] - 10https://gerrit.wikimedia.org/r/1319053 (https://phabricator.wikimedia.org/T432587) [10:06:58] FIRING: [2x] ProbeDown: Service text-https:443 has failed probes (http_text-https_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:07:06] RESOLVED: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-b3-magru:et-0/0/50 (Core: cr2-magru:et-0/0/1 {#70130}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [10:07:07] !ack [10:07:08] 8229 (ACKED) [2x] ProbeDown sre (text-https:443 probes/service magru) [10:07:10] RESOLVED: [4x] BFDdown: BFD session down between cr1-magru and 195.200.68.129 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:07:36] i imagine that's the drain? [10:08:02] (03CR) 10JMeybohm: [C:03+1] golang: add trixie-based golang-1.25 image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1313867 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [10:08:06] XioNoX: ^ [10:09:19] https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes%2Fservice&var-module=$__all&orgId=1&from=now-6h&to=now&timezone=utc&var-site=$__all&var-Filters=&viewPanel=panel-2 [10:09:22] it seems to be magru yes [10:09:35] weird, I'd say transient monitoring packets got lots when cr2-magru came back up [10:09:59] should recover soon on its own [10:10:08] looks like it has, if i'm reading this right [10:10:18] (03PS31) 10Federico Ceratto: mysql: update replication source [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) [10:10:35] (03CR) 10Ladsgroup: [C:03+2] Disallow all audio formats in the format filter [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1318810 (https://phabricator.wikimedia.org/T431671) (owner: 10Samwilson) [10:11:06] (03PS32) 10Federico Ceratto: mysql: update replication source [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) [10:11:40] (03CR) 10Federico Ceratto: "Updated the names to avoid using "source" and "destination" anywhere" [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [10:11:58] RESOLVED: [2x] ProbeDown: Service text-https:443 has failed probes (http_text-https_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:12:03] good [10:12:23] (03CR) 10Marostegui: "@cwilliams@wikimedia.org can you give this another review?" [cookbooks] - 10https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: 10Federico Ceratto) [10:12:52] 06SRE, 10SRE-swift-storage, 05Goal, 06Machine-Learning-Team (Q1 FY2026-27), 07OKR-Work: Provision and wire object storage for TTS v1 audio artifacts: S3 sink, Swift bucket, and credentials - https://phabricator.wikimedia.org/T432944#12166592 (10Ladsgroup) >>! In T432944#12163528, @MatthewVernon wrote: >... [10:13:50] Hi folks, would anyone be around in about an hour to help me with a database deployment to resolve https://phabricator.wikimedia.org/T433429 ? [10:14:02] !log ayounsi@cumin1003 START - Cookbook sre.dns.admin DNS admin: pool magru [reason: router upgrade, T431750] [10:14:04] !log ayounsi@cumin1003 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: router upgrade, T431750] [10:14:06] T431750: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750 [10:14:28] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:14:28] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:14:31] HouseOfM: you'd need someone with deployment grants [10:14:40] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:14:42] 06SRE, 06Infrastructure-Foundations, 10netops: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750#12166601 (10ayounsi) 05Open→03Resolved a:03ayounsi Done. [10:14:59] (03CR) 10David Caro: [C:03+1] "LGTM, emails were sent and such too, will just keep an eye on incoming tasks/irc if anyone did not read them." [puppet] - 10https://gerrit.wikimedia.org/r/1319053 (https://phabricator.wikimedia.org/T432587) (owner: 10Filippo Giunchedi) [10:15:19] @marostegui unfortunately I'm the only one in the team around right now, hence me begging in here for help :) [10:15:35] HouseOfM: :( [10:18:58] RESOLVED: CertAlmostExpired: gNMI TLS certificate for lsw1-f6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:19:42] (03CR) 10Aklapper: "https://phabricator.wikimedia.org/T374926#12166520 implies that we could merge this. Should afterwards check what broke and whether the Di" [puppet] - 10https://gerrit.wikimedia.org/r/1298763 (https://phabricator.wikimedia.org/T405596) (owner: 10Aklapper) [10:21:04] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance [10:21:12] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1186 (T431660)', diff saved to https://phabricator.wikimedia.org/P95493 and previous config saved to /var/cache/conftool/dbconfig/20260729-102111-cwilliams.json [10:21:38] (03Merged) 10jenkins-bot: Disallow all audio formats in the format filter [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1318810 (https://phabricator.wikimedia.org/T431671) (owner: 10Samwilson) [10:27:30] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1186: Maintenance [10:31:18] (03CR) 10Jelto: "good point! I uploaded Id363603cc995cca2cc12ce66f58c471a39deb07a for the helm-linter version bump in the jjb CI definition." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318569 (https://phabricator.wikimedia.org/T388390) (owner: 10Jelto) [10:31:50] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1188: Maintenance [10:32:03] 06SRE, 06Infrastructure-Foundations, 10WMDE Analytics: Make airflow-wmde-ops managed in Bitu - https://phabricator.wikimedia.org/T431077#12166624 (10AndrewTavis_WMDE) [10:32:10] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance [10:32:18] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1197 (T431660)', diff saved to https://phabricator.wikimedia.org/P95496 and previous config saved to /var/cache/conftool/dbconfig/20260729-103217-cwilliams.json [10:33:03] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1175: Maintenance [10:33:23] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance [10:33:32] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1198 (T431660)', diff saved to https://phabricator.wikimedia.org/P95499 and previous config saved to /var/cache/conftool/dbconfig/20260729-103330-cwilliams.json [10:33:37] (03PS1) 10Btullis: hadoop-test: Put the refreshed nodes into insetup mode prior to decom [puppet] - 10https://gerrit.wikimedia.org/r/1319059 (https://phabricator.wikimedia.org/T406746) [10:35:04] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1221: Maintenance [10:35:25] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1238.eqiad.wmnet with reason: Maintenance [10:35:33] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1238 (T431660)', diff saved to https://phabricator.wikimedia.org/P95501 and previous config saved to /var/cache/conftool/dbconfig/20260729-103532-cwilliams.json [10:36:54] (03CR) 10Brouberol: "Maybe gather an-test-master 1001 and 1002 into the same `node` declaration? Same for 1003 and 1004." [puppet] - 10https://gerrit.wikimedia.org/r/1319059 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [10:38:21] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1197: Maintenance [10:39:23] !log ran https://phabricator.wikimedia.org/T432509#12149723 in production (T432509) [10:39:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:39:27] T432509: transclude a project namespace page in main space page will view old revision - https://phabricator.wikimedia.org/T432509 [10:39:52] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1198: Maintenance [10:41:46] (03PS2) 10Jelto: golang: add trixie-based golang-1.25 image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1313867 (https://phabricator.wikimedia.org/T427402) [10:41:57] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1238: Maintenance [10:44:23] (03PS1) 10Ozge: ml-services: Add jina-embeddings production isvc to llm namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319061 (https://phabricator.wikimedia.org/T432717) [10:44:45] (03CR) 10Ozge: "For production deployment." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319061 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:47:14] PROBLEM - Host ml-serve1015 is DOWN: PING CRITICAL - Packet loss = 100% [10:47:49] (03PS2) 10Btullis: hadoop-test: Put the refreshed nodes into insetup mode prior to decom [puppet] - 10https://gerrit.wikimedia.org/r/1319059 (https://phabricator.wikimedia.org/T406746) [10:48:48] (03PS1) 10Marostegui: Revert "db1197: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1319064 [10:49:35] (03CR) 10Marostegui: [C:03+2] Revert "db1197: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1319064 (owner: 10Marostegui) [10:49:42] RECOVERY - Host ml-serve1015 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [10:51:04] (03CR) 10Btullis: "Good call, thanks." [puppet] - 10https://gerrit.wikimedia.org/r/1319059 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [10:52:30] (03CR) 10Kevin Bazira: [C:03+1] ml-services: Add jina-embeddings production isvc to llm namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319061 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:54:33] !log Dropping renamed tables T425066 [10:54:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:54:38] T425066: Drop AbuseFilter tables from closed wikis - https://phabricator.wikimedia.org/T425066 [10:57:16] (03CR) 10Elukey: [C:03+1] base: bump BPO Trixie kernel to 6.19 (only used by big ML GPU hosts atm) [puppet] - 10https://gerrit.wikimedia.org/r/1319051 (https://phabricator.wikimedia.org/T433485) (owner: 10Klausman) [10:57:48] (03CR) 10Ozge: [V:03+2 C:03+2] ml-services: Add jina-embeddings production isvc to llm namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319061 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [10:59:55] (03PS2) 10Bartosz Wójtowicz: ml-services: Remove ml-serve1012 affinity from smaller models. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319050 [11:00:02] (03Merged) 10jenkins-bot: ml-services: Add jina-embeddings production isvc to llm namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319061 (https://phabricator.wikimedia.org/T432717) (owner: 10Ozge) [11:00:05] mvolz: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for Services – Citoid / Zotero. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T1100). [11:00:52] 06SRE, 06Infrastructure-Foundations, 10netops: QFX5120 reporting 0 for neighbor bgp prefix counters on 23.4R2-S8.7 - https://phabricator.wikimedia.org/T433490 (10cmooney) 03NEW p:05Triage→03Medium [11:03:34] 06SRE, 06Infrastructure-Foundations, 10netops: QFX5120 reporting 0 for neighbor bgp prefix counters on 23.4R2-S8.7 - https://phabricator.wikimedia.org/T433490#12166714 (10cmooney) Case 2026-0729-825879 created [11:05:44] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [11:07:15] (03Abandoned) 10Reedy: CirrusSearch-production: Mark 'CirrusSearch Streaming Updater' as a reserved username [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1300182 (https://phabricator.wikimedia.org/T428687) (owner: 10Reedy) [11:08:20] (03CR) 10Btullis: [C:03+2] hadoop-test: Put the refreshed nodes into insetup mode prior to decom [puppet] - 10https://gerrit.wikimedia.org/r/1319059 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [11:12:26] (03CR) 10Kevin Bazira: "besides the Merge Conflict, the patch LGTM!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319050 (owner: 10Bartosz Wójtowicz) [11:12:51] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [11:13:01] (03PS3) 10Bartosz Wójtowicz: ml-services: Remove ml-serve1012 affinity from smaller models. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319050 [11:13:25] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [11:14:12] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1186: Maintenance [11:14:43] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1195.eqiad.wmnet with reason: Maintenance [11:14:51] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1195 (T431660)', diff saved to https://phabricator.wikimedia.org/P95514 and previous config saved to /var/cache/conftool/dbconfig/20260729-111450-cwilliams.json [11:16:26] (03CR) 10Kevin Bazira: [C:03+1] ml-services: Remove ml-serve1012 affinity from smaller models. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319050 (owner: 10Bartosz Wójtowicz) [11:18:03] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: Remove ml-serve1012 affinity from smaller models. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319050 (owner: 10Bartosz Wójtowicz) [11:20:16] (03Merged) 10jenkins-bot: ml-services: Remove ml-serve1012 affinity from smaller models. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319050 (owner: 10Bartosz Wójtowicz) [11:21:16] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1195: Maintenance [11:22:00] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [11:23:51] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [11:26:37] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1198: Maintenance [11:26:58] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance [11:27:20] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance [11:27:28] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1212 (T431660)', diff saved to https://phabricator.wikimedia.org/P95517 and previous config saved to /var/cache/conftool/dbconfig/20260729-112727-cwilliams.json [11:27:48] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply [11:28:32] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply [11:28:49] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:29:02] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1238: Maintenance [11:29:11] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1241.eqiad.wmnet with reason: Maintenance [11:29:19] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1241 (T431660)', diff saved to https://phabricator.wikimedia.org/P95520 and previous config saved to /var/cache/conftool/dbconfig/20260729-112918-cwilliams.json [11:29:45] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply [11:30:11] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply [11:32:13] (03CR) 10Filippo Giunchedi: "Out of curiosity which base images don't ship systemd ?" [debs/pint] - 10https://gerrit.wikimedia.org/r/1319039 (owner: 10Hnowlan) [11:32:14] (03PS1) 10Elukey: services: bump proton's image to pick up security updates [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319068 [11:33:27] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1212: Maintenance [11:35:43] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1241: Maintenance [11:36:50] (03PS1) 10Btullis: datahub-next: Update the MariaDB database to the refreshed host [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319070 (https://phabricator.wikimedia.org/T406746) [11:37:56] (03CR) 10Brouberol: [C:03+1] datahub-next: Update the MariaDB database to the refreshed host [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319070 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [11:43:34] (03CR) 10Btullis: [C:03+2] datahub-next: Update the MariaDB database to the refreshed host [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319070 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [11:45:42] (03Merged) 10jenkins-bot: datahub-next: Update the MariaDB database to the refreshed host [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319070 (https://phabricator.wikimedia.org/T406746) (owner: 10Btullis) [11:47:48] (03PS1) 10Elukey: profile::thanos: fix search sli's recording rule [puppet] - 10https://gerrit.wikimedia.org/r/1319071 [11:48:45] (03CR) 10Elukey: [C:03+2] services: bump proton's image to pick up security updates [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319068 (owner: 10Elukey) [11:49:19] !log elukey@deploy1003 helmfile [staging] START helmfile.d/services/proton: sync [11:50:06] (03CR) 10Elukey: [C:03+2] profile::thanos: fix search sli's recording rule [puppet] - 10https://gerrit.wikimedia.org/r/1319071 (owner: 10Elukey) [11:50:10] !log elukey@deploy1003 helmfile [staging] DONE helmfile.d/services/proton: sync [11:50:10] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2033 is not sending any prefix to lsw1-b7-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [11:50:44] !log btullis@cumin1003 START - Cookbook sre.hosts.decommission for hosts an-test-coord1001.eqiad.wmnet [11:51:26] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply [11:51:43] !log elukey@deploy1003 helmfile [codfw] START helmfile.d/services/proton: sync [11:51:51] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply [11:52:53] !log elukey@deploy1003 helmfile [codfw] DONE helmfile.d/services/proton: sync [11:54:13] (03CR) 10Audrey Penven: [C:03+1] "the diff for the current patch looks much more reasonable - affecting only the service we mean to change." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1318731 (owner: 10Lucas Werkmeister (WMDE)) [11:55:35] (03CR) 10Jelto: [V:03+2 C:03+2] golang: add trixie-based golang-1.25 image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1313867 (https://phabricator.wikimedia.org/T427402) (owner: 10Jelto) [11:55:39] !log btullis@cumin1003 START - Cookbook sre.dns.netbox [11:58:56] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1197: Maintenance [12:00:04] Daimona: Create new tables for the CampaignEvents extension (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T1200). Please do the needful. [12:00:08] !log btullis@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-test-coord1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" [12:00:09] !log Rename tables T425074 [12:00:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:00:15] T425074: Drop CheckUser tables from closed wikis - https://phabricator.wikimedia.org/T425074 [12:00:17] 06SRE, 10hCaptcha, 06Product Safety and Integrity, 06ServiceOps new, and 2 others: memcached errors seen in hCaptcha health checks - https://phabricator.wikimedia.org/T430340#12166872 (10JMeybohm) Hey @kostajh could you please elaborate what the impact of this is to the hCaptcha service and what you would... [12:00:59] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.bmc-user-mgmt (exit_code=0) for 1098 hosts [12:03:12] btullis@cumin1003 decommission (PID 3619094) is awaiting input [12:04:17] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance [12:04:25] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1222 (T431660)', diff saved to https://phabricator.wikimedia.org/P95530 and previous config saved to /var/cache/conftool/dbconfig/20260729-120424-cwilliams.json [12:04:43] (03PS2) 10Federico Ceratto: mysql.multiinstance_reboot_test: switch to mocker [cookbooks] - 10https://gerrit.wikimedia.org/r/1314730 [12:04:52] (03CR) 10Klausman: [C:03+2] base: bump BPO Trixie kernel to 6.19 (only used by big ML GPU hosts atm) [puppet] - 10https://gerrit.wikimedia.org/r/1319051 (https://phabricator.wikimedia.org/T433485) (owner: 10Klausman) [12:05:36] (03CR) 10Federico Ceratto: "Rebased" [cookbooks] - 10https://gerrit.wikimedia.org/r/1314730 (owner: 10Federico Ceratto) [12:07:58] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1195: Maintenance [12:08:08] (03CR) 10CI reject: [V:04-1] mysql.multiinstance_reboot_test: switch to mocker [cookbooks] - 10https://gerrit.wikimedia.org/r/1314730 (owner: 10Federico Ceratto) [12:08:19] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance [12:08:40] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance [12:08:48] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1196 (T431660)', diff saved to https://phabricator.wikimedia.org/P95533 and previous config saved to /var/cache/conftool/dbconfig/20260729-120847-cwilliams.json [12:11:39] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance [12:11:47] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1159 (T431660)', diff saved to https://phabricator.wikimedia.org/P95534 and previous config saved to /var/cache/conftool/dbconfig/20260729-121146-cwilliams.json [12:11:52] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance [12:12:04] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance [12:12:12] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1158 (T431660)', diff saved to https://phabricator.wikimedia.org/P95535 and previous config saved to /var/cache/conftool/dbconfig/20260729-121211-cwilliams.json [12:12:37] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1222: Maintenance [12:14:42] !log Creating new DB tables for the CampaignEvents extension in x1.testwiki, x1.test2wiki, x1.officewiki, and x1.wikishared # T429339 [12:14:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:14:46] T429339: Create DB schema for storing worklists - https://phabricator.wikimedia.org/T429339 [12:15:25] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1196: Maintenance [12:15:51] !log elukey@deploy1003 helmfile [eqiad] START helmfile.d/services/proton: sync [12:17:03] !log elukey@deploy1003 helmfile [eqiad] DONE helmfile.d/services/proton: sync [12:17:29] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1159: Maintenance [12:19:02] I'm done with my tables. [12:19:10] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1212: Maintenance [12:19:29] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance [12:19:38] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1223 (T431660)', diff saved to https://phabricator.wikimedia.org/P95540 and previous config saved to /var/cache/conftool/dbconfig/20260729-121937-cwilliams.json [12:20:21] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1158: Maintenance [12:20:40] !log elukey@cumin1003 START - Cookbook sre.hosts.bmc-user-mgmt for 1051 hosts [12:22:26] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1241: Maintenance [12:22:46] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1242.eqiad.wmnet with reason: Maintenance [12:22:54] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1242 (T431660)', diff saved to https://phabricator.wikimedia.org/P95544 and previous config saved to /var/cache/conftool/dbconfig/20260729-122254-cwilliams.json [12:23:56] (03PS2) 10Anzx: remove throttle exceptions for concluded events [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319075 [12:25:42] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319075 (owner: 10Anzx) [12:25:45] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1223: Maintenance [12:28:16] 14SRE-Sprint-Week-Sustainability-March2023, 06Traffic, 13Patch-For-Review, 07Sustainability (Incident Followup): Experiment with single backend CDN nodes - https://phabricator.wikimedia.org/T288106#12166920 (10BBlack) Looks "reasonable"? The biggest diff is in the loss in revalidation hits (which makes se... [12:29:28] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1242: Maintenance [12:31:17] (03CR) 10Marostegui: [C:03+1] "The compiler seems to be fine: https://puppet-compiler.wmflabs.org/output/1319029/7392/" [puppet] - 10https://gerrit.wikimedia.org/r/1319029 (https://phabricator.wikimedia.org/T433344) (owner: 10CWilliams) [12:32:52] !log klausman@cumin1003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{ml-serve101[2-5].eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad) [12:32:56] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1012.eqiad.wmnet [12:33:16] !log btullis@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-test-coord1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" [12:33:16] !log btullis@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:33:17] !log btullis@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-test-coord1001.eqiad.wmnet [12:33:24] (03PS1) 10STran: SI: Instrument abuse filter hits link [extensions/CheckUser] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1319077 (https://phabricator.wikimedia.org/T433053) [12:33:36] (03PS1) 10STran: SI: Instrument abuse filter hits link [extensions/CheckUser] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1319078 (https://phabricator.wikimedia.org/T433053) [12:33:47] (03PS1) 10STran: SI: Instrument interaction timeline link [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1319079 (https://phabricator.wikimedia.org/T433053) [12:33:57] (03PS1) 10STran: SI: Instrument interaction timeline link [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1319080 (https://phabricator.wikimedia.org/T433053) [12:34:06] !log btullis@cumin1003 START - Cookbook sre.hosts.decommission for hosts an-test-master[1001-1002].eqiad.wmnet [12:34:12] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/CheckUser] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1319077 (https://phabricator.wikimedia.org/T433053) (owner: 10STran) [12:34:20] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/CheckUser] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1319078 (https://phabricator.wikimedia.org/T433053) (owner: 10STran) [12:34:27] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.12) - 10https://gerrit.wikimedia.org/r/1319079 (https://phabricator.wikimedia.org/T433053) (owner: 10STran) [12:34:34] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/WikimediaCustomizations] (wmf/1.47.0-wmf.13) - 10https://gerrit.wikimedia.org/r/1319080 (https://phabricator.wikimedia.org/T433053) (owner: 10STran) [12:35:30] (03CR) 10Ayounsi: [C:03+2] Add missing PTR include for private1-23-ulsfo and private1-604-eqsin (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1318769 (owner: 10Ayounsi) [12:35:53] !log ayounsi@dns1004 START - running authdns-update [12:37:59] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1012.eqiad.wmnet [12:38:03] !log ayounsi@dns1004 END - running authdns-update [12:38:05] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, July 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1309675 (https://phabricator.wikimedia.org/T431830) (owner: 10Dreamrimmer) [12:39:12] (03CR) 10Ssingh: [V:03+1 C:03+2] service: add definition for urldownloader service [puppet] - 10https://gerrit.wikimedia.org/r/1313947 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [12:40:11] !log cwilliams@cumin1003 START - Cookbook sre.mysql.multiinstance_reboot for db-test2001.codfw.wmnet [12:40:52] !log cwilliams@cumin1003 END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for db-test2001.codfw.wmnet [12:41:47] (03CR) 10CWilliams: [C:03+1] "The detection of a single replica is still working." [cookbooks] - 10https://gerrit.wikimedia.org/r/1315819 (https://phabricator.wikimedia.org/T420203) (owner: 10FNegri) [12:44:23] !log btullis@cumin1003 START - Cookbook sre.dns.netbox [12:45:01] !log sudo cumin 'O:url_downloader' 'disable-puppet "merging CR 1313948"': T429175 [12:45:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:45:04] T429175: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175 [12:45:12] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1012.eqiad.wmnet [12:45:13] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1012.eqiad.wmnet [12:45:21] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1013.eqiad.wmnet [12:46:56] (03CR) 10Ssingh: [V:03+1 C:03+2] hiera: url_downloader: enable LVS pool and IPIP [puppet] - 10https://gerrit.wikimedia.org/r/1313948 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [12:48:49] !log btullis@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-test-master[1001-1002].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1003" [12:50:06] !log sudo cumin 'O:url_downloader' 'run-puppet-agent --enable "merging CR 1313948"': T429175 [12:50:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:50:10] T429175: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175 [12:50:27] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet [12:51:53] btullis@cumin1003 decommission (PID 3626581) is awaiting input [12:51:56] (03PS1) 10Ssingh: service: move urldownloaders to lvs_setup [puppet] - 10https://gerrit.wikimedia.org/r/1319082 (https://phabricator.wikimedia.org/T429175) [12:56:06] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet [12:56:07] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet [12:56:13] !log klausman@cumin1003 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet [12:57:06] (03CR) 10Slyngshede: [C:03+1] "Looks good, matches the documentation :-)" [puppet] - 10https://gerrit.wikimedia.org/r/1319082 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [12:57:13] (03PS1) 10Jcrespo: dbbackups: Setup db1265 & db1285 as new backup sources [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) [12:57:48] !log sudo cumin 'A:lvs and (A:eqiad or A:codfw)' 'disable-puppet "adding new service urldownloader"': T429175 [12:57:49] 10ops-codfw, 06SRE, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team (Q1 FY2026-27): hw troubleshooting: Memory errors for ml-serve2002.codfw.wmnet - https://phabricator.wikimedia.org/T433476#12167051 (10isarantopoulos) [12:57:50] FIRING: KubernetesCalicoDown: ml-serve1013.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s-mlserve&var-instance=ml-serve1013.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:57:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:57:52] T429175: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175 [12:57:58] 10ops-codfw, 06SRE, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team (Q1 FY2026-27): hw troubleshooting: memory errors on ml-serve2004.codfw.wmnet - https://phabricator.wikimedia.org/T433478#12167052 (10isarantopoulos) [12:58:18] (03CR) 10Ssingh: [C:03+2] service: move urldownloaders to lvs_setup [puppet] - 10https://gerrit.wikimedia.org/r/1319082 (https://phabricator.wikimedia.org/T429175) (owner: 10Ssingh) [12:58:33] (03PS2) 10Jcrespo: dbbackups: Setup db1265 & db1285 as new backup sources [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) [12:58:40] (03CR) 10Jcrespo: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [12:58:49] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [12:58:51] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team (Q1 FY2026-27): hw troubleshooting: failing disk in ml-serve1001.eqiad.wmnet - https://phabricator.wikimedia.org/T432105#12167058 (10isarantopoulos) [12:59:21] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1222: Maintenance [12:59:43] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance [12:59:51] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1229 (T431660)', diff saved to https://phabricator.wikimedia.org/P95559 and previous config saved to /var/cache/conftool/dbconfig/20260729-125950-cwilliams.json [13:00:04] Lucas_WMDE, urbanecm, and TheresNoTime: UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260729T1300). Please do the needful. [13:00:04] anzx, Tran, and DreamRimmer: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:09] o/ [13:00:17] o/ [13:00:24] o/ [13:00:28] !log jgiannelos@deploy1003 helmfile [staging] START helmfile.d/services/kartotherian: apply [13:00:29] I’m in a meeting and can’t deploy, sorry [13:00:32] !log jgiannelos@deploy1003 helmfile [staging] DONE helmfile.d/services/kartotherian: apply [13:01:17] I can self-deploy but am also in a meeting so am not sure I can help out after [13:01:19] !log cmooney@cumin1003 START - Cookbook sre.dns.netbox [13:01:22] !log sukhe@lvs1020:~$ sudo systemctl restart pybal.service [13:01:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:01:48] (⇒ https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/254) [13:02:24] Tran: could you start with yours self-deploying, and then I can do DreamRimmer and anzx 's if needed [13:02:31] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1196: Maintenance [13:02:50] RESOLVED: KubernetesCalicoDown: ml-serve1013.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s-mlserve&var-instance=ml-serve1013.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:02:51] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance [13:02:59] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1206 (T431660)', diff saved to https://phabricator.wikimedia.org/P95562 and previous config saved to /var/cache/conftool/dbconfig/20260729-130258-cwilliams.json [13:03:17] (03CR) 10Jcrespo: [C:04-1] "Hey, Manuel, this is not in a rush, I still have to reimage these hosts, but want to check with you (review request) for a quick sanity ch" [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [13:03:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:04:14] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - urldownloader_8080: Servers urldownloader1005.wikimedia.org are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [13:04:26] TheresNoTime: I forgot keys rotated and haven't updated. Can you start? I'll self-deploy after. [13:04:36] (03PS3) 10Jcrespo: dbbackups: Setup db1265 & db1285 as new backup sources [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) [13:04:39] Okay! [13:04:40] (03CR) 10Jcrespo: [C:04-1] "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [13:04:43] anzx: starting with yours [13:05:10] (03CR) 10TrainBranchBot: [C:03+2] "Approved by samtar@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319075 (owner: 10Anzx) [13:05:15] (03PS4) 10Jcrespo: dbbackups: Setup db1265 & db1285 as new backup sources [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) [13:05:18] (03PS1) 10Bartosz Wójtowicz: ml-services: Add minReplicas=1 to gpt-oss and qwen36 deployments. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1319085 (https://phabricator.wikimedia.org/T432933) [13:05:20] (03CR) 10Marostegui: "This looks good, just a minor non blocking comment." [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [13:05:26] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1159: Maintenance [13:05:31] !log mvernon@cumin2003 START - Cookbook sre.hosts.reimage for host ms-be2073.codfw.wmnet with OS trixie [13:05:34] (03CR) 10Marostegui: [C:03+1] dbbackups: Setup db1265 & db1285 as new backup sources [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [13:05:42] !log cmooney@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add mgmt IPs new switches - cmooney@cumin1003" [13:05:47] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance [13:05:47] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12167092 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2003 for host ms-be2073.codfw.wmnet with OS trixie [13:05:55] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1229: Maintenance [13:06:00] !log mvernon@cumin1003 START - Cookbook sre.hosts.reimage for host ms-be1077.eqiad.wmnet with OS trixie [13:06:03] TheresNoTime: ok [13:06:08] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance [13:06:14] 06SRE, 10SRE-swift-storage, 07Essential-Work: Migrate production swift clusters to trixie - https://phabricator.wikimedia.org/T429630#12167094 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-be1077.eqiad.wmnet with OS trixie [13:06:17] nothing to test mine [13:06:17] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1161 (T431660)', diff saved to https://phabricator.wikimedia.org/P95564 and previous config saved to /var/cache/conftool/dbconfig/20260729-130616-cwilliams.json [13:06:19] !log klausman@cumin1003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet [13:06:29] (03Merged) 10jenkins-bot: remove throttle exceptions for concluded events [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319075 (owner: 10Anzx) [13:06:36] anzx: (ack, will just run it straight through) [13:07:02] !log root@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1158: Maintenance [13:07:18] (03CR) 10Jcrespo: dbbackups: Setup db1265 & db1285 as new backup sources (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1319083 (https://phabricator.wikimedia.org/T407942) (owner: 10Jcrespo) [13:07:18] !log cmooney@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add mgmt IPs new switches - cmooney@cumin1003" [13:07:18] !log cmooney@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:07:22] !log cwilliams@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance [13:07:31] !log cwilliams@cumin1003 dbctl commit (dc=all): 'Depooling db1170 (T431660)', diff saved to https://phabricator.wikimedia.org/P95566 and previous config saved to /var/cache/conftool/dbconfig/20260729-130730-cwilliams.json [13:07:38] !log samtar@deploy1003 Started scap sync-world: Backport for [[gerrit:1319075|remove throttle exceptions for concluded events]] [13:08:53] !log sukhe@lvs2014:~$ sudo systemctl restart pybal.service [13:08:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:09:07] !log root@cumin1003 START - Cookbook sre.mysql.pool pool db1206: Maintenance [13:09:17] (03PS1) 10Mpostoronca: WikimediaAntiAbuse: Document required load order after Echo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319086 (https://phabricator.wikimedia.org/T432452) [13:09:45] !log samtar@deploy1003 anzx, samtar: Backport for [[gerrit:1319075|remove throttle exceptions for concluded events]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:10:18] !log samtar@deploy1003 anzx, samtar: Continuing with deployment