[00:01:02] 06SRE: Bing Webmaster Tools access request for Ash Han - https://phabricator.wikimedia.org/T439646#12377560 (10Maryana) Giving my approval as Ash's manager! [00:21:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:23:40] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 600.10 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:37:31] FIRING: Traffic bill over quota: Alert for device cr2-magru.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [00:57:31] RESOLVED: Traffic bill over quota: Alert for device cr2-magru.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [01:08:57] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1346194 [01:08:57] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1346194 (owner: 10TrainBranchBot) [01:16:28] RECOVERY - MariaDB Replica Lag: s4 on db1265 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [01:16:56] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1346194 (owner: 10TrainBranchBot) [01:29:01] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [01:47:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [01:49:01] FIRING: SystemdUnitFailed: wmf_auto_restart_krb5-admin-server.service on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:49:23] (03PS2) 10AKhatun: Add article-feature-counts alerts [alerts] - 10https://gerrit.wikimedia.org/r/1344530 (https://phabricator.wikimedia.org/T438384) [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T0200) [02:00:33] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:01:37] (03PS5) 10Krinkle: Profiler: Fix excimer component regex skipping the last frame in each stack [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328191 (owner: 10SomeRandomDeveloper) [02:01:52] (03CR) 10Krinkle: Profiler: Fix excimer component regex skipping the last frame in each stack [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328191 (owner: 10SomeRandomDeveloper) [02:01:57] (03PS6) 10Krinkle: Profiler: Fold excimer line parsing into one method [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1329640 [02:03:40] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.03 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:08:14] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 40s) [02:08:28] PROBLEM - MariaDB Replica Lag: s3 on db1265 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 627.89 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:34:01] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [03:10:54] 10SRE-swift-storage, 07Wikimedia-production-error: swift not found trying to revert image no Commons - https://phabricator.wikimedia.org/T439477#12377657 (10Ladsgroup) [03:29:01] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [03:55:28] RECOVERY - MariaDB Replica Lag: s3 on db1265 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [04:21:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:22:31] (03PS1) 10KartikMistry: Update Apertium to 2026-09-30-051826-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346288 (https://phabricator.wikimedia.org/T437213) [05:25:12] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [05:29:01] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:30:47] (03PS1) 10Kevin Bazira: ml-services: deploy qwen3-14b-fp8 LLMInferenceService in LiftWing staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346299 (https://phabricator.wikimedia.org/T439659) [05:47:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [05:49:01] FIRING: SystemdUnitFailed: wmf_auto_restart_krb5-admin-server.service on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:51:19] (03CR) 10Giuseppe Lavagetto: [C:04-1] "Overall LGTM; there is mostly one trivial error left, and a couple questions for you to answer; the rest of the comments are optional and " [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958) (owner: 10Dduvall) [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T0600) [06:04:33] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1346165 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [06:06:58] (03PS1) 10Catrope: Email confirmation: implement delayed enforcement experiment [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346327 (https://phabricator.wikimedia.org/T439283) [06:07:12] (03PS1) 10Catrope: Email confirmation: Add server-side enforcement [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346328 (https://phabricator.wikimedia.org/T439283) [06:07:30] (03PS1) 10Catrope: Email confirmation: Prepare for next experiment [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346329 (https://phabricator.wikimedia.org/T439283) [06:08:28] (03CR) 10Muehlenhoff: "Sorry, I missed that you were off. I'll merge and deploy in a bit." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1345920 (owner: 10Muehlenhoff) [06:08:31] (03CR) 10Muehlenhoff: [C:03+2] thumbor-plugins: Rebuild against latest package versions in Trixie [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1345920 (owner: 10Muehlenhoff) [06:26:18] (03PS1) 10Awight: [WIP] Remove unused StatsD configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346336 (https://phabricator.wikimedia.org/T228380) [06:26:38] (03PS1) 10Filippo Giunchedi: install_server: add partman settings for megaraid 10 jbod [puppet] - 10https://gerrit.wikimedia.org/r/1346337 (https://phabricator.wikimedia.org/T419892) [06:26:40] (03PS1) 10Filippo Giunchedi: install_server: use hwraid-10jbod for cloudcephosd105[3-6] [puppet] - 10https://gerrit.wikimedia.org/r/1346338 (https://phabricator.wikimedia.org/T419892) [06:32:17] !log filippo@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1055.eqiad.wmnet with OS bookworm [06:32:31] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware), 13Patch-For-Review: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12377823 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1004 for host cloudcephosd1055.eqiad.wmn... [06:33:17] (03CR) 10Bartosz Wójtowicz: [C:03+1] ml-services: deploy qwen3-14b-fp8 LLMInferenceService in LiftWing staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346299 (https://phabricator.wikimedia.org/T439659) (owner: 10Kevin Bazira) [06:33:53] !log filippo@cumin1004 START - Cookbook sre.hosts.dhcp for host cloudcephosd1056.eqiad.wmnet [06:34:01] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [06:36:56] filippo@cumin1004 dhcp (PID 1745781) is awaiting input [06:43:22] mmhh why yes, by design waiting for input [06:48:56] !log filippo@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudcephosd1056.eqiad.wmnet [06:49:01] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [06:51:24] (03PS1) 10Ryan Kemper: kafka-configurator: install on brokers [puppet] - 10https://gerrit.wikimedia.org/r/1346361 (https://phabricator.wikimedia.org/T435282) [06:51:33] !log filippo@cumin1004 START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm [06:52:00] 06SRE, 06Infrastructure-Foundations: Unresolved failures from url-downloader - https://phabricator.wikimedia.org/T439560#12377843 (10RThomas-WMF) Thanks Cathal! I think those logs will be valuable to investigate this issue. To clarify a few things from what was visible in the AWS infrastructure: All the fai... [06:52:39] RESOLVED: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [06:53:04] (03PS1) 10Muehlenhoff: thumbor: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346363 [06:53:54] (03CR) 10Ryan Kemper: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1346361 (https://phabricator.wikimedia.org/T435282) (owner: 10Ryan Kemper) [06:54:01] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [06:59:32] (03CR) 10Filippo Giunchedi: [C:03+2] "Will try this on cloudcephosd105[3-6] and report back, hosts are not in service" [puppet] - 10https://gerrit.wikimedia.org/r/1346337 (https://phabricator.wikimedia.org/T419892) (owner: 10Filippo Giunchedi) [06:59:49] (03CR) 10Filippo Giunchedi: [C:03+2] "Hosts are not in service, self-merging" [puppet] - 10https://gerrit.wikimedia.org/r/1346338 (https://phabricator.wikimedia.org/T419892) (owner: 10Filippo Giunchedi) [07:00:05] Amir1, urbanecm, and awight: gettimeofday() says it's time for UTC morning backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T0700) [07:00:05] dcausse: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:00:17] o/ [07:00:28] I can deploy [07:01:26] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346012 (owner: 10DCausse) [07:01:57] (03CR) 10Elukey: [C:03+2] Move the pki service entry to service_setup [puppet] - 10https://gerrit.wikimedia.org/r/1338937 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [07:02:34] (03Merged) 10jenkins-bot: Search: optimize morelike queries [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346012 (owner: 10DCausse) [07:03:35] !log dcausse@deploy1003 Started scap sync-world: Backport for [[gerrit:1346012|Search: optimize morelike queries]] [07:06:30] (03CR) 10JMeybohm: python-webapp: Update ingress.istio to 1.4.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344831 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [07:06:36] PROBLEM - gdnsd checkconf #page on dns4004 is CRITICAL: CRITICAL: gdnsd -S checkconf failure https://wikitech.wikimedia.org/wiki/DNS%23gdnsd_checkconf [07:07:37] PROBLEM - gdnsd checkconf #page on dns1004 is CRITICAL: CRITICAL: gdnsd -S checkconf failure https://wikitech.wikimedia.org/wiki/DNS%23gdnsd_checkconf [07:07:55] (03CR) 10JMeybohm: [C:03+1] runtimeClasses: add gVisor-kvm RuntimeClass [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342964 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [07:07:57] !log dcausse@deploy1003 dcausse: Backport for [[gerrit:1346012|Search: optimize morelike queries]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:09:43] !log dcausse@deploy1003 dcausse: Continuing with deployment [07:09:44] hey elukey, would this be you? [07:09:48] Sep 30 07:05:08 dns1004 gdnsd[2119199]: plugin_geoip: Invalid resource name 'disc-pki' detected from zonefile lookup [07:09:54] hnowlan: yes yes I am, see #sre, sorry :( [07:10:04] ahhh ok [07:10:05] no worries [07:11:41] FIRING: [2x] ConfdResourceFailed: confd resource _srv_config-master_pybal_codfw_pki.toml has errors - https://wikitech.wikimedia.org/wiki/Confd#Monitoring - https://grafana.wikimedia.org/d/OUJF1VI4k/confd - https://alerts.wikimedia.org/?q=alertname%3DConfdResourceFailed [07:12:40] !log filippo@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm [07:13:26] !log filippo@cumin1004 START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm [07:13:36] (03CR) 10JMeybohm: [C:04-1] shellbox-timeline: run under kvm in eqiad (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342965 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [07:13:59] (03CR) 10Brouberol: [C:03+1] "Nice!" [puppet] - 10https://gerrit.wikimedia.org/r/1346361 (https://phabricator.wikimedia.org/T435282) (owner: 10Ryan Kemper) [07:14:17] (03PS1) 10Elukey: Revert "Move the pki service entry to service_setup" [puppet] - 10https://gerrit.wikimedia.org/r/1346373 [07:14:22] (03PS1) 10Muehlenhoff: Add explicit replica hostnames to LDAP cert SNIs [puppet] - 10https://gerrit.wikimedia.org/r/1346374 (https://phabricator.wikimedia.org/T331699) [07:15:02] !log dcausse@deploy1003 Finished scap sync-world: Backport for [[gerrit:1346012|Search: optimize morelike queries]] (duration: 11m 27s) [07:15:21] (03CR) 10Hnowlan: [C:03+1] Revert "Move the pki service entry to service_setup" [puppet] - 10https://gerrit.wikimedia.org/r/1346373 (owner: 10Elukey) [07:15:33] (03CR) 10Elukey: [C:03+2] Revert "Move the pki service entry to service_setup" [puppet] - 10https://gerrit.wikimedia.org/r/1346373 (owner: 10Elukey) [07:17:37] RECOVERY - gdnsd checkconf #page on dns1004 is OK: OK: gdnsd -S checkconf success https://wikitech.wikimedia.org/wiki/DNS%23gdnsd_checkconf [07:18:16] !log filippo@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm [07:18:24] (03CR) 10Kevin Bazira: [C:03+2] ml-services: deploy qwen3-14b-fp8 LLMInferenceService in LiftWing staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346299 (https://phabricator.wikimedia.org/T439659) (owner: 10Kevin Bazira) [07:18:38] !log filippo@cumin1004 START - Cookbook sre.hosts.reimage for host cloudcephosd1056.eqiad.wmnet with OS bookworm [07:19:37] RECOVERY - gdnsd checkconf #page on dns4004 is OK: OK: gdnsd -S checkconf success https://wikitech.wikimedia.org/wiki/DNS%23gdnsd_checkconf [07:20:50] (03Merged) 10jenkins-bot: ml-services: deploy qwen3-14b-fp8 LLMInferenceService in LiftWing staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346299 (https://phabricator.wikimedia.org/T439659) (owner: 10Kevin Bazira) [07:23:05] (03PS3) 10Giuseppe Lavagetto: shellbox-timeline: run under kvm in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342965 (https://phabricator.wikimedia.org/T438287) [07:23:59] (03PS1) 10Elukey: Remove the pki discovery settings [dns] - 10https://gerrit.wikimedia.org/r/1346377 (https://phabricator.wikimedia.org/T436809) [07:24:04] (03CR) 10JMeybohm: bird: add a new prometheus-exporter for bird. (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1343943 (https://phabricator.wikimedia.org/T423851) (owner: 10Blake) [07:25:03] (03CR) 10CI reject: [V:04-1] Remove the pki discovery settings [dns] - 10https://gerrit.wikimedia.org/r/1346377 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [07:28:19] (03CR) 10Slyngshede: [C:03+1] Add explicit replica hostnames to LDAP cert SNIs [puppet] - 10https://gerrit.wikimedia.org/r/1346374 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [07:29:56] (03PS2) 10Elukey: Remove the pki discovery settings [dns] - 10https://gerrit.wikimedia.org/r/1346377 (https://phabricator.wikimedia.org/T436809) [07:32:03] (03PS1) 10Elukey: Revert^2 "Move the pki service entry to service_setup" [puppet] - 10https://gerrit.wikimedia.org/r/1346379 [07:32:25] (03CR) 10Elukey: "To be merged after https://gerrit.wikimedia.org/r/c/operations/dns/+/1346377" [puppet] - 10https://gerrit.wikimedia.org/r/1346379 (owner: 10Elukey) [07:32:36] (03CR) 10Vgutierrez: [C:03+1] Remove the pki discovery settings [dns] - 10https://gerrit.wikimedia.org/r/1346377 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [07:35:38] !log installing pcre2/openssl security updates [07:35:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:41:41] RESOLVED: [2x] ConfdResourceFailed: confd resource _srv_config-master_pybal_codfw_pki.toml has errors - https://wikitech.wikimedia.org/wiki/Confd#Monitoring - https://grafana.wikimedia.org/d/OUJF1VI4k/confd - https://alerts.wikimedia.org/?q=alertname%3DConfdResourceFailed [07:43:58] (03CR) 10JMeybohm: [C:03+1] shellbox-timeline: run under kvm in eqiad (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342965 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [07:44:08] (03CR) 10Elukey: [C:03+2] Remove the pki discovery settings [dns] - 10https://gerrit.wikimedia.org/r/1346377 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [07:44:26] !log elukey@dns1004 START - running authdns-update [07:44:54] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [07:44:59] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [07:45:18] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 233.51 ms [07:46:54] !log elukey@dns1004 END - running authdns-update [07:48:57] (03CR) 10Vgutierrez: [C:03+1] Revert^2 "Move the pki service entry to service_setup" [puppet] - 10https://gerrit.wikimedia.org/r/1346379 (owner: 10Elukey) [07:49:15] (03CR) 10Elukey: [C:03+2] Revert^2 "Move the pki service entry to service_setup" [puppet] - 10https://gerrit.wikimedia.org/r/1346379 (owner: 10Elukey) [07:49:49] !log filippo@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcephosd1056.eqiad.wmnet with OS bookworm [07:51:09] (03PS1) 10SarthakSingh2904: ruwikisource: Add Cyrillic 'А' alias for Author namespace Add the alias 'А:' (Cyrillic А, U+0410) for the Author namespace (NS 102) on ruwikisource, per community consensus. Bug: T439568 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346421 (https://phabricator.wikimedia.org/T439568) [07:53:53] (03PS2) 10SarthakSingh2904: ruwikisource: Add Cyrillic 'А' alias for Author namespace [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346421 (https://phabricator.wikimedia.org/T439568) [07:53:58] 10SRE-Access-Requests: Add FIDO-backed SSH keys for kharlan - https://phabricator.wikimedia.org/T439665 (10kostajh) 03NEW [07:55:45] (03PS1) 10Kosta Harlan: admin: Add FIDO SSH keys for kharlan [puppet] - 10https://gerrit.wikimedia.org/r/1346423 (https://phabricator.wikimedia.org/T439665) [07:55:45] FIRING: WidespreadPuppetFailure: Puppet has failed in ulsfo - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [07:55:47] (03PS1) 10Kosta Harlan: admin: Remove non-FIDO SSH key for kharlan [puppet] - 10https://gerrit.wikimedia.org/r/1346424 (https://phabricator.wikimedia.org/T439665) [07:56:05] (03CR) 10Kosta Harlan: [C:04-1] "Need to verify the Yubikeys first" [puppet] - 10https://gerrit.wikimedia.org/r/1346424 (https://phabricator.wikimedia.org/T439665) (owner: 10Kosta Harlan) [07:56:57] (03CR) 10CI reject: [V:04-1] admin: Add FIDO SSH keys for kharlan [puppet] - 10https://gerrit.wikimedia.org/r/1346423 (https://phabricator.wikimedia.org/T439665) (owner: 10Kosta Harlan) [07:57:21] (03CR) 10CI reject: [V:04-1] admin: Remove non-FIDO SSH key for kharlan [puppet] - 10https://gerrit.wikimedia.org/r/1346424 (https://phabricator.wikimedia.org/T439665) (owner: 10Kosta Harlan) [07:58:05] (03PS2) 10Kosta Harlan: admin: Add FIDO SSH keys for kharlan [puppet] - 10https://gerrit.wikimedia.org/r/1346423 (https://phabricator.wikimedia.org/T439665) [07:58:12] (03PS2) 10Kosta Harlan: admin: Remove non-FIDO SSH key for kharlan [puppet] - 10https://gerrit.wikimedia.org/r/1346424 (https://phabricator.wikimedia.org/T439665) [07:58:16] (03PS3) 10Kosta Harlan: admin: Remove non-FIDO SSH key for kharlan [puppet] - 10https://gerrit.wikimedia.org/r/1346424 (https://phabricator.wikimedia.org/T439665) [07:58:41] (03PS7) 10Elukey: role::pki: add lvs configurations [puppet] - 10https://gerrit.wikimedia.org/r/1338936 (https://phabricator.wikimedia.org/T436809) [07:58:41] (03PS4) 10Elukey: Move pki.discovery.wmnet to lvs_setup state [puppet] - 10https://gerrit.wikimedia.org/r/1342214 (https://phabricator.wikimedia.org/T436809) [07:59:13] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12377978 (10fgiunchedi) I was mistaken again, even with drives on two different buses the ordering is not predictable. New idea/plan to get hwraid + jbod w... [08:00:14] (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1338936 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [08:04:23] (03CR) 10Elukey: [C:03+2] role::pki: add lvs configurations [puppet] - 10https://gerrit.wikimedia.org/r/1338936 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [08:05:45] FIRING: [3x] WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [08:08:09] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.15 point update - https://phabricator.wikimedia.org/T434631#12378044 (10MoritzMuehlenhoff) [08:09:41] FIRING: [2x] ConfdResourceFailed: confd resource _srv_config-master_pybal_codfw_pki.toml has errors - https://wikitech.wikimedia.org/wiki/Confd#Monitoring - https://grafana.wikimedia.org/d/OUJF1VI4k/confd - https://alerts.wikimedia.org/?q=alertname%3DConfdResourceFailed [08:13:24] (03PS2) 10Jgiannelos: prv: Enable parsoid rendering for 5 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345584 (https://phabricator.wikimedia.org/T439444) [08:14:41] FIRING: [4x] ConfdResourceFailed: confd resource _srv_config-master_pybal_codfw_pki.toml has errors - https://wikitech.wikimedia.org/wiki/Confd#Monitoring - https://grafana.wikimedia.org/d/OUJF1VI4k/confd - https://alerts.wikimedia.org/?q=alertname%3DConfdResourceFailed [08:15:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [08:16:48] (03CR) 10Giuseppe Lavagetto: [C:03+2] gVisor: disable on bookworm in containerd [puppet] - 10https://gerrit.wikimedia.org/r/1342961 (https://phabricator.wikimedia.org/T436649) (owner: 10Giuseppe Lavagetto) [08:18:15] (03CR) 10Muehlenhoff: [C:03+2] Add explicit replica hostnames to LDAP cert SNIs [puppet] - 10https://gerrit.wikimedia.org/r/1346374 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [08:18:24] elukey: ^^ the puppet failure seems to be triggered by your pki work [08:18:29] Could not evaluate: Could not retrieve file metadata for http://pki.discovery.wmnet/bundles/network_devices.pem: Request to http://pki.discovery.wmnet/bundles/network_devices.pem failed after 0.011 seconds: Failed to open TCP connection to pki.discovery.wmnet:80 (getaddrinfo: Name or service not known) [08:18:42] 06SRE, 06Collaboration-Services, 06Release-Engineering-Team, 07ci-test-error: Builds fail on integration-agent-docker-1086 due to expired snapshot.debian.org Release file - https://phabricator.wikimedia.org/T439529#12378071 (10hashar) p:05Triage→03High I have checked the erroring build https://integrat... [08:19:01] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [08:19:21] vgutierrez: okok checking thanks [08:19:26] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [08:20:45] FIRING: [5x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [08:21:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:22:25] !log jmm@cumin2003 START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-codfw [08:24:24] !log jmm@cumin2003 END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-codfw [08:24:57] !log elukey@cumin1004 conftool action : set/pooled=yes:weight=1; selector: cluster=pki,service=cfssl-multirootca [08:25:31] !log jmm@cumin2003 START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe-eqiad [08:25:33] should be fixed by --^ [08:25:38] nodes needed to be pooled [08:27:32] !log jmm@cumin2003 END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe-eqiad [08:29:41] RESOLVED: [4x] ConfdResourceFailed: confd resource _srv_config-master_pybal_codfw_pki.toml has errors - https://wikitech.wikimedia.org/wiki/Confd#Monitoring - https://grafana.wikimedia.org/d/OUJF1VI4k/confd - https://alerts.wikimedia.org/?q=alertname%3DConfdResourceFailed [08:33:08] (03PS5) 10Arnaudb: gerrit: add failover, DNS emergency and demotion cookbooks [cookbooks] - 10https://gerrit.wikimedia.org/r/1344697 (https://phabricator.wikimedia.org/T439025) [08:33:08] (03CR) 10Arnaudb: "All cookbooks have been reviewed and tested by a human (me) and could benefit from other reviewers" [cookbooks] - 10https://gerrit.wikimedia.org/r/1344697 (https://phabricator.wikimedia.org/T439025) (owner: 10Arnaudb) [08:33:13] (03PS5) 10Arnaudb: gerrit: read the replica flag from the loaded unit [cookbooks] - 10https://gerrit.wikimedia.org/r/1344698 (https://phabricator.wikimedia.org/T439025) [08:33:16] (03PS5) 10Arnaudb: gerrit: reuse the shared helpers in sibling cookbooks [cookbooks] - 10https://gerrit.wikimedia.org/r/1344699 (https://phabricator.wikimedia.org/T439025) [08:38:14] (03CR) 10Joal: sqoop_mediawiki: Add weekly change_tag sqoop (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [08:38:53] (03CR) 10Joal: sqoop_mediawiki: Add weekly full snapshot of the actor table (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1343954 (https://phabricator.wikimedia.org/T437961) (owner: 10A-pizzata) [08:45:21] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops, 06Machine-Learning-Team (Q1 FY2026-27): eqiad row A&B host migration details request for Machine Learning - https://phabricator.wikimedia.org/T432649#12378252 (10klausman) [08:49:49] 06SRE, 06Collaboration-Services, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07ci-test-error: Builds fail on integration-agent-docker-1086 due to expired snapshot.debian.org Release file - https://phabricator.wikimedia.org/T439529#12378271 (10hashar) The `operations/alerts` repos... [08:51:50] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345971 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [08:52:57] 06SRE, 06Infrastructure-Foundations: Unresolved failures from url-downloader - https://phabricator.wikimedia.org/T439560#12378282 (10cmooney) >>! In T439560#12377843, @RThomas-WMF wrote: > Hence, this suggests the request may be rejected by the proxy --> urldownloader before reaching AWS. If AWS rejects a call... [08:54:34] (03PS9) 10JMeybohm: gVisor: use kvm as a platform when hardware virtualization is available [puppet] - 10https://gerrit.wikimedia.org/r/1342962 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [08:55:12] (03CR) 10Elukey: [C:03+2] Move pki.discovery.wmnet to lvs_setup state [puppet] - 10https://gerrit.wikimedia.org/r/1342214 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [08:55:35] (03CR) 10Daniel Kertesz: [V:03+1] "I'm not particularly fond of having this hardcoded list of ports and ideally I would prefer some smarter way to get it, like for example a" [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) (owner: 10Daniel Kertesz) [08:57:08] 06SRE, 06Collaboration-Services, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07ci-test-error: Builds fail on integration-agent-docker-1086 due to expired snapshot.debian.org Release file - https://phabricator.wikimedia.org/T439529#12378293 (10hashar) As an immediate fix I have de... [08:58:42] (03CR) 10Btullis: [C:03+2] dse-k8s-eqiad: Retire the old Pod IP range [puppet] - 10https://gerrit.wikimedia.org/r/1345971 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [08:59:08] (03CR) 10Giuseppe Lavagetto: [C:03+2] gVisor: use kvm as a platform when hardware virtualization is available [puppet] - 10https://gerrit.wikimedia.org/r/1342962 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [09:01:33] !log elukey@cumin1004:~$ sudo cumin 'A:lvs-secondary-codfw' 'systemctl restart pybal.service' [09:01:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:01:43] (03PS1) 10Muehlenhoff: Add Cumin alias for wdqs::alternatives [puppet] - 10https://gerrit.wikimedia.org/r/1346444 [09:01:58] (03CR) 10Tiziano Fogli: [C:03+2] hiera/pdu_families: update regexps to match new models [puppet] - 10https://gerrit.wikimedia.org/r/1345124 (https://phabricator.wikimedia.org/T438915) (owner: 10Tiziano Fogli) [09:02:24] PROBLEM - PyBal IPVS diff check on lvs2013 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [09:03:33] yep this is me --^ [09:03:38] PROBLEM - PyBal connections to etcd on lvs2013 is CRITICAL: CRITICAL: 79 connections established with conf2004.codfw.wmnet:4001 (min=80) https://wikitech.wikimedia.org/wiki/PyBal [09:03:41] (03CR) 10Giuseppe Lavagetto: [C:03+2] kubernetes::node: add gvisor label based on hardware virtualization [puppet] - 10https://gerrit.wikimedia.org/r/1342963 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [09:04:18] !log elukey@cumin1004:~$ sudo cumin 'A:lvs-low-traffic-codfw' 'systemctl restart pybal.service' [09:04:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:05:51] (03CR) 10Btullis: [C:03+2] dse-k8s: Remove the reverse DNS delegations for the old Pod range [dns] - 10https://gerrit.wikimedia.org/r/1345974 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [09:05:53] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, September 30 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#de" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345584 (https://phabricator.wikimedia.org/T439444) (owner: 10Jgiannelos) [09:06:17] !log btullis@dns1004 START - running authdns-update [09:06:58] (03PS10) 10Arnaudb: mesh: Copy mesh.configuration 1.16.0 to 1.16.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338959 (https://phabricator.wikimedia.org/T436657) [09:06:58] (03PS16) 10Arnaudb: mesh: add opt-in websocket support in configuration 1.16.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338751 (https://phabricator.wikimedia.org/T436657) [09:07:18] (03CR) 10Tiziano Fogli: [C:03+2] aptrepo/updates: remove upper bound on Grafana version [puppet] - 10https://gerrit.wikimedia.org/r/1345569 (https://phabricator.wikimedia.org/T438454) (owner: 10Tiziano Fogli) [09:07:24] RECOVERY - PyBal IPVS diff check on lvs2013 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [09:08:22] (03CR) 10Volans: "[this is not a review] I just did a quick pass for general patterns. Left some comments inline." [cookbooks] - 10https://gerrit.wikimedia.org/r/1344697 (https://phabricator.wikimedia.org/T439025) (owner: 10Arnaudb) [09:08:38] RECOVERY - PyBal connections to etcd on lvs2013 is OK: OK: 80 connections established with conf2004.codfw.wmnet:4001 (min=80) https://wikitech.wikimedia.org/wiki/PyBal [09:08:46] !log btullis@dns1004 END - running authdns-update [09:09:44] (03CR) 10Btullis: [C:03+1] "Great, thanks." [puppet] - 10https://gerrit.wikimedia.org/r/1338126 (owner: 10Muehlenhoff) [09:10:02] (03PS5) 10A-pizzata: sqoop_mediawiki: Add weekly change_tag sqoop [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [09:11:04] (03PS4) 10A-pizzata: sqoop_mediawiki: Add weekly full snapshot of the actor table [puppet] - 10https://gerrit.wikimedia.org/r/1343954 (https://phabricator.wikimedia.org/T437961) [09:12:48] FIRING: PuppetFailure: Puppet has failed on netflow1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:14:48] FIRING: PuppetFailure: Puppet has failed on cumin2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:16:28] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip_encapsulation for mathoid, mwdebug* in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1345498 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [09:16:54] (03CR) 10A-pizzata: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [09:17:00] (03CR) 10A-pizzata: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343954 (https://phabricator.wikimedia.org/T437961) (owner: 10A-pizzata) [09:17:48] FIRING: [4x] PuppetFailure: Puppet has failed on netflow1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:18:06] (03CR) 10Joal: [C:03+1] "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [09:18:20] (03CR) 10Joal: [C:03+1] sqoop_mediawiki: Add weekly change_tag sqoop (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1345214 (https://phabricator.wikimedia.org/T437961) (owner: 10Aqu) [09:19:24] 06SRE, 06Infrastructure-Foundations: Unresolved failures from url-downloader - https://phabricator.wikimedia.org/T439560#12378360 (10cmooney) With fresh eyes the logs are there I was filtering incorrectly. I can see the connection at 12:19 on Sept 24th was rejected with a 403 code by the proxy: ` { "http.re... [09:20:58] (03CR) 10Joal: [C:03+1] "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/1343954 (https://phabricator.wikimedia.org/T437961) (owner: 10A-pizzata) [09:21:52] (03PS1) 10Elukey: Add monitors section to the pki LVS config [puppet] - 10https://gerrit.wikimedia.org/r/1346462 [09:22:48] FIRING: [7x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:23:08] (03PS5) 10Blake: bird: add a new prometheus-exporter for bird. [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1343943 (https://phabricator.wikimedia.org/T423851) [09:23:31] (03CR) 10Blake: bird: add a new prometheus-exporter for bird. (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1343943 (https://phabricator.wikimedia.org/T423851) (owner: 10Blake) [09:23:49] (03PS1) 10Btullis: kubernetes: Include the lvm class on all dse-k8s workers [puppet] - 10https://gerrit.wikimedia.org/r/1346463 (https://phabricator.wikimedia.org/T438058) [09:23:51] (03PS1) 10Btullis: dse-k8s-eqiad: Add dse-k8s-worker104[2-4] with TopoLVM storage [puppet] - 10https://gerrit.wikimedia.org/r/1346464 (https://phabricator.wikimedia.org/T438058) [09:25:14] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346463 (https://phabricator.wikimedia.org/T438058) (owner: 10Btullis) [09:26:27] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346464 (https://phabricator.wikimedia.org/T438058) (owner: 10Btullis) [09:27:11] (03CR) 10Elukey: [C:03+2] Add monitors section to the pki LVS config [puppet] - 10https://gerrit.wikimedia.org/r/1346462 (owner: 10Elukey) [09:27:44] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346464 (https://phabricator.wikimedia.org/T438058) (owner: 10Btullis) [09:27:48] FIRING: [9x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:28:11] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-codfw@codfw [09:32:48] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [09:32:48] FIRING: [11x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:32:53] (03PS1) 10Gkyziridis: ml-services: move llm-qwen3-14b to ml-serve1015 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346469 (https://phabricator.wikimedia.org/T439555) [09:33:46] !log jelto@cumin1004 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [09:33:46] !log jelto@cumin1004 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-codfw@codfw [09:34:46] (03PS2) 10Jelto: service::catalog: Set ipip_encapsulation for mathoid, mwdebug* in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1345499 (https://phabricator.wikimedia.org/T420436) [09:34:48] FIRING: [2x] PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:37:36] (03CR) 10Jelto: [C:03+2] service::catalog: Set ipip_encapsulation for mathoid, mwdebug* in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1345499 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [09:37:48] FIRING: [13x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:38:47] (03CR) 10Giuseppe Lavagetto: [C:03+2] runtimeClasses: add gVisor-kvm RuntimeClass [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342964 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [09:38:50] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad [09:39:58] !log jelto@cumin1004 END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for alias: wikikube-worker-eqiad@eqiad [09:42:52] jelto: o/ I am doing a LVS deployment for pki, so if you have restarted pybal you picked up my changes too :( [09:45:13] (03PS1) 10JMeybohm: Explicit restart config in systemd units [debs/kubernetes] (v1.34) - 10https://gerrit.wikimedia.org/r/1346480 [09:46:13] elukey: I restarted pyball only in codfw, eqiad failed because of puppet disabled [09:46:45] jelto: ahh yes ok it is me [09:47:11] (03CR) 10Giuseppe Lavagetto: [C:03+1] "It's possible 3 seconds is not enough as a pause, but LGTM for now" [debs/kubernetes] (v1.34) - 10https://gerrit.wikimedia.org/r/1346480 (owner: 10JMeybohm) [09:47:34] elukey: if you run puppet or restart pyball in eqiad, you should see some changes for mathoid, mwdebug and mwdebug next [09:47:53] if that's blocking you I can also revert the change [09:47:55] (03CR) 10Aghirelli: git::clone: add optional sparse-checkout support (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1343101 (owner: 10Aghirelli) [09:48:24] jelto: nono I hope to do it in a bit, and I'll ping you [09:49:01] FIRING: SystemdUnitFailed: wmf_auto_restart_krb5-admin-server.service on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:49:04] ack, thank you :) [09:49:32] (03CR) 10OSleger: [C:03+1] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345584 (https://phabricator.wikimedia.org/T439444) (owner: 10Jgiannelos) [09:49:51] (03Merged) 10jenkins-bot: runtimeClasses: add gVisor-kvm RuntimeClass [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342964 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [09:53:19] (03Abandoned) 10Marco Fossati: Squashed backport of 1345436, 1345890, 1345969 [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1345999 (https://phabricator.wikimedia.org/T439534) (owner: 10Marco Fossati) [09:54:52] (03PS1) 10Elukey: Revert "Move pki.discovery.wmnet to lvs_setup state" [puppet] - 10https://gerrit.wikimedia.org/r/1346491 [09:56:05] (03CR) 10Elukey: [C:03+2] Revert "Move pki.discovery.wmnet to lvs_setup state" [puppet] - 10https://gerrit.wikimedia.org/r/1346491 (owner: 10Elukey) [09:58:24] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346464 (https://phabricator.wikimedia.org/T438058) (owner: 10Btullis) [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T1000) [10:01:07] jelto: puppet unblocked on eqiad lvses, you can re-run the cookbook, sorry for the wait [10:01:34] (I am currently running puppet, wait a min) [10:01:53] all done! [10:01:57] you are free to go [10:02:12] ack thanks! I'll re-run the cookbook now [10:02:12] Sorry for fumbling into your pki work [10:03:07] (03PS1) 10Elukey: prometheus::blackbox_exporter: temporary fix for gnmi checks [puppet] - 10https://gerrit.wikimedia.org/r/1346494 [10:03:11] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.migrate-service-ipip for alias: wikikube-worker-eqiad@eqiad [10:03:21] jelto: my bad, don't worry! [10:03:39] 06SRE, 06Infrastructure-Foundations: Unresolved failures from url-downloader - https://phabricator.wikimedia.org/T439560#12378615 (10cmooney) Ok I see what happened here. The patch I linked in the above message was merged on the 21st, however due to how the network names are mapped it wasn't added to the Squi... [10:03:41] (03PS1) 10Muehlenhoff: Revert LDAP settings for idp-test1005/2005 [puppet] - 10https://gerrit.wikimedia.org/r/1346495 (https://phabricator.wikimedia.org/T331699) [10:04:11] (03CR) 10Cathal Mooney: [C:03+1] "Thanks for working on this!" [puppet] - 10https://gerrit.wikimedia.org/r/1346494 (owner: 10Elukey) [10:04:24] PROBLEM - PyBal IPVS diff check on lvs2013 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [10:04:34] (03CR) 10Elukey: [C:03+2] prometheus::blackbox_exporter: temporary fix for gnmi checks [puppet] - 10https://gerrit.wikimedia.org/r/1346494 (owner: 10Elukey) [10:05:16] (03CR) 10Slyngshede: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1346495 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [10:05:24] !log jelto@cumin1004 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [10:05:42] PROBLEM - PyBal IPVS diff check on lvs2014 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [10:05:44] (03CR) 10Muehlenhoff: [C:03+2] Revert LDAP settings for idp-test1005/2005 [puppet] - 10https://gerrit.wikimedia.org/r/1346495 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [10:06:32] !log jelto@cumin1004 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [10:06:32] !log jelto@cumin1004 END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for alias: wikikube-worker-eqiad@eqiad [10:07:22] (03CR) 10Vgutierrez: LVS IPIP encapsulation: support backend servers using nftables (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) (owner: 10Cathal Mooney) [10:11:01] (03PS1) 10Elukey: Swap pki.discovery with pki1003 temporarily to allow puppet runs [puppet] - 10https://gerrit.wikimedia.org/r/1346497 [10:12:18] (03CR) 10Ayounsi: [C:03+1] Swap pki.discovery with pki1003 temporarily to allow puppet runs [puppet] - 10https://gerrit.wikimedia.org/r/1346497 (owner: 10Elukey) [10:13:50] (03CR) 10CI reject: [V:04-1] Swap pki.discovery with pki1003 temporarily to allow puppet runs [puppet] - 10https://gerrit.wikimedia.org/r/1346497 (owner: 10Elukey) [10:14:39] (03PS1) 10Giuseppe Lavagetto: sre.hosts.provision: add --no-reboot option to skip host reboot [cookbooks] - 10https://gerrit.wikimedia.org/r/1346499 (https://phabricator.wikimedia.org/T438287) [10:14:43] (03PS2) 10Elukey: Swap pki.discovery with pki1003 temporarily to allow puppet runs [puppet] - 10https://gerrit.wikimedia.org/r/1346497 [10:19:58] (03CR) 10Elukey: [C:03+2] Swap pki.discovery with pki1003 temporarily to allow puppet runs [puppet] - 10https://gerrit.wikimedia.org/r/1346497 (owner: 10Elukey) [10:21:37] 06SRE, 06Infrastructure-Foundations: Unresolved failures from url-downloader - https://phabricator.wikimedia.org/T439560#12378680 (10RThomas-WMF) Thanks very much Cathal for the analysis. AFAIK Enterprise is one of the few airflow DAGs (I am aware of 2 at the least) dependent on the urldownloader, Hence, it w... [10:24:48] FIRING: [2x] PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:25:45] FIRING: [5x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [10:26:02] (03PS3) 10Blake: sidecars: do not add a pre-stop sleep if we're a native sidecar [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346502 (https://phabricator.wikimedia.org/T417800) [10:26:25] (03CR) 10JMeybohm: sre.hosts.provision: add --no-reboot option to skip host reboot (032 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1346499 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [10:27:03] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [10:27:11] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1384.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [10:27:48] FIRING: [13x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:28:42] (03CR) 10AikoChou: [C:03+1] "LGTM! Just one small nit" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346469 (https://phabricator.wikimedia.org/T439555) (owner: 10Gkyziridis) [10:30:45] FIRING: [5x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [10:32:00] !log Grafana upgrade from 13.2.0 to 13.2.3 - T438454 [10:32:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:32:48] FIRING: [13x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:33:12] !log oblivian@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on D{wikikube-worker1384.eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [10:33:16] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1384.eqiad.wmnet [10:33:18] (03PS1) 10JavierMonton: stream: mediawiki.job.pageviewTrendingRelativeUpdate [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346516 (https://phabricator.wikimedia.org/T437875) [10:33:48] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1384.eqiad.wmnet [10:34:01] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [10:35:00] FIRING: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker1042 has a BGP session which is not in the 'established' state. [10:35:45] FIRING: [5x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [10:36:48] (03CR) 10Atsuko: [C:03+1] kubernetes: Include the lvm class on all dse-k8s workers [puppet] - 10https://gerrit.wikimedia.org/r/1346463 (https://phabricator.wikimedia.org/T438058) (owner: 10Btullis) [10:37:12] (03CR) 10Clément Goubert: [C:03+2] mediawiki-vhost: Redirect /api/ to /w/rest.php [puppet] - 10https://gerrit.wikimedia.org/r/1329607 (https://phabricator.wikimedia.org/T433547) (owner: 10Clément Goubert) [10:37:15] (03CR) 10Atsuko: [C:03+1] "more k8s!" [puppet] - 10https://gerrit.wikimedia.org/r/1346464 (https://phabricator.wikimedia.org/T438058) (owner: 10Btullis) [10:37:48] FIRING: [13x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:38:28] !log `sudo ipvsadm --delete-service --tcp-service 10.2.1.96:443` on lvs201[34] to remove the pki service [10:38:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:39:24] RECOVERY - PyBal IPVS diff check on lvs2013 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [10:40:23] (03PS10) 10Clément Goubert: mediawiki: Redirect /api/ to /w/rest.php [deployment-charts] - 10https://gerrit.wikimedia.org/r/1330505 (https://phabricator.wikimedia.org/T433547) [10:40:42] RECOVERY - PyBal IPVS diff check on lvs2014 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [10:40:45] RESOLVED: [5x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [10:40:47] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1384.eqiad.wmnet [10:40:48] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1384.eqiad.wmnet [10:40:48] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D{wikikube-worker1384.eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [10:42:48] FIRING: [12x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:45:00] FIRING: [4x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker1042 has a BGP session which is not in the 'established' state. [10:47:48] FIRING: [10x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:48:20] (03CR) 10Clément Goubert: [C:03+2] mediawiki: Redirect /api/ to /w/rest.php [deployment-charts] - 10https://gerrit.wikimedia.org/r/1330505 (https://phabricator.wikimedia.org/T433547) (owner: 10Clément Goubert) [10:50:00] FIRING: [6x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker1042 has a BGP session which is not in the 'established' state. [10:53:01] (03Merged) 10jenkins-bot: mediawiki: Redirect /api/ to /w/rest.php [deployment-charts] - 10https://gerrit.wikimedia.org/r/1330505 (https://phabricator.wikimedia.org/T433547) (owner: 10Clément Goubert) [10:54:07] (03PS2) 10Slyngshede: P:IDP Tomcat 11 for CAS 8.X [puppet] - 10https://gerrit.wikimedia.org/r/1345921 (https://phabricator.wikimedia.org/T439509) [10:54:07] (03PS1) 10Slyngshede: P:idp Java 25 and Tomcat 11 [puppet] - 10https://gerrit.wikimedia.org/r/1346531 (https://phabricator.wikimedia.org/T439509) [10:54:57] (03PS2) 10Slyngshede: P:idp Java 25 and Tomcat 11 [puppet] - 10https://gerrit.wikimedia.org/r/1346531 (https://phabricator.wikimedia.org/T439509) [10:55:09] (03CR) 10Btullis: [C:03+2] kubernetes: Include the lvm class on all dse-k8s workers [puppet] - 10https://gerrit.wikimedia.org/r/1346463 (https://phabricator.wikimedia.org/T438058) (owner: 10Btullis) [10:56:21] !log cgoubert@deploy1003 Started scap sync-world: 1330505: mediawiki: Redirect /api/ to /w/rest.php | T433547 [10:56:24] T433547: [SPIKE] Validate index.php Apache Behavior for canonical API Routing - https://phabricator.wikimedia.org/T433547 [10:56:28] (03CR) 10Slyngshede: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346531 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [10:57:48] RESOLVED: [3x] PuppetFailure: Puppet has failed on netflow2004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:59:34] !log cgoubert@deploy1003 cgoubert: 1330505: mediawiki: Redirect /api/ to /w/rest.php | T433547 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:00:05] mvolz: Your horoscope predicts another Services – Citoid / Zotero deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T1100). [11:00:14] (03PS3) 10Slyngshede: P:idp Java 25 and Tomcat 11 [puppet] - 10https://gerrit.wikimedia.org/r/1346531 (https://phabricator.wikimedia.org/T439509) [11:00:17] (03CR) 10Slyngshede: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346531 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [11:01:45] !log cgoubert@deploy1003 cgoubert: Continuing with deployment [11:02:46] (03PS2) 10Gkyziridis: ml-services: move llm-qwen3-14b to ml-serve1015 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346469 (https://phabricator.wikimedia.org/T439555) [11:03:18] (03CR) 10Gkyziridis: ml-services: move llm-qwen3-14b to ml-serve1015 (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346469 (https://phabricator.wikimedia.org/T439555) (owner: 10Gkyziridis) [11:06:18] I don't need the window today FYI [11:07:08] !log cgoubert@deploy1003 Finished scap sync-world: 1330505: mediawiki: Redirect /api/ to /w/rest.php | T433547 (duration: 11m 12s) [11:07:11] T433547: [SPIKE] Validate index.php Apache Behavior for canonical API Routing - https://phabricator.wikimedia.org/T433547 [11:10:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1132:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1132 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [11:12:17] (03PS4) 10Slyngshede: P:idp Java 25 and Tomcat 11 [puppet] - 10https://gerrit.wikimedia.org/r/1346531 (https://phabricator.wikimedia.org/T439509) [11:12:28] (03PS4) 10Blake: sidecars: remove pre-stop sleep, stop envoy last [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346502 (https://phabricator.wikimedia.org/T417800) [11:12:37] (03CR) 10Slyngshede: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346531 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [11:12:41] (03CR) 10CI reject: [V:04-1] sidecars: remove pre-stop sleep, stop envoy last [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346502 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [11:14:04] (03PS5) 10Blake: sidecars: remove pre-stop sleep, stop envoy last [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346502 (https://phabricator.wikimedia.org/T417800) [11:14:48] RESOLVED: PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [11:15:03] (03PS1) 10Zabe: Add script to drop duplicate oldimage/filerevision rows [extensions/WikimediaMaintenance] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346537 (https://phabricator.wikimedia.org/T67264) [11:15:22] (03PS1) 10Zabe: Add script to drop duplicate oldimage/filerevision rows [extensions/WikimediaMaintenance] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1346538 (https://phabricator.wikimedia.org/T67264) [11:15:25] (03CR) 10Btullis: [C:03+2] dse-k8s-eqiad: Add dse-k8s-worker104[2-4] with TopoLVM storage [puppet] - 10https://gerrit.wikimedia.org/r/1346464 (https://phabricator.wikimedia.org/T438058) (owner: 10Btullis) [11:15:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1132:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1132 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [11:18:27] (03PS5) 10Slyngshede: P:idp Java 25 and Tomcat 11 [puppet] - 10https://gerrit.wikimedia.org/r/1346531 (https://phabricator.wikimedia.org/T439509) [11:22:44] !log cgoubert@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-jobrunner: apply [11:22:46] !log cgoubert@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: apply [11:22:52] !log cgoubert@deploy1003 helmfile [codfw] START helmfile.d/services/mw-jobrunner: apply [11:22:55] !log cgoubert@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: apply [11:23:35] !log cgoubert@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [11:24:01] !log cgoubert@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-experimental: apply [11:24:03] !log cgoubert@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [11:24:07] !log cgoubert@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [11:24:31] !log cgoubert@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply [11:24:40] FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1132:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1132 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [11:24:40] !log cgoubert@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [11:24:46] !log cgoubert@deploy1003 helmfile [codfw] START helmfile.d/services/mw-experimental: apply [11:25:00] FIRING: [6x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker1042 has a BGP session which is not in the 'established' state. [11:25:17] !log cgoubert@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply [11:27:00] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [11:29:48] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [11:30:00] FIRING: [6x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker1042 has a BGP session which is not in the 'established' state. [11:31:09] (03PS1) 10Kosta Harlan: CheckUser: Enable ClientHintsInEditRequest on group1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346541 (https://phabricator.wikimedia.org/T401808) [11:31:12] (03PS1) 10Kosta Harlan: CheckUser: Enable ClientHintsInEditRequest on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346542 (https://phabricator.wikimedia.org/T401808) [11:32:32] !log btullis@puppetserver1001 conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet [11:32:42] !log btullis@puppetserver1001 conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1042.eqiad.wmnet [11:32:49] !log btullis@puppetserver1001 conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet [11:32:57] !log btullis@puppetserver1001 conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet [11:33:04] !log btullis@puppetserver1001 conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1044.eqiad.wmnet [11:33:29] !log btullis@puppetserver1001 conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1043.eqiad.wmnet [11:34:10] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1042.eqiad.wmnet [11:34:31] (03PS6) 10Ozge: httpbb(liftwing): Update Lift Wing suites and add a runner [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) [11:36:02] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [11:38:14] (03CR) 10David Caro: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1343544 (https://phabricator.wikimedia.org/T438333) (owner: 10Majavah) [11:39:45] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1042.eqiad.wmnet [11:40:00] RESOLVED: [6x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-worker1042 has a BGP session which is not in the 'established' state. [11:40:14] (03CR) 10Majavah: [C:03+2] hieradata: Add new project-proxy hosts to cache_hosts [puppet] - 10https://gerrit.wikimedia.org/r/1343544 (https://phabricator.wikimedia.org/T438333) (owner: 10Majavah) [11:40:21] (03CR) 10Majavah: [V:03+1 C:03+2] cloudlb: Allow specifying multiple addresses per frontend [puppet] - 10https://gerrit.wikimedia.org/r/1260004 (https://phabricator.wikimedia.org/T420921) (owner: 10Majavah) [11:41:10] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [11:44:40] RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1132:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1132 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [11:45:49] (03CR) 10Volans: transports: disable Tqdm monitor thread (031 comment) [software/cumin] - 10https://gerrit.wikimedia.org/r/1341033 (owner: 10Volans) [11:47:57] (03PS2) 10Clément Goubert: git::clone: add optional sparse-checkout support [puppet] - 10https://gerrit.wikimedia.org/r/1343101 (owner: 10Aghirelli) [11:47:59] (03CR) 10Clément Goubert: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343101 (owner: 10Aghirelli) [11:49:56] (03CR) 10Ozge: "thank you! I added new services to httpbb.pp." [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) (owner: 10Ozge) [11:52:52] 06SRE, 06Collaboration-Services, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07ci-test-error: Builds fail on integration-agent-docker-1086 due to expired snapshot.debian.org Release file - https://phabricator.wikimedia.org/T439529#12379009 (10MoritzMuehlenhoff) >>! In T439529#123... [11:54:03] 06SRE, 06Collaboration-Services, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 07ci-test-error: Builds fail on integration-agent-docker-1086 due to expired snapshot.debian.org Release file - https://phabricator.wikimedia.org/T439529#12379010 (10MoritzMuehlenhoff) >>! In T439529#123... [11:55:02] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1043.eqiad.wmnet [11:55:18] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1044.eqiad.wmnet [11:55:54] (03PS7) 10Ozge: httpbb(liftwing): Update Lift Wing suites and add a runner [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) [11:58:40] (03CR) 10Gkyziridis: [C:03+1] httpbb(liftwing): Update Lift Wing suites and add a runner [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) (owner: 10Ozge) [12:00:36] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1044.eqiad.wmnet [12:02:10] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1043.eqiad.wmnet [12:02:34] (03PS8) 10Ozge: httpbb(liftwing): Update Lift Wing suites and add a runner [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) [12:05:39] (03PS1) 10Tiziano Fogli: Revert "kafka-logging100[78]: disable icinga notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1346552 [12:06:58] (03CR) 10Tiziano Fogli: [C:03+2] Revert "kafka-logging100[78]: disable icinga notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1346552 (owner: 10Tiziano Fogli) [12:07:15] (03PS1) 10Dpogorzelski: wikimedia.org: add liftwing-studio [dns] - 10https://gerrit.wikimedia.org/r/1346553 [12:09:27] (03CR) 10Jelto: [C:03+2] cache-text: set caching for etherpad-next "websockets" [puppet] - 10https://gerrit.wikimedia.org/r/1341894 (https://phabricator.wikimedia.org/T435509) (owner: 10Jelto) [12:11:44] (03CR) 10Ozge: "Hey Tobias and Dawid," [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) (owner: 10Ozge) [12:12:08] !log tappof@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on kafka-logging[1001-1003].eqiad.wmnet with reason: stop service on kafka-logging100[1-3] [12:12:15] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T439559#12379112 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=7b8a0c1c-1367-440b-8201-f5bd9fbc3171) set by tappof@cumin... [12:13:11] (03PS2) 10Dpogorzelski: wikimedia.org: add liftwing-studio [dns] - 10https://gerrit.wikimedia.org/r/1346553 (https://phabricator.wikimedia.org/T439416) [12:13:50] (03PS1) 10Dpogorzelski: idp: accept liftwing-studio's public callback URL [puppet] - 10https://gerrit.wikimedia.org/r/1346556 (https://phabricator.wikimedia.org/T439416) [12:13:52] (03PS1) 10Dpogorzelski: cache: expose liftwing-studio at liftwing-studio.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1346557 (https://phabricator.wikimedia.org/T439416) [12:13:55] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [12:14:03] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1380.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [12:14:11] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [12:14:20] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1381.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [12:14:35] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [12:14:45] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1382.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [12:14:54] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [12:15:03] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1383.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [12:16:08] !log oblivian@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on D{wikikube-worker138[0-3].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [12:16:12] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1380.eqiad.wmnet [12:16:46] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1380.eqiad.wmnet [12:17:45] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T439559#12379128 (10tappof) [12:17:55] (03PS2) 10Dpogorzelski: idp: switch liftwing-studio to liftwing-studio.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1346556 (https://phabricator.wikimedia.org/T439416) [12:17:56] (03PS2) 10Dpogorzelski: cache: expose liftwing-studio at liftwing-studio.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1346557 (https://phabricator.wikimedia.org/T439416) [12:19:14] (03CR) 10AikoChou: [C:03+1] "LGTM!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346469 (https://phabricator.wikimedia.org/T439555) (owner: 10Gkyziridis) [12:20:02] 06SRE, 06Infrastructure-Foundations: Unresolved failures from url-downloader - https://phabricator.wikimedia.org/T439560#12379129 (10cmooney) >>! In T439560#12378680, @RThomas-WMF wrote: > 2. Do you know the timeline to address this open request: T429175? The work there is basically done, the service is now b... [12:20:26] (03PS1) 10Dpogorzelski: dse-k8s: add liftwing-studio.wikimedia.org to the namespace certificate [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346562 (https://phabricator.wikimedia.org/T439416) [12:20:28] (03PS1) 10Dpogorzelski: liftwing-studio: serve on liftwing-studio.wikimedia.org [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346563 (https://phabricator.wikimedia.org/T439416) [12:20:39] (03CR) 10JMeybohm: "My interpretation was that this will lead to 5 restarts, so 5*" [debs/kubernetes] (v1.34) - 10https://gerrit.wikimedia.org/r/1346480 (owner: 10JMeybohm) [12:21:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:21:41] (03CR) 10Gkyziridis: [C:03+2] ml-services: move llm-qwen3-14b to ml-serve1015 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346469 (https://phabricator.wikimedia.org/T439555) (owner: 10Gkyziridis) [12:23:45] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1380.eqiad.wmnet [12:23:46] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1380.eqiad.wmnet [12:23:51] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1381.eqiad.wmnet [12:24:01] (03PS1) 10Slyngshede: CAS 8.0.2 [software/cas-overlay-template] - 10https://gerrit.wikimedia.org/r/1346564 [12:24:01] FIRING: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:24:03] (03Merged) 10jenkins-bot: ml-services: move llm-qwen3-14b to ml-serve1015 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346469 (https://phabricator.wikimedia.org/T439555) (owner: 10Gkyziridis) [12:24:22] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1381.eqiad.wmnet [12:24:55] (03PS3) 10Clément Goubert: git::clone: add optional sparse-checkout support [puppet] - 10https://gerrit.wikimedia.org/r/1343101 (owner: 10Aghirelli) [12:24:57] (03CR) 10Clément Goubert: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343101 (owner: 10Aghirelli) [12:25:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328191 (owner: 10SomeRandomDeveloper) [12:26:10] (03PS5) 10Tiziano Fogli: kafka-logging: remove kafka-logging100[12] [puppet] - 10https://gerrit.wikimedia.org/r/1342557 (https://phabricator.wikimedia.org/T432444) [12:26:29] (03Merged) 10jenkins-bot: Profiler: Fix excimer component regex skipping the last frame in each stack [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1328191 (owner: 10SomeRandomDeveloper) [12:26:49] !log krinkle@deploy1003 Started scap sync-world: Backport for [[gerrit:1328191|Profiler: Fix excimer component regex skipping the last frame in each stack]] [12:27:55] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic: Track down memory leak with URL downloaders in Trixie - https://phabricator.wikimedia.org/T439712 (10MoritzMuehlenhoff) 03NEW [12:28:06] (03CR) 10Clément Goubert: [C:04-2] "This doesn't need to be done in ATS lua. Because we can't easily rebuild restbase, we should do it in haproxy where it would be a couple o" [puppet] - 10https://gerrit.wikimedia.org/r/1339947 (https://phabricator.wikimedia.org/T431372) (owner: 10Aaron Schulz) [12:28:11] 06SRE, 06Infrastructure-Foundations, 06Traffic: Track down memory leak with URL downloaders in Trixie - https://phabricator.wikimedia.org/T439712#12379180 (10MoritzMuehlenhoff) p:05Triage→03Medium [12:29:01] RESOLVED: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-c2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:29:44] jouncebot: nowandnext [12:29:45] No deployments scheduled for the next 0 hour(s) and 30 minute(s) [12:29:45] In 0 hour(s) and 30 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T1300) [12:30:41] (03PS6) 10Seanleong-wmde: Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd [mediawiki-config] - 10https://gerrit.wikimedia.org/r/697110 (https://phabricator.wikimedia.org/T222516) (owner: 10Luca Mauri) [12:30:58] !log krinkle@deploy1003 somerandomdeveloper, krinkle: Backport for [[gerrit:1328191|Profiler: Fix excimer component regex skipping the last frame in each stack]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:31:22] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1381.eqiad.wmnet [12:31:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1381.eqiad.wmnet [12:31:29] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1382.eqiad.wmnet [12:32:01] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1382.eqiad.wmnet [12:32:30] !log krinkle@deploy1003 somerandomdeveloper, krinkle: Continuing with deployment [12:33:08] !log fetching new haproxy packages on reprepro (3.2.23-1~bpo13+1 -> 3.2.25-1~bpo13+1) - T439667 [12:33:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:33:11] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [12:34:46] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713 (10MoritzMuehlenhoff) 03NEW [12:36:37] (03CR) 10Muehlenhoff: [C:03+2] Add cookbook to roll-restart/reboot URL downloaders [cookbooks] - 10https://gerrit.wikimedia.org/r/1329523 (https://phabricator.wikimedia.org/T429175) (owner: 10Muehlenhoff) [12:37:46] !log krinkle@deploy1003 Finished scap sync-world: Backport for [[gerrit:1328191|Profiler: Fix excimer component regex skipping the last frame in each stack]] (duration: 10m 56s) [12:39:00] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1382.eqiad.wmnet [12:39:01] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1382.eqiad.wmnet [12:39:07] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1383.eqiad.wmnet [12:39:39] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1383.eqiad.wmnet [12:41:45] !log dkertesz@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp7001.*} and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [12:41:48] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [12:42:18] Krinkle: if you're done with your deployment, can I take over and go ahead with mine before the UTC afternoon backport window takes place? [12:42:22] yes [12:42:24] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [12:42:25] thanks [12:42:37] (03CR) 10SimmeD: [C:03+1] change logo at zh-classical wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1276284 (https://phabricator.wikimedia.org/T424128) (owner: 10WAN233) [12:42:41] (03CR) 10Tiziano Fogli: [C:03+2] kafka-logging: remove kafka-logging100[12] [puppet] - 10https://gerrit.wikimedia.org/r/1342557 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [12:43:03] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [12:43:28] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [12:43:56] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [12:44:27] (03CR) 10Aqu: [C:03+1] stream: mediawiki.job.pageviewTrendingRelativeUpdate [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346516 (https://phabricator.wikimedia.org/T437875) (owner: 10JavierMonton) [12:44:54] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [12:45:48] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [12:46:49] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1383.eqiad.wmnet [12:46:50] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1383.eqiad.wmnet [12:46:51] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D{wikikube-worker138[0-3].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [12:47:03] !log dkertesz@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp7001.*} and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [12:47:05] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [12:48:32] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [12:49:09] (03CR) 10Awight: [C:03+1] Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd [mediawiki-config] - 10https://gerrit.wikimedia.org/r/697110 (https://phabricator.wikimedia.org/T222516) (owner: 10Luca Mauri) [12:50:15] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, September 30 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#de" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/697110 (https://phabricator.wikimedia.org/T222516) (owner: 10Luca Mauri) [12:53:04] !log tappof@deploy1003 helmfile [eqiad] START helmfile.d/admin 'sync'. [12:53:19] !log tappof@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'sync'. [12:53:39] !log tappof@deploy1003 helmfile [codfw] START helmfile.d/admin 'sync'. [12:54:15] !log tappof@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'sync'. [12:54:32] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [12:54:33] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [12:55:23] (03PS6) 10Arnaudb: gerrit: add failover, DNS emergency and demotion cookbooks [cookbooks] - 10https://gerrit.wikimedia.org/r/1344697 (https://phabricator.wikimedia.org/T439025) [12:55:23] (03PS6) 10Arnaudb: gerrit: read the replica flag from the loaded unit [cookbooks] - 10https://gerrit.wikimedia.org/r/1344698 (https://phabricator.wikimedia.org/T439025) [12:55:23] (03PS6) 10Arnaudb: gerrit: reuse the shared helpers in sibling cookbooks [cookbooks] - 10https://gerrit.wikimedia.org/r/1344699 (https://phabricator.wikimedia.org/T439025) [12:56:22] 06SRE-OnFire, 06MediaWiki-Engineering, 06ServiceOps, 06SRE Observability, 07Sustainability (Incident Followup): Reduce the amount of messages sent through channel:Memcached during failures - https://phabricator.wikimedia.org/T390529#12379393 (10MLechvien-WMF) @herron I linked see you linked {T438375}, wi... [12:56:41] !log tappof@deploy1003 Started scap sync-world: T432444 - remove kafka-logging100[12] [12:56:45] T432444: Provision kafka-logging100[6-8] and kafka-logging200[6-8] - https://phabricator.wikimedia.org/T432444 [12:56:54] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [12:58:05] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346586 [12:58:51] (03PS2) 10Slyngshede: CAS 8.0.2 [software/cas-overlay-template] - 10https://gerrit.wikimedia.org/r/1346564 (https://phabricator.wikimedia.org/T439509) [12:59:38] PROBLEM - Check unit status of statograph_post on alert1002 is CRITICAL: CRITICAL: Status of the systemd unit statograph_post https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [13:00:04] urbanecm and TheresNoTime: #bothumor My software never has bugs. It just develops random features. Rise for UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T1300). [13:00:04] MichaelG_WMF, nemo-yiannis, and seanleong-wmde: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:47] o/ [13:00:53] (03CR) 10Ottomata: [C:03+1] "Nice!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346516 (https://phabricator.wikimedia.org/T437875) (owner: 10JavierMonton) [13:00:54] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - thanos-query_443: Servers titan1001.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [13:01:10] * MichaelG_WMF is here [13:01:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:01:30] I'm not able to deploy myself [13:01:40] Will start with yours :) [13:01:54] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [13:02:01] TheresNoTime: Thank you! <3 [13:02:05] TheresNoTime: I'm currently deploying an MW infra patch. It should be finished in a few seconds.. [13:02:06] (03CR) 10TrainBranchBot: [C:03+2] "Approved by samtar@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346060 (https://phabricator.wikimedia.org/T438908) (owner: 10Michael Große) [13:02:25] (03PS2) 10Btullis: dumps: web: Add a brownout for v1 meta-current and meta-history [puppet] - 10https://gerrit.wikimedia.org/r/1346007 (https://phabricator.wikimedia.org/T437461) [13:02:39] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [13:02:41] tappof: ah damn, that'll ^ take a few minutes to merge, should be okay? [13:02:41] (03CR) 10Elukey: [C:03+1] "I don't have a lot of context with the code but the refactoring looks good, I left a couple of comments that you can skip :)" [cookbooks] - 10https://gerrit.wikimedia.org/r/1345990 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [13:03:42] (03Merged) 10jenkins-bot: fix(AccountSetup): add missing i18n key to extension.json [extensions/GrowthExperiments] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346060 (https://phabricator.wikimedia.org/T438908) (owner: 10Michael Große) [13:04:10] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [13:04:19] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2370.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [13:04:25] (okay, hit the deployment lock, waiting) [13:04:29] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [13:04:39] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2371.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [13:04:48] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [13:04:55] TheresNoTime almost finished [13:04:57] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2372.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [13:05:01] np! :) [13:05:20] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [13:05:24] Hi o7 [13:05:29] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2373.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [13:05:47] !log tappof@deploy1003 Finished scap sync-world: T432444 - remove kafka-logging100[12] (duration: 10m 41s) [13:05:51] TheresNoTime: Aaaand it's done! [13:05:52] T432444: Provision kafka-logging100[6-8] and kafka-logging200[6-8] - https://phabricator.wikimedia.org/T432444 [13:06:03] !log samtar@deploy1003 Started scap sync-world: Backport for [[gerrit:1346060|fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] [13:06:06] woo :) [13:06:10] T438908: [testwiki wmf.21] Homepage interest selector - when no matching articiles found, infinite loading state is displayed - https://phabricator.wikimedia.org/T438908 [13:06:10] T436872: Account Setup experiment: test end-to-end flow and verify translations on pilot wikis (DE1.3.1) - https://phabricator.wikimedia.org/T436872 [13:06:13] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12379456 (10MoritzMuehlenhoff) I created two followups for the open followup work: - https://phabricator.wikimedia.... [13:06:14] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [13:06:24] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2374.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [13:07:14] !log oblivian@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on D{wikikube-worker237[0-5].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [13:07:17] is there a ongoing deployment right now? [13:07:18] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2370.codfw.wmnet [13:07:45] seanleong-wmde: yup, just doing MichaelG_WMF's patch, are you able to self-deploy after? [13:07:53] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2370.codfw.wmnet [13:08:22] ahh yea sure, I can self-deploy. Thanks! [13:08:29] will ping you when done :) [13:08:35] okk! :) [13:09:10] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06tools-platform-team, 06Traffic: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12379479 (10fnegri) [13:09:17] nemo-yiannis: are you able to self-deploy or do you need a deployer btw? [13:09:38] RECOVERY - Check unit status of statograph_post on alert1002 is OK: OK: Status of the systemd unit statograph_post https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [13:09:48] i can deploy [13:09:58] should i go ahead ? [13:10:04] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06tools-platform-team, 06Traffic: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12379487 (10MoritzMuehlenhoff) [13:10:13] nemo-yiannis: doing a deploy rn, but you're next in the list [13:10:15] !log samtar@deploy1003 migr, samtar: Backport for [[gerrit:1346060|fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:10:16] ok [13:10:28] MichaelG_WMF: ready for testing ^ [13:10:37] !log dkertesz@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P{cp7001.*} and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [13:10:41] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [13:10:53] ready! [13:10:58] testing.. [13:11:31] @nemo-yiannis, I'll go after you [13:11:54] ok [13:12:48] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [13:12:56] TheresNoTime: works as expected! Ready to move forward from my side. [13:13:10] !log samtar@deploy1003 migr, samtar: Continuing with deployment [13:13:16] ack :) [13:14:39] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346593 [13:15:01] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2370.codfw.wmnet [13:15:02] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T439559#12379526 (10tappof) [13:15:02] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2370.codfw.wmnet [13:15:09] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2371.codfw.wmnet [13:15:44] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2371.codfw.wmnet [13:16:58] (03PS1) 10Kevin Bazira: ml-services: update qwen3-14b-fp8 LLMInferenceService in LiftWing staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346594 (https://phabricator.wikimedia.org/T439659) [13:18:27] !log samtar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1346060|fix(AccountSetup): add missing i18n key to extension.json (T438908 T436872)]] (duration: 12m 23s) [13:18:31] T438908: [testwiki wmf.21] Homepage interest selector - when no matching articiles found, infinite loading state is displayed - https://phabricator.wikimedia.org/T438908 [13:18:32] T436872: Account Setup experiment: test end-to-end flow and verify translations on pilot wikis (DE1.3.1) - https://phabricator.wikimedia.org/T436872 [13:18:39] (03PS1) 10Btullis: dse-k8s: remove the old POD IPv4 block 10.67.24.0/21 from prefix lists [homer/public] - 10https://gerrit.wikimedia.org/r/1346596 (https://phabricator.wikimedia.org/T430658) [13:18:39] Done :) all yours nemo-yiannis [13:19:04] ok [13:19:20] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jgiannelos@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345584 (https://phabricator.wikimedia.org/T439444) (owner: 10Jgiannelos) [13:20:04] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346007 (https://phabricator.wikimedia.org/T437461) (owner: 10Btullis) [13:20:07] 06SRE, 06Infrastructure-Foundations, 06ServiceOps, 06Traffic, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12379542 (10ssingh) 05Open→03Resolved >>! In T429175#12379456, @MoritzMuehlenhoff wrote: > I created two fol... [13:20:19] (03CR) 10Kevin Bazira: [C:03+2] ml-services: update qwen3-14b-fp8 LLMInferenceService in LiftWing staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346594 (https://phabricator.wikimedia.org/T439659) (owner: 10Kevin Bazira) [13:20:42] !log tappof@cumin1004 START - Cookbook sre.hosts.decommission for hosts kafka-logging1003.eqiad.wmnet [13:21:15] TheresNoTime: thank you for the deployment 🙏 [13:21:24] (03Merged) 10jenkins-bot: prv: Enable parsoid rendering for 5 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1345584 (https://phabricator.wikimedia.org/T439444) (owner: 10Jgiannelos) [13:21:27] you're welcome ^^ [13:21:43] !log jgiannelos@deploy1003 Started scap sync-world: Backport for [[gerrit:1345584|prv: Enable parsoid rendering for 5 wikis (T439444)]] [13:21:46] T439444: Rollout parsoid read views on wikisource - week of 25 Sep - https://phabricator.wikimedia.org/T439444 [13:22:43] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2371.codfw.wmnet [13:22:45] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2371.codfw.wmnet [13:22:45] (03Merged) 10jenkins-bot: ml-services: update qwen3-14b-fp8 LLMInferenceService in LiftWing staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346594 (https://phabricator.wikimedia.org/T439659) (owner: 10Kevin Bazira) [13:22:53] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2372.codfw.wmnet [13:23:27] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2372.codfw.wmnet [13:23:32] !log andrew@cumin2003 START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm [13:23:50] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [13:23:54] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [13:24:47] 06SRE, 06Collaboration-Services, 06Infrastructure-Foundations, 06ServiceOps, and 2 others: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12379559 (10Jelto) Adding #collaboration-services tag for wmf-navigator [13:25:56] !log jgiannelos@deploy1003 jgiannelos: Backport for [[gerrit:1345584|prv: Enable parsoid rendering for 5 wikis (T439444)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:26:42] !log tappof@cumin1004 START - Cookbook sre.dns.netbox [13:27:40] !log jgiannelos@deploy1003 jgiannelos: Continuing with deployment [13:30:29] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2372.codfw.wmnet [13:30:30] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2372.codfw.wmnet [13:30:36] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2373.codfw.wmnet [13:31:10] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2373.codfw.wmnet [13:31:49] !log tappof@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" [13:32:23] !log tappof@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kafka-logging1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - tappof@cumin1004" [13:32:23] !log tappof@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:32:24] !log tappof@cumin1004 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kafka-logging1003.eqiad.wmnet [13:32:30] (03PS1) 10Majavah: hieradata: cloudlb: Bind all codfw1dev services to specific IPs [puppet] - 10https://gerrit.wikimedia.org/r/1346602 (https://phabricator.wikimedia.org/T420921) [13:32:30] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T439559#12379613 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by tappof@cumin1004 for hosts: `kafka-logging1003.eqiad.wmne... [13:32:56] !log jgiannelos@deploy1003 Finished scap sync-world: Backport for [[gerrit:1345584|prv: Enable parsoid rendering for 5 wikis (T439444)]] (duration: 11m 13s) [13:32:59] T439444: Rollout parsoid read views on wikisource - week of 25 Sep - https://phabricator.wikimedia.org/T439444 [13:33:21] jouncebot: nowandnext [13:33:21] For the next 0 hour(s) and 26 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T1300) [13:33:21] In 0 hour(s) and 26 minute(s): Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T1400) [13:33:29] (03CR) 10CI reject: [V:04-1] hieradata: cloudlb: Bind all codfw1dev services to specific IPs [puppet] - 10https://gerrit.wikimedia.org/r/1346602 (https://phabricator.wikimedia.org/T420921) (owner: 10Majavah) [13:34:00] (03PS2) 10Majavah: hieradata: cloudlb: Bind all codfw1dev services to specific IPs [puppet] - 10https://gerrit.wikimedia.org/r/1346602 (https://phabricator.wikimedia.org/T420921) [13:34:41] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9488/co" [puppet] - 10https://gerrit.wikimedia.org/r/1346602 (https://phabricator.wikimedia.org/T420921) (owner: 10Majavah) [13:35:20] seanleong-wmde: Is your patch currently being deployed? [13:35:39] not yet, I am waiting for nemo-yiannis [13:35:40] (03PS1) 10Majavah: cloudlb: Update LABS_NETWORKS to CLOUD_NETWORKS [puppet] - 10https://gerrit.wikimedia.org/r/1346604 [13:35:49] can I continue already? [13:36:12] not sure, but nemo-yiannis sync finished a few min ago [13:36:23] ok done [13:36:40] thanks! I'll deploy mine now [13:36:41] (03PS2) 10Majavah: cloudlb: Update LABS_NETWORKS to CLOUD_NETWORKS [puppet] - 10https://gerrit.wikimedia.org/r/1346604 [13:37:05] seanleong-wmde: could you ping me once you are done? [13:37:37] (03CR) 10TrainBranchBot: [C:03+2] "Approved by seanleong-wmde@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/697110 (https://phabricator.wikimedia.org/T222516) (owner: 10Luca Mauri) [13:37:52] zabe: yup! [13:38:12] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2373.codfw.wmnet [13:38:13] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2373.codfw.wmnet [13:38:18] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2374.codfw.wmnet [13:38:53] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2374.codfw.wmnet [13:39:25] (03Merged) 10jenkins-bot: Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd [mediawiki-config] - 10https://gerrit.wikimedia.org/r/697110 (https://phabricator.wikimedia.org/T222516) (owner: 10Luca Mauri) [13:39:46] !log seanleong-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:697110|Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] [13:39:52] T222516: Adapt http://www.mediawiki.org/xml/sitelist-1.0/ for the new XSD file - https://phabricator.wikimedia.org/T222516 [13:39:53] T439093: Fix language code for special wikis in SiteMatrixLookup - https://phabricator.wikimedia.org/T439093 [13:41:45] !log andrew@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm [13:43:44] !log seanleong-wmde@deploy1003 seanleong-wmde, lucamauri: Backport for [[gerrit:697110|Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:46:13] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2374.codfw.wmnet [13:46:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2374.codfw.wmnet [13:46:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D{wikikube-worker237[0-5].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [13:46:33] !log andrew@cumin2003 START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm [13:46:34] (03PS1) 10Elukey: Remove PKI mTLS settings for cloud [puppet] - 10https://gerrit.wikimedia.org/r/1346609 (https://phabricator.wikimedia.org/T436809) [13:46:35] !log seanleong-wmde@deploy1003 seanleong-wmde, lucamauri: Continuing with deployment [13:48:48] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [13:49:02] FIRING: SystemdUnitFailed: wmf_auto_restart_krb5-admin-server.service on krb2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:50:31] (03PS1) 10Awight: Fix xsd xml version [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346611 (https://phabricator.wikimedia.org/T439093) [13:50:50] seanleong-wmde: ^ one more for the window [13:51:39] (03PS2) 10Jforrester: wikifunctions: Log at INFO level in the orchestrator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346099 (https://phabricator.wikimedia.org/T439592) [13:51:39] (03PS2) 10Jforrester: wikifunctions: Drop support for unused php.containers template block [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346098 [13:51:39] (03PS1) 10Jforrester: wikifunctions: Upgrade evaluators from 2026-09-22-135346 to 2026-09-30-134115 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346612 (https://phabricator.wikimedia.org/T433459) [13:51:48] !log seanleong-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:697110|Create new http://www.mediawiki.org/xml/sitelist-1.1/ to reference sitelist-1.1.xsd (T222516 T439093)]] (duration: 12m 01s) [13:51:53] T222516: Adapt http://www.mediawiki.org/xml/sitelist-1.0/ for the new XSD file - https://phabricator.wikimedia.org/T222516 [13:51:53] (03PS1) 10Jforrester: wikifunctions: Upgrade orchestrator from 2026-09-23-042024 to 2026-09-30-134715 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346613 (https://phabricator.wikimedia.org/T404869) [13:51:53] T439093: Fix language code for special wikis in SiteMatrixLookup - https://phabricator.wikimedia.org/T439093 [13:52:58] (03CR) 10TrainBranchBot: [C:03+2] "Approved by seanleong-wmde@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346611 (https://phabricator.wikimedia.org/T439093) (owner: 10Awight) [13:53:09] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [13:54:11] (03Merged) 10jenkins-bot: Fix xsd xml version [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346611 (https://phabricator.wikimedia.org/T439093) (owner: 10Awight) [13:54:32] !log seanleong-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1346611|Fix xsd xml version (T439093)]] [13:54:34] 06SRE, 06Infrastructure-Foundations: Unresolved failures from url-downloader - https://phabricator.wikimedia.org/T439560#12379788 (10RThomas-WMF) Ok thanks for sharing the details. We will follow-up with the on-call expectations separately. [13:55:09] (03CR) 10Majavah: "I've no knowledge of the current setup in the `pki` project so I don't think I can properly review this, although if a similar change was " [puppet] - 10https://gerrit.wikimedia.org/r/1346609 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [13:56:09] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [13:58:09] (03CR) 10Klausman: [C:03+1] httpbb(liftwing): Update Lift Wing suites and add a runner [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) (owner: 10Ozge) [13:58:47] !log seanleong-wmde@deploy1003 awight, seanleong-wmde: Backport for [[gerrit:1346611|Fix xsd xml version (T439093)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:58:51] T439093: Fix language code for special wikis in SiteMatrixLookup - https://phabricator.wikimedia.org/T439093 [14:00:05] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T1400) [14:00:39] !log seanleong-wmde@deploy1003 awight, seanleong-wmde: Continuing with deployment [14:00:39] !log bking@krb2002 disable wmf_auto_restart krb admin timers/services as this host is no longer primary T437937 [14:00:40] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [14:00:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:00:43] T437937: Attempt to use VM as Kerberos primary - https://phabricator.wikimedia.org/T437937 [14:00:58] (03PS2) 10Elukey: Remove PKI mTLS settings for cloud [puppet] - 10https://gerrit.wikimedia.org/r/1346609 (https://phabricator.wikimedia.org/T436809) [14:00:58] (03PS1) 10Elukey: profile::pki::multirooca: simplify vhost config [puppet] - 10https://gerrit.wikimedia.org/r/1346615 (https://phabricator.wikimedia.org/T436809) [14:01:00] (03PS1) 10Elukey: Disable pki vhost on port 8443 [puppet] - 10https://gerrit.wikimedia.org/r/1346616 (https://phabricator.wikimedia.org/T436809) [14:01:02] (03PS1) 10Elukey: profile::pki::multirootca: remove support for port 8443 [puppet] - 10https://gerrit.wikimedia.org/r/1346617 (https://phabricator.wikimedia.org/T436809) [14:01:43] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [14:01:55] (03PS1) 10Jforrester: [DNM] mlwiki: Enable Abstract Wikimedia client mode [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346619 [14:02:14] (03CR) 10Klausman: [C:03+2] httpbb(liftwing): Update Lift Wing suites and add a runner [puppet] - 10https://gerrit.wikimedia.org/r/1345534 (https://phabricator.wikimedia.org/T439385) (owner: 10Ozge) [14:02:26] (03CR) 10Ecarg: [C:03+2] wikifunctions: Upgrade evaluators from 2026-09-22-135346 to 2026-09-30-134115 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346612 (https://phabricator.wikimedia.org/T433459) (owner: 10Jforrester) [14:02:31] (03PS2) 10Jforrester: mlwiki: Enable Abstract Wikimedia client mode [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346619 [14:03:13] (03CR) 10CI reject: [V:04-1] profile::pki::multirootca: remove support for port 8443 [puppet] - 10https://gerrit.wikimedia.org/r/1346617 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [14:03:50] (03PS1) 10Gkyziridis: ml-services: move llm-qwen36-27b to ml-serve1013/1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346621 (https://phabricator.wikimedia.org/T439555) [14:04:08] !log dkertesz@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P{cp7001.*} and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [14:04:12] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [14:04:48] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [14:04:51] !log mwscript-k8s --comment="T358266" --local_dblist=<(expanddblist all | grep -v commons) -- extensions/TimedMediaHandler/maintenance/requeueTranscodes.php --key '360p.mpeg4.mov' --throttle --video --missing (T358266) [14:04:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:04:54] T358266: TimedMediaHandler: Improve video quality on older iPhones (MPEG-4 Visual) - https://phabricator.wikimedia.org/T358266 [14:05:08] (03Merged) 10jenkins-bot: wikifunctions: Upgrade evaluators from 2026-09-22-135346 to 2026-09-30-134115 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346612 (https://phabricator.wikimedia.org/T433459) (owner: 10Jforrester) [14:05:12] 06SRE, 06Infrastructure-Foundations, 06Traffic, 07LDAP: Add a new LVS ldap-ro-next service - https://phabricator.wikimedia.org/T439730 (10MoritzMuehlenhoff) 03NEW [14:05:23] (03PS2) 10Elukey: profile::pki::multirootca: remove support for port 8443 [puppet] - 10https://gerrit.wikimedia.org/r/1346617 (https://phabricator.wikimedia.org/T436809) [14:05:34] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [14:05:47] andrew@cumin2003 reimage (PID 2000751) is awaiting input [14:06:05] !log seanleong-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1346611|Fix xsd xml version (T439093)]] (duration: 11m 33s) [14:06:08] T439093: Fix language code for special wikis in SiteMatrixLookup - https://phabricator.wikimedia.org/T439093 [14:06:12] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [14:06:29] zabe: Done o7, sorry for the delay due to an additional emergency patch. [14:07:13] !log ecarg@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:08:02] !log ecarg@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:09:16] (03CR) 10Elukey: "@taavi@wikimedia.org I can give you some background if you want! It is fine if you don't want to add a +1, but the more people know the be" [puppet] - 10https://gerrit.wikimedia.org/r/1346609 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [14:09:25] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [14:09:39] !log ecarg@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:10:08] zabe: Were you about to deploy? I have an MW thing for this window, but I can wait a bit whilst we do the services instead of doing them in parallel. [14:10:54] James_F: I did not start yet, so I can also wait until you are done with your deployments. Whatever you prefer. :) [14:11:01] I’ll be quick. [14:11:03] (He says.) [14:11:07] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jforrester@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346619 (owner: 10Jforrester) [14:11:40] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [14:12:12] (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346615 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [14:12:16] (03Merged) 10jenkins-bot: mlwiki: Enable Abstract Wikimedia client mode [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346619 (owner: 10Jforrester) [14:12:24] (03PS2) 10Elukey: profile::pki::multirooca: simplify vhost config [puppet] - 10https://gerrit.wikimedia.org/r/1346615 (https://phabricator.wikimedia.org/T436809) [14:12:24] (03PS2) 10Elukey: Disable pki vhost on port 8443 [puppet] - 10https://gerrit.wikimedia.org/r/1346616 (https://phabricator.wikimedia.org/T436809) [14:12:24] (03PS3) 10Elukey: profile::pki::multirootca: remove support for port 8443 [puppet] - 10https://gerrit.wikimedia.org/r/1346617 (https://phabricator.wikimedia.org/T436809) [14:12:36] !log jforrester@deploy1003 Started scap sync-world: Backport for [[gerrit:1346619|mlwiki: Enable Abstract Wikimedia client mode]] [14:13:30] (03PS4) 10Elukey: profile::pki::multirootca: remove support for port 8443 [puppet] - 10https://gerrit.wikimedia.org/r/1346617 (https://phabricator.wikimedia.org/T436809) [14:13:41] (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346617 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [14:13:44] !log ecarg@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:14:02] (03PS1) 10Jforrester: mlwiki: Enable Wikifunctions client mode [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346622 [14:14:08] !log ecarg@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:14:10] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [14:14:19] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1330.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [14:14:27] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [14:14:36] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1331.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [14:14:44] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [14:14:53] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1332.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [14:15:03] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [14:15:11] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1333.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [14:15:20] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [14:15:29] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1334.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [14:15:36] !log dkertesz@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [14:15:39] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [14:16:43] !log jforrester@deploy1003 jforrester: Backport for [[gerrit:1346619|mlwiki: Enable Abstract Wikimedia client mode]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:16:49] !log ecarg@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:17:06] !log oblivian@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on D{wikikube-worker133[0-4].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [14:17:11] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1330.eqiad.wmnet [14:17:11] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1330.eqiad.wmnet [14:17:25] (03CR) 10Ecarg: [C:03+2] wikifunctions: Upgrade orchestrator from 2026-09-23-042024 to 2026-09-30-134715 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346613 (https://phabricator.wikimedia.org/T404869) (owner: 10Jforrester) [14:18:11] 06SRE, 06Collaboration-Services, 06Infrastructure-Foundations, 06ServiceOps, and 2 others: Remove the legacy CNAME endpoints for the URL downloaders - https://phabricator.wikimedia.org/T439713#12379989 (10aputhin) Separating the Toolhub work onto its own subtask: T439732 [14:18:14] (03PS1) 10Muehlenhoff: sre.misc-clusters.roll-restart-reboot-docker-registry: Adapt service names [cookbooks] - 10https://gerrit.wikimedia.org/r/1346625 (https://phabricator.wikimedia.org/T427175) [14:18:49] !log jforrester@deploy1003 jforrester: Continuing with deployment [14:19:18] (03CR) 10Elukey: [C:03+1] sre.misc-clusters.roll-restart-reboot-docker-registry: Adapt service names [cookbooks] - 10https://gerrit.wikimedia.org/r/1346625 (https://phabricator.wikimedia.org/T427175) (owner: 10Muehlenhoff) [14:19:19] (03CR) 10Jforrester: "Can I help in getting this rolled out further?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311060 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [14:19:47] (03Merged) 10jenkins-bot: wikifunctions: Upgrade orchestrator from 2026-09-23-042024 to 2026-09-30-134715 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346613 (https://phabricator.wikimedia.org/T404869) (owner: 10Jforrester) [14:20:45] !log ecarg@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:21:02] !log ecarg@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:21:25] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [14:21:29] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [14:21:31] !log ecarg@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:21:43] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [14:21:53] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2365.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [14:22:06] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [14:22:16] !log ecarg@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:22:23] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [14:22:33] !log ecarg@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:23:12] !log ecarg@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:24:07] !log jforrester@deploy1003 Finished scap sync-world: Backport for [[gerrit:1346619|mlwiki: Enable Abstract Wikimedia client mode]] (duration: 11m 31s) [14:24:22] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jforrester@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346622 (owner: 10Jforrester) [14:24:24] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [14:24:33] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [14:24:34] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2366.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [14:24:43] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [14:24:53] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2367.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [14:25:05] (03Merged) 10jenkins-bot: mlwiki: Enable Wikifunctions client mode [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346622 (owner: 10Jforrester) [14:25:11] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [14:25:21] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2368.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [14:25:25] !log jforrester@deploy1003 Started scap sync-world: Backport for [[gerrit:1346622|mlwiki: Enable Wikifunctions client mode]] [14:25:30] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [14:25:30] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [14:25:39] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2369.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [14:26:34] (03CR) 10Ecarg: [C:03+2] wikifunctions: Log at INFO level in the orchestrator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346099 (https://phabricator.wikimedia.org/T439592) (owner: 10Jforrester) [14:26:38] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [14:26:48] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [14:27:19] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1330.eqiad.wmnet [14:27:20] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1330.eqiad.wmnet [14:27:25] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1331.eqiad.wmnet [14:27:26] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1331.eqiad.wmnet [14:28:57] (03Merged) 10jenkins-bot: wikifunctions: Log at INFO level in the orchestrator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346099 (https://phabricator.wikimedia.org/T439592) (owner: 10Jforrester) [14:29:19] (03PS1) 10Xcollazo: dumps: Keep pages-meta-history only in the 3 newest runs on replicas [puppet] - 10https://gerrit.wikimedia.org/r/1346629 (https://phabricator.wikimedia.org/T437463) [14:29:32] !log jforrester@deploy1003 jforrester: Backport for [[gerrit:1346622|mlwiki: Enable Wikifunctions client mode]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:29:38] (03PS1) 10Elukey: Remove mTLS support for pki clients [puppet] - 10https://gerrit.wikimedia.org/r/1346630 (https://phabricator.wikimedia.org/T436809) [14:29:48] !log ecarg@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:29:58] !log oblivian@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on D{wikikube-worker236[5-9].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [14:30:01] !log ecarg@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:30:02] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2365.codfw.wmnet [14:30:03] !log jforrester@deploy1003 jforrester: Continuing with deployment [14:30:04] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T1400) [14:30:04] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T1430) [14:30:13] (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346630 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [14:30:15] (03CR) 10CI reject: [V:04-1] dumps: Keep pages-meta-history only in the 3 newest runs on replicas [puppet] - 10https://gerrit.wikimedia.org/r/1346629 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [14:30:26] !log ecarg@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:30:37] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2365.codfw.wmnet [14:31:09] !log ecarg@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:31:19] (03CR) 10CI reject: [V:04-1] Remove mTLS support for pki clients [puppet] - 10https://gerrit.wikimedia.org/r/1346630 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [14:31:21] !log ecarg@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:32:00] !log ecarg@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:32:29] (03CR) 10Ecarg: [C:03+2] wikifunctions: Drop support for unused php.containers template block [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346098 (owner: 10Jforrester) [14:33:00] (03PS1) 10Tiziano Fogli: kafka-logging: update brokers list [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346631 (https://phabricator.wikimedia.org/T432444) [14:34:01] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [14:35:09] (03Merged) 10jenkins-bot: wikifunctions: Drop support for unused php.containers template block [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346098 (owner: 10Jforrester) [14:35:16] !log jforrester@deploy1003 Finished scap sync-world: Backport for [[gerrit:1346622|mlwiki: Enable Wikifunctions client mode]] (duration: 09m 51s) [14:36:14] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1331.eqiad.wmnet [14:36:15] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1331.eqiad.wmnet [14:36:21] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1332.eqiad.wmnet [14:36:21] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1332.eqiad.wmnet [14:36:27] !log ecarg@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:37:01] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [14:37:06] !log ecarg@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:37:26] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [14:37:37] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2365.codfw.wmnet [14:37:39] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2365.codfw.wmnet [14:37:41] !log ecarg@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:37:42] (03PS2) 10Xcollazo: dumps: Keep pages-meta-history only in the 3 newest runs on replicas [puppet] - 10https://gerrit.wikimedia.org/r/1346629 (https://phabricator.wikimedia.org/T437463) [14:37:44] (03CR) 10Filippo Giunchedi: [C:03+1] "I'm in the same boat as Taavi re: knowledge, however I'll fearlessly +1 on the basis that cloud and prod should be aligned" [puppet] - 10https://gerrit.wikimedia.org/r/1346609 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [14:37:46] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2366.codfw.wmnet [14:38:03] !log andrew@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm [14:38:21] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2366.codfw.wmnet [14:38:36] (03CR) 10Filippo Giunchedi: [C:03+1] "Can't say I'm familiar with the change, though again +1 on the cloud/prod alignment argument" [puppet] - 10https://gerrit.wikimedia.org/r/1346616 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [14:38:52] zabe: Over to you. [14:39:57] (03PS1) 10Andrew Bogott: partman_early_command: re-enable configure_cephosd_disks for cloudceph [puppet] - 10https://gerrit.wikimedia.org/r/1346634 (https://phabricator.wikimedia.org/T419892) [14:40:50] Thanks! [14:40:59] (03CR) 10Zabe: [C:03+2] Add script to drop duplicate oldimage/filerevision rows [extensions/WikimediaMaintenance] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1346538 (https://phabricator.wikimedia.org/T67264) (owner: 10Zabe) [14:40:59] (03CR) 10Zabe: [C:03+2] Add script to drop duplicate oldimage/filerevision rows [extensions/WikimediaMaintenance] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346537 (https://phabricator.wikimedia.org/T67264) (owner: 10Zabe) [14:41:42] (03CR) 10Andrew Bogott: [C:03+2] partman_early_command: re-enable configure_cephosd_disks for cloudceph [puppet] - 10https://gerrit.wikimedia.org/r/1346634 (https://phabricator.wikimedia.org/T419892) (owner: 10Andrew Bogott) [14:43:06] (03Merged) 10jenkins-bot: Add script to drop duplicate oldimage/filerevision rows [extensions/WikimediaMaintenance] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1346538 (https://phabricator.wikimedia.org/T67264) (owner: 10Zabe) [14:43:15] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [14:43:26] !log oblivian@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [14:43:29] !log oblivian@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [14:43:59] (03Merged) 10jenkins-bot: Add script to drop duplicate oldimage/filerevision rows [extensions/WikimediaMaintenance] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346537 (https://phabricator.wikimedia.org/T67264) (owner: 10Zabe) [14:44:00] (03PS2) 10Elukey: Remove mTLS support for pki clients [puppet] - 10https://gerrit.wikimedia.org/r/1346630 (https://phabricator.wikimedia.org/T436809) [14:44:06] !log oblivian@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'. [14:44:10] !log oblivian@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [14:44:36] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1346537|Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538|Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] [14:44:39] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1332.eqiad.wmnet [14:44:40] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1332.eqiad.wmnet [14:44:40] T67264: File storage uses (title,timestamp) as a unique key, but this is not unique - https://phabricator.wikimedia.org/T67264 [14:44:40] T428406: old file revisions missing of File:A_Warm_Shade_of_Ivory_-_Henry_Mancini_album_cover.jpg - https://phabricator.wikimedia.org/T428406 [14:44:45] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1333.eqiad.wmnet [14:44:46] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1333.eqiad.wmnet [14:45:00] !log oblivian@deploy1003 helmfile [codfw] START helmfile.d/admin 'apply'. [14:45:04] !log oblivian@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'apply'. [14:45:22] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2366.codfw.wmnet [14:45:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2366.codfw.wmnet [14:45:23] !log oblivian@deploy1003 helmfile [eqiad] START helmfile.d/admin 'apply'. [14:45:26] !log oblivian@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'apply'. [14:45:29] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2367.codfw.wmnet [14:45:59] (03CR) 10Xcollazo: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346629 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [14:46:04] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2367.codfw.wmnet [14:46:05] (03CR) 10Vgutierrez: [C:03+1] "TBH I share the same concerns as @bcornwall@wikimedia.org but I thought of this CR as a quick way of not running again on the same issue. " [puppet] - 10https://gerrit.wikimedia.org/r/1345963 (https://phabricator.wikimedia.org/T433005) (owner: 10Daniel Kertesz) [14:47:09] (03PS3) 10Xcollazo: analytics: Keep File Exports for 170 days [puppet] - 10https://gerrit.wikimedia.org/r/1346119 (https://phabricator.wikimedia.org/T437463) [14:47:09] (03CR) 10Xcollazo: "Opened [T439733 File Exports: Consider storing the exports in Ceph instead of HDFS](https://phabricator.wikimedia.org/T439733)" [puppet] - 10https://gerrit.wikimedia.org/r/1346119 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [14:47:19] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1346609 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [14:47:25] (03CR) 10Xcollazo: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346119 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [14:47:58] !log ecarg@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:48:01] (03PS4) 10Scott French: hieradata: Divert testwiki traffic to mw-pretrain at ATS [puppet] - 10https://gerrit.wikimedia.org/r/1326930 (https://phabricator.wikimedia.org/T427666) [14:48:33] !log ecarg@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:48:41] !log zabe@deploy1003 zabe: Backport for [[gerrit:1346537|Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538|Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:48:49] !log ecarg@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:49:09] !log ecarg@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:49:14] !log zabe@deploy1003 zabe: Continuing with deployment [14:49:21] (03CR) 10Muehlenhoff: [C:03+2] sre.misc-clusters.roll-restart-reboot-docker-registry: Adapt service names [cookbooks] - 10https://gerrit.wikimedia.org/r/1346625 (https://phabricator.wikimedia.org/T427175) (owner: 10Muehlenhoff) [14:49:40] !log ecarg@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:51:27] !log dkertesz@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - upgrade haproxy to 3.2.25-1 (T439667) [14:51:30] T439667: Upgrade HAProxy to 3.2.25 on cp hosts - https://phabricator.wikimedia.org/T439667 [14:51:58] (03CR) 10Krinkle: "Probably not: https://phabricator.wikimedia.org/T271001#12375659" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311060 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [14:53:16] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2367.codfw.wmnet [14:53:18] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2367.codfw.wmnet [14:53:18] 06SRE, 06Infrastructure-Foundations, 06Traffic, 07LDAP: Add a new LVS ldap-ro-next service - https://phabricator.wikimedia.org/T439730#12380195 (10ssingh) Let us know how/if we can help :) [14:53:25] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2368.codfw.wmnet [14:53:25] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1333.eqiad.wmnet [14:53:25] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1333.eqiad.wmnet [14:53:31] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1334.eqiad.wmnet [14:53:32] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker1334.eqiad.wmnet [14:53:37] Krinkle: Alas. Will follow the task, thanks. [14:53:58] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2368.codfw.wmnet [14:54:29] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1346537|Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]], [[gerrit:1346538|Add script to drop duplicate oldimage/filerevision rows (T67264 T428406)]] (duration: 09m 54s) [14:54:34] T67264: File storage uses (title,timestamp) as a unique key, but this is not unique - https://phabricator.wikimedia.org/T67264 [14:54:34] T428406: old file revisions missing of File:A_Warm_Shade_of_Ivory_-_Henry_Mancini_album_cover.jpg - https://phabricator.wikimedia.org/T428406 [14:58:50] !log jmm@cumin2003 START - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry rolling restart_daemons on A:docker-registry [14:59:40] (03CR) 10Muehlenhoff: [C:03+1] "Looks good!" [puppet] - 10https://gerrit.wikimedia.org/r/1346615 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [14:59:53] James_F: np :) [15:01:08] !log jmm@cumin2003 END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-docker-registry (exit_code=0) rolling restart_daemons on A:docker-registry [15:01:17] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2368.codfw.wmnet [15:01:19] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2368.codfw.wmnet [15:01:24] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2369.codfw.wmnet [15:01:56] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for Tsevener - https://phabricator.wikimedia.org/T439138#12380262 (10ayounsi) [15:02:01] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1334.eqiad.wmnet [15:02:02] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1334.eqiad.wmnet [15:02:03] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D{wikikube-worker133[0-4].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [15:02:22] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users and deployment for lwatson - https://phabricator.wikimedia.org/T438926#12380267 (10ayounsi) [15:02:34] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2369.codfw.wmnet [15:03:08] !log jmm@cumin2003 START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling restart_daemons on A:ncredir [15:03:58] James_F: between https://phabricator.wikimedia.org/T438388#12347252 (showing mathjax.org > demo > right click MathML) and https://www.mediawiki.org/wiki/User:Krinkle/MathML_Shortlist (the template using with native vs mathjax) you can see the work for a typical such bug. However the patches aren't as straight forward in my experience, so hard to do without a lot of MathML familiarity and judgement. The easy [15:03:59] part (parsing LateX into MathML) broadly has been done already. Now we're into the hard/weird/two-bad-choices stuff. [15:04:54] Krinkle: I think we should slowly add rendering bugs to the Mathoid endpoint until people accept that MathML is the better of the two. [15:06:41] (03PS1) 10Jforrester: Factor out Abstract Wikipedia client mode into a dblist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346643 [15:06:41] (03PS1) 10Jforrester: Abstractwiki: Drop temporary memcached/mainstash config, no longer needed [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346644 [15:07:53] James_F: he, well, we do actually have a growing list of things that only work in MathJax 4. [15:08:11] so in a sense, by using MathJax 2 in Mathoid, we have bugs/limitations there and we're starting to notice them [15:08:23] because sometimes when you look for differences, you find them, but not in the direction you expect :) [15:08:57] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.15 point update - https://phabricator.wikimedia.org/T434631#12380309 (10MoritzMuehlenhoff) [15:09:01] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [15:09:11] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [15:09:34] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2369.codfw.wmnet [15:09:35] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2369.codfw.wmnet [15:09:36] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on D{wikikube-worker236[5-9].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [15:11:53] !log jmm@cumin2003 END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling restart_daemons on A:ncredir [15:14:42] 06SRE-OnFire, 06MediaWiki-Engineering, 06ServiceOps, 06SRE Observability, 07Sustainability (Incident Followup): Reduce the amount of messages sent through channel:Memcached during failures - https://phabricator.wikimedia.org/T390529#12380363 (10herron) >>! In T390529#12379393, @MLechvien-WMF wrote: > @he... [15:15:14] !log andrew@cumin2003 START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm [15:17:36] (03CR) 10Ayounsi: [C:03+2] admin: Add FIDO SSH keys for kharlan [puppet] - 10https://gerrit.wikimedia.org/r/1346423 (https://phabricator.wikimedia.org/T439665) (owner: 10Kosta Harlan) [15:18:57] (03CR) 10Elukey: [C:03+2] Remove PKI mTLS settings for cloud [puppet] - 10https://gerrit.wikimedia.org/r/1346609 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [15:19:48] (03PS7) 101Veertje: Fix login form: responsive labels and mobile-friendly layout [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 [15:23:22] (03PS1) 10CDanis: admin: ml-lab-users: add cdanis [puppet] - 10https://gerrit.wikimedia.org/r/1346652 [15:25:43] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [15:25:45] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [15:27:19] (03PS1) 10Aaron Schulz: developer-portal: update the service tag [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346655 (https://phabricator.wikimedia.org/T434137) [15:27:35] (03CR) 10Herron: [C:03+1] kafka-logging: update brokers list [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346631 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [15:31:04] (03CR) 10Giuseppe Lavagetto: [C:03+2] shellbox-timeline: run under kvm in eqiad (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342965 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [15:33:21] (03PS1) 10Ayounsi: Add eniroomand to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1346659 (https://phabricator.wikimedia.org/T439644) [15:33:35] (03Merged) 10jenkins-bot: shellbox-timeline: run under kvm in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342965 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [15:35:49] (03PS9) 10Dduvall: buildx: New driver for building images via Buildx/BuildKit [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958) [15:35:56] (03CR) 10Dduvall: buildx: New driver for building images via Buildx/BuildKit (036 comments) [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958) (owner: 10Dduvall) [15:35:59] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 13Patch-For-Review: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12380514 (10ayounsi) @eniroomand Please familiarize yourself with https://wikitech.wikimedia.org/wiki/Data_Platform/Data_access#Us... [15:39:52] (03PS1) 10Ayounsi: Add lwatson to deployment and analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1346660 (https://phabricator.wikimedia.org/T438926) [15:41:06] !log oblivian@deploy1003 helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply [15:41:26] !log oblivian@deploy1003 helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply [15:41:54] (03CR) 10Muehlenhoff: "Patch looks fine, but Ilias or Chris Albon and your manager need to approve (either here on Gerrit or alternatively create a task)" [puppet] - 10https://gerrit.wikimedia.org/r/1346652 (owner: 10CDanis) [15:42:03] (03PS1) 10Ayounsi: add tsev to deployment [puppet] - 10https://gerrit.wikimedia.org/r/1346661 (https://phabricator.wikimedia.org/T439138) [15:42:12] !log andrew@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage [15:43:44] (03CR) 10Btullis: "We only use CephFS as an interim file store for dumps v1. They are pushed from here to the clouddumps100[1-2] servers, from where they are" [puppet] - 10https://gerrit.wikimedia.org/r/1346119 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [15:44:05] (03PS2) 10Aaron Schulz: developer-portal: update the service tag for developer.wikimedia.org [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346655 (https://phabricator.wikimedia.org/T434137) [15:45:37] (03PS1) 10Bking: wmnet: Add records for matomo and matomo-next [dns] - 10https://gerrit.wikimedia.org/r/1346662 (https://phabricator.wikimedia.org/T436003) [15:46:07] (03CR) 10Hnowlan: [C:03+1] alert: add a dummy Klaxon managers incident URL [labs/private] - 10https://gerrit.wikimedia.org/r/1345218 (owner: 10Andrea Denisse) [15:47:26] !log andrew@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage [15:47:49] !log andrew@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcephosd1043.eqiad.wmnet with OS bookworm [15:49:43] (03PS1) 10Lerickson: Revert "Annotate wdqs-proxy pods so metrics are scraped." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346663 [15:49:50] (03CR) 10Btullis: "I think that the changes we need to make are around this file, not on the HDFS retention times:" [puppet] - 10https://gerrit.wikimedia.org/r/1346119 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [15:49:53] !log andrew@cumin2003 START - Cookbook sre.hosts.reimage for host cloudcephosd1043.eqiad.wmnet with OS bookworm [15:50:23] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1346661 (https://phabricator.wikimedia.org/T439138) (owner: 10Ayounsi) [15:56:42] (03CR) 10Andrew Bogott: [C:03+2] "ftr @fgiunchedi@wikimedia.org this got 1043 imaging properly." [puppet] - 10https://gerrit.wikimedia.org/r/1346634 (https://phabricator.wikimedia.org/T419892) (owner: 10Andrew Bogott) [15:59:37] (03CR) 10JHathaway: [C:03+2] puppet-agent: add option to send legacy facts [puppet] - 10https://gerrit.wikimedia.org/r/1346165 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [16:04:06] (03CR) 10Alex Paskulin: [C:03+1] "Reviewed the relevant patches and checked the site in staging. Looks good!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346655 (https://phabricator.wikimedia.org/T434137) (owner: 10Aaron Schulz) [16:07:06] (03CR) 10Aaron Schulz: [C:03+2] developer-portal: update the service tag for developer.wikimedia.org [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346655 (https://phabricator.wikimedia.org/T434137) (owner: 10Aaron Schulz) [16:08:15] !log andrew@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage [16:08:48] (03Merged) 10jenkins-bot: developer-portal: update the service tag for developer.wikimedia.org [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346655 (https://phabricator.wikimedia.org/T434137) (owner: 10Aaron Schulz) [16:11:14] !log aaron@deploy1003 helmfile [staging] START helmfile.d/services/developer-portal: apply [16:11:31] !log aaron@deploy1003 helmfile [staging] DONE helmfile.d/services/developer-portal: apply [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:12:38] !log andrew@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcephosd1043.eqiad.wmnet with reason: host reimage [16:13:18] !log aaron@deploy1003 helmfile [codfw] START helmfile.d/services/developer-portal: apply [16:13:44] !log aaron@deploy1003 helmfile [codfw] DONE helmfile.d/services/developer-portal: apply [16:14:38] !log aaron@deploy1003 helmfile [eqiad] START helmfile.d/services/developer-portal: apply [16:14:51] !log aaron@deploy1003 helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:30:52] (03CR) 10Kamila Součková: [V:03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9489/console" [puppet] - 10https://gerrit.wikimedia.org/r/1343092 (https://phabricator.wikimedia.org/T438208) (owner: 10Aghirelli) [16:37:10] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12380977 (10GGoncalves-WMF) [16:37:56] (03CR) 10Kamila Součková: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9490/co" [puppet] - 10https://gerrit.wikimedia.org/r/1343092 (https://phabricator.wikimedia.org/T438208) (owner: 10Aghirelli) [16:38:34] !log andrew@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcephosd1043.eqiad.wmnet with OS bookworm [16:39:18] (03CR) 10Kamila Součková: [V:03+1 C:03+1] eventschemas: add api-platform-schemas repository [puppet] - 10https://gerrit.wikimedia.org/r/1343092 (https://phabricator.wikimedia.org/T438208) (owner: 10Aghirelli) [16:40:33] (03CR) 10Ilias Sarantopoulos: [C:03+1] "approved!" [puppet] - 10https://gerrit.wikimedia.org/r/1346652 (owner: 10CDanis) [16:40:44] 06SRE, 10Kafka-Infrastructure, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Monitor kafka-configurator runs and pending changes - https://phabricator.wikimedia.org/T439506#12381003 (10Ottomata) [16:46:25] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12381050 (10eniroomand) @ayounsi I confirm that I have read the guidelines on the responsibilities for... [16:48:36] (03CR) 10Kamila Součková: "Um, I will push the stuff on my laptop that may or may not address a lot of the above '^^ probably not this week, but I will do my best to" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1120500 (owner: 10Volans) [16:51:50] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [16:51:57] (03CR) 10Kamila Součková: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343101 (owner: 10Aghirelli) [16:52:21] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [16:58:54] (03CR) 10Kamila Součková: [C:03+1] "Can't actually see it in PCC since it's not in use, but let's find out 😄" [puppet] - 10https://gerrit.wikimedia.org/r/1343101 (owner: 10Aghirelli) [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T1700) [17:02:53] (03CR) 10Xcollazo: "Right, HDFS is an interim file store here as well. Will close this thread, but let's keep the conversation going at T439733." [puppet] - 10https://gerrit.wikimedia.org/r/1346119 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [17:04:53] (03PS1) 10Bking: dse-k8s: Add ingress for matomo and matomo-next services [puppet] - 10https://gerrit.wikimedia.org/r/1346703 (https://phabricator.wikimedia.org/T436003) [17:05:35] (03PS1) 10Ryan Kemper: Add Sparqlbridge to WDQS/WCQS allowlist [puppet] - 10https://gerrit.wikimedia.org/r/1346704 (https://phabricator.wikimedia.org/T426878) [17:06:08] (03CR) 10Ryan Kemper: [V:03+2 C:03+2] Add Sparqlbridge to WDQS/WCQS allowlist [puppet] - 10https://gerrit.wikimedia.org/r/1346704 (https://phabricator.wikimedia.org/T426878) (owner: 10Ryan Kemper) [17:08:32] (03CR) 10Btullis: [C:03+1] "Looks good." [dns] - 10https://gerrit.wikimedia.org/r/1346662 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [17:08:57] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [17:09:01] (03CR) 10Btullis: [C:03+1] dse-k8s: Add ingress for matomo and matomo-next services [puppet] - 10https://gerrit.wikimedia.org/r/1346703 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [17:09:29] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [17:12:38] (03CR) 10Bking: [C:03+2] wmnet: Add records for matomo and matomo-next [dns] - 10https://gerrit.wikimedia.org/r/1346662 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [17:13:17] !log bking@dns2004 START - running authdns-update [17:14:21] (03PS1) 10Herron: Test new rsyslog image on mw-experimental and mw-web canary [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346712 (https://phabricator.wikimedia.org/T438375) [17:14:25] (03CR) 10Kamila Součková: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9491/co" [puppet] - 10https://gerrit.wikimedia.org/r/1343109 (owner: 10Aghirelli) [17:14:42] (03PS1) 10Atsuko: airflow3: smtp_default connection [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346713 (https://phabricator.wikimedia.org/T433383) [17:15:50] !log bking@dns2004 END - running authdns-update [17:16:16] (03CR) 10Ebernhardson: [C:03+2] semantic ssd test: Drop to 3 masters, fix affinity [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344715 (https://phabricator.wikimedia.org/T438058) (owner: 10Ebernhardson) [17:17:41] (03Merged) 10jenkins-bot: semantic ssd test: Drop to 3 masters, fix affinity [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344715 (https://phabricator.wikimedia.org/T438058) (owner: 10Ebernhardson) [17:17:51] (03CR) 10Bking: [C:03+2] dse-k8s: Add ingress for matomo and matomo-next services [puppet] - 10https://gerrit.wikimedia.org/r/1346703 (https://phabricator.wikimedia.org/T436003) (owner: 10Bking) [17:18:20] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.restart [17:18:32] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.restart [17:21:29] !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 () [17:21:34] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [17:21:37] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [17:21:38] !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 () [17:24:34] (03CR) 10Lerickson: [C:03+2] Revert "Annotate wdqs-proxy pods so metrics are scraped." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346663 (owner: 10Lerickson) [17:25:21] (03CR) 10RLazarus: [C:03+1] "The config change looks good, optimistic +1 pending manager approval on the task." [puppet] - 10https://gerrit.wikimedia.org/r/1346659 (https://phabricator.wikimedia.org/T439644) (owner: 10Ayounsi) [17:26:00] (03Merged) 10jenkins-bot: Revert "Annotate wdqs-proxy pods so metrics are scraped." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346663 (owner: 10Lerickson) [17:27:30] (03Abandoned) 10DLynch: editcheck-headless: Route the ingress to the plaintext gRPC port [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344809 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [17:27:53] (03Abandoned) 10DLynch: python-webapp: Add grpc_ingress for a plaintext gRPC port [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344808 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [17:28:05] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [17:29:49] (03CR) 10Kamila Součková: [V:03+1 C:03+1] "LGTM, I think 😄 (I'm not really familiar with sparse checkouts, but at least it shouldn't break anything else :D)" [puppet] - 10https://gerrit.wikimedia.org/r/1343109 (owner: 10Aghirelli) [17:33:41] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [17:33:42] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [17:34:35] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [17:34:39] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [17:38:59] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q2:rack/setup/install (3) new cloud dump hosts - https://phabricator.wikimedia.org/T439754 (10RobH) 03NEW [17:40:38] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q2:rack/setup/install (3) new cloud dump hosts - https://phabricator.wikimedia.org/T439754#12381419 (10RobH) a:03Andrew @Andrew, In the racking details for these 3 new hosts you listed hostnames dumps100[123], but we already have existing '... [17:41:00] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q2:rack/setup/install (3) new cloud dump hosts - https://phabricator.wikimedia.org/T439754#12381428 (10RobH) [17:41:20] 10ops-eqiad, 06SRE, 06DC-Ops, 06cloud-services-team (Hardware): Q2:rack/setup/install (3) dumps100[123] - https://phabricator.wikimedia.org/T439754#12381431 (10RobH) [17:47:21] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.restart (exit_code=0) [17:55:53] (03CR) 10Ryan Kemper: [C:04-1] "One blocking finding, rest looks good. The current approach happens to work, but if we flipped the values from 'True' to 'False' they woul" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346713 (https://phabricator.wikimedia.org/T433383) (owner: 10Atsuko) [18:00:05] brennen and jeena: I, the Bot under the Fountain, call upon thee, The Deployer, to do MediaWiki train - Utc-7 Version deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T1800). [18:01:02] o/ [18:01:04] (03CR) 10Dreamy Jazz: [C:03+1] CheckUser: Enable ClientHintsInEditRequest on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346542 (https://phabricator.wikimedia.org/T401808) (owner: 10Kosta Harlan) [18:01:16] (03CR) 10Dreamy Jazz: [C:03+1] CheckUser: Enable ClientHintsInEditRequest on group1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346541 (https://phabricator.wikimedia.org/T401808) (owner: 10Kosta Harlan) [18:05:00] (03Abandoned) 10Xcollazo: analytics: Keep File Exports for 170 days [puppet] - 10https://gerrit.wikimedia.org/r/1346119 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [18:10:36] !log 1.47.0-wmf.22 train status (T438218): no current blockers, rolling to group1 [18:10:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:10:48] T438218: 1.47.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T438218 [18:13:11] (03PS1) 10TrainBranchBot: group1 to 1.47.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346742 (https://phabricator.wikimedia.org/T438218) [18:13:14] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by brennen@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346742 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [18:14:17] (03Merged) 10jenkins-bot: group1 to 1.47.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346742 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [18:18:55] (03PS1) 10Lerickson: Annotate wdqs-proxy pods so metrics are scraped. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346743 (https://phabricator.wikimedia.org/T439449) [18:19:43] (03PS2) 10Lerickson: Annotate wdqs-proxy pods so metrics are scraped. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346743 (https://phabricator.wikimedia.org/T439449) [18:20:04] (03PS3) 10Lerickson: Annotate wdqs-proxy pods so metrics are scraped. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346743 (https://phabricator.wikimedia.org/T439449) [18:20:48] (03PS4) 10Lerickson: Annotate wdqs-proxy pods so metrics are scraped. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346743 (https://phabricator.wikimedia.org/T439449) [18:22:25] !log brennen@deploy1003 Rolling back deployment [18:24:46] !log brennen@deploy1003 rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs T438218 [18:24:49] T438218: 1.47.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T438218 [18:30:02] (03PS4) 10Dzahn: add attribution static site to helmfile, create values file [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341335 (https://phabricator.wikimedia.org/T437635) [18:30:34] (03CR) 10Dzahn: [C:03+2] "thanks! just updated the image name" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341335 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn) [18:31:08] (03PS1) 10JHathaway: puppet: disable sending legacy facts [puppet] - 10https://gerrit.wikimedia.org/r/1346748 (https://phabricator.wikimedia.org/T372666) [18:31:21] (03PS1) 10TrainBranchBot: group0 to 1.47.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346749 (https://phabricator.wikimedia.org/T438218) [18:31:23] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by brennen@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346749 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [18:31:30] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346748 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [18:32:35] (03Merged) 10jenkins-bot: group0 to 1.47.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346749 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [18:32:43] (03PS1) 10Xcollazo: dumps: Keep File Exports on the distribution hosts, not in HDFS [puppet] - 10https://gerrit.wikimedia.org/r/1346752 (https://phabricator.wikimedia.org/T437463) [18:33:25] (03Merged) 10jenkins-bot: add attribution static site to helmfile, create values file [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341335 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn) [18:34:02] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [18:37:06] !log flipping flag to disable legacy facts in puppet prod [18:37:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:37:13] (03CR) 10JHathaway: [C:03+2] puppet: disable sending legacy facts [puppet] - 10https://gerrit.wikimedia.org/r/1346748 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [18:40:19] !log brennen@deploy1003 rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.22 refs T438218 [18:40:23] T438218: 1.47.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T438218 [18:41:30] (03PS1) 10TrainBranchBot: group1 to 1.47.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346755 (https://phabricator.wikimedia.org/T438218) [18:41:32] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by brennen@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346755 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [18:42:35] (03Merged) 10jenkins-bot: group1 to 1.47.0-wmf.22 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346755 (https://phabricator.wikimedia.org/T438218) (owner: 10TrainBranchBot) [18:52:04] !log brennen@deploy1003 rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.22 refs T438218 [18:52:07] T438218: 1.47.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T438218 [18:55:08] !log dzahn@deploy1003 helmfile [staging] START helmfile.d/services/miscweb: apply [18:55:24] !log dzahn@deploy1003 helmfile [staging] DONE helmfile.d/services/miscweb: apply [18:55:35] !log dzahn@deploy1003 helmfile [codfw] START helmfile.d/services/miscweb: apply [18:56:08] !log dzahn@deploy1003 helmfile [codfw] DONE helmfile.d/services/miscweb: apply [18:56:27] !log dzahn@deploy1003 helmfile [eqiad] START helmfile.d/services/miscweb: apply [18:56:38] !log dzahn@deploy1003 helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [18:57:00] Amir1: https://attribution.wikimedia.org/ [18:57:33] (just in time delivery before October) [19:00:26] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, September 30 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346327 (https://phabricator.wikimedia.org/T439283) (owner: 10Catrope) [19:00:42] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, September 30 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346328 (https://phabricator.wikimedia.org/T439283) (owner: 10Catrope) [19:00:54] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, September 30 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346329 (https://phabricator.wikimedia.org/T439283) (owner: 10Catrope) [19:02:32] (03CR) 10Dzahn: [C:03+2] "site is live !" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341335 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn) [19:14:41] (03PS1) 10JHathaway: lvs: replace legacy fact [puppet] - 10https://gerrit.wikimedia.org/r/1346762 (https://phabricator.wikimedia.org/T372666) [19:14:55] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346762 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [19:16:51] (03PS2) 10Dzahn: microsites: add monitoring for attribution.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1342791 (https://phabricator.wikimedia.org/T437635) [19:16:53] (03PS1) 10Arlolra: Set a default for wgSplitParsoidParserCache in prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346763 (https://phabricator.wikimedia.org/T432760) [19:18:19] (03PS1) 10Xcollazo: dumps: Write mirror rsync lists that mix v1 dumps and File Exports [puppet] - 10https://gerrit.wikimedia.org/r/1346764 (https://phabricator.wikimedia.org/T437463) [19:18:21] (03PS1) 10Xcollazo: dumps: Use the mixed rsync lists in the mirror modules [puppet] - 10https://gerrit.wikimedia.org/r/1346765 (https://phabricator.wikimedia.org/T437463) [19:19:02] (03PS1) 10JHathaway: presto: replace legacy fact [puppet] - 10https://gerrit.wikimedia.org/r/1346766 (https://phabricator.wikimedia.org/T372666) [19:19:16] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, September 30 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346763 (https://phabricator.wikimedia.org/T432760) (owner: 10Arlolra) [19:19:17] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346766 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [19:19:17] (03CR) 10CI reject: [V:04-1] dumps: Write mirror rsync lists that mix v1 dumps and File Exports [puppet] - 10https://gerrit.wikimedia.org/r/1346764 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [19:20:51] (03CR) 10JHathaway: [C:03+2] lvs: replace legacy fact [puppet] - 10https://gerrit.wikimedia.org/r/1346762 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [19:22:08] (03PS2) 10Xcollazo: dumps: Write mirror rsync lists that mix v1 dumps and File Exports [puppet] - 10https://gerrit.wikimedia.org/r/1346764 (https://phabricator.wikimedia.org/T437463) [19:22:09] (03PS2) 10Xcollazo: dumps: Use the mixed rsync lists in the mirror modules [puppet] - 10https://gerrit.wikimedia.org/r/1346765 (https://phabricator.wikimedia.org/T437463) [19:25:00] (03PS1) 10JHathaway: kiwix: replace legacy fact [puppet] - 10https://gerrit.wikimedia.org/r/1346767 (https://phabricator.wikimedia.org/T372666) [19:25:14] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346767 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [19:25:29] (03CR) 10JHathaway: [C:03+2] presto: replace legacy fact [puppet] - 10https://gerrit.wikimedia.org/r/1346766 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [19:26:05] (03CR) 10Dzahn: [C:03+2] microsites: add monitoring for attribution.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1342791 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn) [19:26:11] (03PS2) 10DLynch: editcheck-headless: Delay and relax the liveness probe [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344371 (https://phabricator.wikimedia.org/T436689) [19:26:11] (03PS1) 10DLynch: editcheck-headless: upgrade image to 2026-09-30-162247-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346768 (https://phabricator.wikimedia.org/T436689) [19:26:55] (03CR) 10JHathaway: [C:03+2] kiwix: replace legacy fact [puppet] - 10https://gerrit.wikimedia.org/r/1346767 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [19:27:32] jhathaway: feel free to merge both [19:27:47] done! [19:27:53] thanks [19:28:52] (03PS1) 10Majavah: Remove meet accountmanager classes [puppet] - 10https://gerrit.wikimedia.org/r/1346769 [19:32:10] (03CR) 10DLynch: [C:03+2] editcheck-headless: upgrade image to 2026-09-30-162247-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346768 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:32:16] (03CR) 10DLynch: [C:03+2] editcheck-headless: Delay and relax the liveness probe [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344371 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:33:38] (03Merged) 10jenkins-bot: editcheck-headless: Delay and relax the liveness probe [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344371 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:34:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:34:48] (03Merged) 10jenkins-bot: editcheck-headless: upgrade image to 2026-09-30-162247-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346768 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [19:44:47] (03PS1) 10JHathaway: Revert "puppet: disable sending legacy facts" [puppet] - 10https://gerrit.wikimedia.org/r/1346774 [19:44:50] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346774 (owner: 10JHathaway) [19:45:47] !log reverting legacy facts disablement, discovered some remaining usage [19:45:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:46:14] (03CR) 10JHathaway: [C:03+2] Revert "puppet: disable sending legacy facts" [puppet] - 10https://gerrit.wikimedia.org/r/1346774 (owner: 10JHathaway) [19:50:39] !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_codfw - 7.1.1-2~bpo13+wmf3 () [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: OwO what's this, a deployment window?? UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T2000). nyaa~ [20:00:05] maryum and arlolra: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:40] (03CR) 10Subramanya Sastry: [C:03+1] Set a default for wgSplitParsoidParserCache in prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346763 (https://phabricator.wikimedia.org/T432760) (owner: 10Arlolra) [20:00:43] hi! I can use spiderpig to do my backports [20:02:48] same [20:03:34] arlolra do you want to go first? [20:03:59] I can but you're first on the list [20:04:36] If you have i18n changes, I'll be quick [20:05:26] no i18n changes for me [20:05:31] I have three diff changes [20:05:39] !log kemayo@deploy1003 helmfile [staging] START helmfile.d/services/editcheck-headless: apply [20:05:42] so it'll be longer but I can go [20:06:22] go for it [20:06:26] !log kemayo@deploy1003 helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply [20:06:42] okay cool about to go [20:07:32] mine does have i18n and it looks like it'll take about 20 mins [20:07:53] only one of the changes has i18n though [20:08:00] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mstyles@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346327 (https://phabricator.wikimedia.org/T439283) (owner: 10Catrope) [20:08:00] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mstyles@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346328 (https://phabricator.wikimedia.org/T439283) (owner: 10Catrope) [20:08:00] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mstyles@deploy1003 using scap backport" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346329 (https://phabricator.wikimedia.org/T439283) (owner: 10Catrope) [20:08:09] it's fine, no rush on my part [20:10:13] (03Merged) 10jenkins-bot: Email confirmation: implement delayed enforcement experiment [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346327 (https://phabricator.wikimedia.org/T439283) (owner: 10Catrope) [20:10:52] PROBLEM - MariaDB Replica Lag: s1 on db1240 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 636.28 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:10:56] PROBLEM - MariaDB Replica Lag: s8 on db2198 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 640.18 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:11:06] PROBLEM - MariaDB Replica Lag: s8 on db1285 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 650.69 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:11:07] (03Merged) 10jenkins-bot: Email confirmation: Add server-side enforcement [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346328 (https://phabricator.wikimedia.org/T439283) (owner: 10Catrope) [20:11:08] PROBLEM - MariaDB Replica Lag: s1 on db2250 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 652.99 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:11:09] (03Merged) 10jenkins-bot: Email confirmation: Prepare for next experiment [extensions/WikimediaEvents] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346329 (https://phabricator.wikimedia.org/T439283) (owner: 10Catrope) [20:11:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-int releases routed via main at eqiad: 19.76% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-int&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:11:34] !log mstyles@deploy1003 Started scap sync-world: Backport for [[gerrit:1346327|Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328|Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329|Email confirmation: Prepare for next experiment (T439283)]] [20:11:38] T439283: Implement delayed (edit attempt time) email confirmation enforcement experiment - https://phabricator.wikimedia.org/T439283 [20:11:56] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [20:16:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-int releases routed via main at eqiad: 23.19% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-int&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:17:31] (03PS2) 10Ahmon Dancy: kubernetes.yaml: Revise pretrain logstash thresholds [puppet] - 10https://gerrit.wikimedia.org/r/1345206 (https://phabricator.wikimedia.org/T435419) [20:18:27] (03CR) 10Ahmon Dancy: "Revised" [puppet] - 10https://gerrit.wikimedia.org/r/1345206 (https://phabricator.wikimedia.org/T435419) (owner: 10Ahmon Dancy) [20:20:04] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [20:20:09] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [20:20:41] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [20:31:23] !log mstyles@deploy1003 mstyles, catrope: Backport for [[gerrit:1346327|Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328|Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329|Email confirmation: Prepare for next experiment (T439283)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:31:26] T439283: Implement delayed (edit attempt time) email confirmation enforcement experiment - https://phabricator.wikimedia.org/T439283 [20:32:19] !log mstyles@deploy1003 mstyles, catrope: Continuing with deployment [20:38:37] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [20:38:55] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [20:41:43] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv4 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133212 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [20:44:22] !log mstyles@deploy1003 Finished scap sync-world: Backport for [[gerrit:1346327|Email confirmation: implement delayed enforcement experiment (T439283)]], [[gerrit:1346328|Email confirmation: Add server-side enforcement (T439283)]], [[gerrit:1346329|Email confirmation: Prepare for next experiment (T439283)]] (duration: 32m 47s) [20:44:25] T439283: Implement delayed (edit attempt time) email confirmation enforcement experiment - https://phabricator.wikimedia.org/T439283 [20:45:27] arlolra_ deploy finally finished! you can go ahead [20:45:33] thanks [20:46:17] (03CR) 10TrainBranchBot: [C:03+2] "Approved by arlolra@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346763 (https://phabricator.wikimedia.org/T432760) (owner: 10Arlolra) [20:47:15] (03Merged) 10jenkins-bot: Set a default for wgSplitParsoidParserCache in prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346763 (https://phabricator.wikimedia.org/T432760) (owner: 10Arlolra) [20:47:40] !log arlolra@deploy1003 Started scap sync-world: Backport for [[gerrit:1346763|Set a default for wgSplitParsoidParserCache in prod (T432760)]] [20:47:43] T432760: Add configuration options to core for Parsoid by default - https://phabricator.wikimedia.org/T432760 [20:51:43] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv4 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133212 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [20:52:01] !log arlolra@deploy1003 arlolra: Backport for [[gerrit:1346763|Set a default for wgSplitParsoidParserCache in prod (T432760)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:52:47] !log arlolra@deploy1003 arlolra: Continuing with deployment [20:53:55] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, October 05 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-it" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346053 (https://phabricator.wikimedia.org/T438779) (owner: 10Jdlrobson) [20:57:55] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [20:58:01] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [20:59:08] (03PS1) 10Anne Tomasevich: Update pulsating blue dot behavior for anonymous bookmark [extensions/ReadingLists] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346793 (https://phabricator.wikimedia.org/T438779) [20:59:42] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, October 05 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-it" [extensions/ReadingLists] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346793 (https://phabricator.wikimedia.org/T438779) (owner: 10Anne Tomasevich) [21:00:04] Deploy window Wikifunctions Services UTC Late (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T2100) [21:00:06] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [21:00:08] !log arlolra@deploy1003 Finished scap sync-world: Backport for [[gerrit:1346763|Set a default for wgSplitParsoidParserCache in prod (T432760)]] (duration: 12m 28s) [21:00:13] T432760: Add configuration options to core for Parsoid by default - https://phabricator.wikimedia.org/T432760 [21:00:16] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [21:01:03] done with the backport window [21:04:46] Any wikifunctions deployments happening now? If not, I’d like to get an update out for a security mitigation. [21:09:02] !log bking@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [21:09:12] !log bking@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [21:12:56] RECOVERY - MariaDB Replica Lag: s8 on db2198 is OK: OK slave_sql_lag Replication lag: 0.25 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:14:06] RECOVERY - MariaDB Replica Lag: s8 on db1285 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:16:42] (03PS3) 10Jdlrobson: Enable ReadingList CTA on Minerva for all reading list wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346053 (https://phabricator.wikimedia.org/T438779) [21:16:58] (03CR) 10Jdlrobson: [C:03+1] Update pulsating blue dot behavior for anonymous bookmark [extensions/ReadingLists] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1346793 (https://phabricator.wikimedia.org/T438779) (owner: 10Anne Tomasevich) [21:22:09] !log Deployed updated mitigation for T435455 [21:22:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:23:26] (03PS1) 10JHathaway: pybal: replace legacy fact [puppet] - 10https://gerrit.wikimedia.org/r/1346800 (https://phabricator.wikimedia.org/T372666) [21:23:44] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346800 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:24:36] 10ops-codfw, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439786 (10phaultfinder) 03NEW [21:26:56] PROBLEM - MariaDB Replica Lag: x3 on db2200 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 648.42 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:32:26] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1346660 (https://phabricator.wikimedia.org/T438926) (owner: 10Ayounsi) [21:33:38] (03PS2) 10Dzahn: zuul: decom VM zuul1001 as a main zuul, keep zuul2001 as is [puppet] - 10https://gerrit.wikimedia.org/r/1346139 (https://phabricator.wikimedia.org/T427353) [21:34:00] (03PS3) 10Dzahn: zuul: decom VM zuul1001 as a main zuul, keep zuul2001 as is [puppet] - 10https://gerrit.wikimedia.org/r/1346139 (https://phabricator.wikimedia.org/T427353) [21:36:52] RECOVERY - MariaDB Replica Lag: s1 on db1240 is OK: OK slave_sql_lag Replication lag: 0.33 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:41:56] RECOVERY - MariaDB Replica Lag: x3 on db2200 is OK: OK slave_sql_lag Replication lag: 0.42 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:48:04] PROBLEM - MariaDB Replica Lag: x3 on db1216 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 612.95 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:49:05] (03CR) 10Kamila Součková: "Why are you changing both experimental and mw-web-canary at the same time? (This is not a complaint, just asking for the reasoning.)" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346712 (https://phabricator.wikimedia.org/T438375) (owner: 10Herron) [21:54:08] RECOVERY - MariaDB Replica Lag: s1 on db2250 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:54:17] (03CR) 10JHathaway: "ready for review" [puppet] - 10https://gerrit.wikimedia.org/r/1346800 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [21:59:04] RECOVERY - MariaDB Replica Lag: x3 on db1216 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260930T2200) [22:13:32] PROBLEM - MariaDB Replica Lag: s4 on db1265 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 648.01 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:19:32] PROBLEM - MariaDB Replica Lag: s4 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 628.45 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:23:05] !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_codfw - 7.1.1-2~bpo13+wmf3 () [22:34:02] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [22:49:32] (03CR) 10Dzahn: [C:03+2] zuul: decom VM zuul1001 as a main zuul, keep zuul2001 as is [puppet] - 10https://gerrit.wikimedia.org/r/1346139 (https://phabricator.wikimedia.org/T427353) (owner: 10Dzahn) [22:50:38] (03CR) 10Herron: "To be honest I'm replaying the steps used on the last round of rsyslog updates (based on topic https://gerrit.wikimedia.org/r/q/topic:%22T" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346712 (https://phabricator.wikimedia.org/T438375) (owner: 10Herron) [22:57:58] (03PS3) 10Dzahn: zuul: decom VM zuul1002, replaced by zuul1005, keep zuul2002 [puppet] - 10https://gerrit.wikimedia.org/r/1346164 (https://phabricator.wikimedia.org/T427353) [23:34:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:37:50] (03PS1) 10Dzahn: wikistats: remove absented lxde and referata jobs and tables [puppet] - 10https://gerrit.wikimedia.org/r/1346814 (https://phabricator.wikimedia.org/T262148) [23:38:58] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1346815 [23:38:58] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1346815 (owner: 10TrainBranchBot) [23:42:46] (03PS1) 10Dzahn: wikistats: absent the update timers for gentoo and wikkii [puppet] - 10https://gerrit.wikimedia.org/r/1346817 [23:46:13] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1346815 (owner: 10TrainBranchBot) [23:47:13] (03PS2) 10Dzahn: wikistats: absent the update timers for gentoo and wikkii [puppet] - 10https://gerrit.wikimedia.org/r/1346817 [23:47:13] (03PS1) 10Dzahn: wikistats: remove absented gentoo and wikkii jobs and tables [puppet] - 10https://gerrit.wikimedia.org/r/1346819 [23:55:33] RECOVERY - MariaDB Replica Lag: s4 on db2239 is OK: OK slave_sql_lag Replication lag: 0.09 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response