[00:09:24] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job cfssl in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:31:29] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 641.07 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [01:09:04] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1352311 [01:09:04] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1352311 (owner: 10TrainBranchBot) [01:12:01] RECOVERY - MariaDB Replica Lag: s4 on db1265 is OK: OK slave_sql_lag Replication lag: 0.21 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [01:15:51] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1352311 (owner: 10TrainBranchBot) [01:29:49] (03PS1) 10Mamouri: fawiki: keep navigation boxes on the mobile site [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352316 (https://phabricator.wikimedia.org/T124168) [01:30:17] (03CR) 10Mamouri: [C:04-1] "Self -1 to prevent premature deployment: this is not ready to merge." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352316 (https://phabricator.wikimedia.org/T124168) (owner: 10Mamouri) [01:49:24] FIRING: KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1152.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T0200) [02:00:45] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:06:01] PROBLEM - MariaDB Replica Lag: s3 on db1265 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 648.96 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:08:20] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) [02:10:29] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [02:11:42] FIRING: [3x] JobUnavailable: Reduced availability for job cfssl in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] FIRING: [4x] JobUnavailable: Reduced availability for job cfssl in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:21:42] FIRING: [4x] JobUnavailable: Reduced availability for job cfssl in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:48:01] RECOVERY - MariaDB Replica Lag: s3 on db1265 is OK: OK slave_sql_lag Replication lag: 0.07 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [03:59:13] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-eqiad:et-1/1/2 (Transport: cr1-codfw:et-1/0/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [03:59:24] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [03:59:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [04:09:25] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:45:48] (03PS2) 10KartikMistry: Update Apertium to 2026-09-30-051826-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346288 (https://phabricator.wikimedia.org/T437213) [04:47:50] (03CR) 10KartikMistry: [C:03+2] Update Apertium to 2026-09-30-051826-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346288 (https://phabricator.wikimedia.org/T437213) (owner: 10KartikMistry) [04:49:13] (03Merged) 10jenkins-bot: Update Apertium to 2026-09-30-051826-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346288 (https://phabricator.wikimedia.org/T437213) (owner: 10KartikMistry) [04:49:59] Deploying Apertium; mostly minor changes. [04:50:31] !log kartik@deploy1003 helmfile [staging] START helmfile.d/services/apertium: apply [04:50:54] !log kartik@deploy1003 helmfile [staging] DONE helmfile.d/services/apertium: apply [04:50:57] !log Stop mariadb on x4 db2245 to clone the backup host [04:50:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:51:19] !log marostegui@cumin1004 START - Cookbook sre.mysql.depool depool db2245: to clone the backup host T437279 [04:51:21] T437279: Setup x4 backups - https://phabricator.wikimedia.org/T437279 [04:51:32] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2245: to clone the backup host T437279 [04:52:08] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2245.codfw.wmnet with reason: cloning [04:54:37] !log Stop mariadb on x4 db2245 to clone the backup host T437279 [04:54:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:59:51] (03PS1) 10Marostegui: db2250: Add x4 to backups [puppet] - 10https://gerrit.wikimedia.org/r/1352343 (https://phabricator.wikimedia.org/T437279) [05:00:31] !log kartik@deploy1003 helmfile [codfw] START helmfile.d/services/apertium: apply [05:01:06] !log kartik@deploy1003 helmfile [codfw] DONE helmfile.d/services/apertium: apply [05:04:03] !log kartik@deploy1003 helmfile [eqiad] START helmfile.d/services/apertium: apply [05:04:35] !log kartik@deploy1003 helmfile [eqiad] DONE helmfile.d/services/apertium: apply [05:08:22] !log oblivian@cumin1004 START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [05:08:26] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2067.codfw.wmnet [05:09:01] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2067.codfw.wmnet [05:09:12] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:10:36] !log Updated Apertium to 2026-09-30-051826-production (T437213) [05:10:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:10:39] T437213: Review and update LPL service base images – 2026Q3 - https://phabricator.wikimedia.org/T437213 [05:12:15] (03CR) 10Marostegui: mediabackups: add mediabackup_worker metrics (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1346031 (https://phabricator.wikimedia.org/T439526) (owner: 10CWilliams) [05:17:04] (03CR) 10Marostegui: [C:03+2] db2250: Add x4 to backups [puppet] - 10https://gerrit.wikimedia.org/r/1352343 (https://phabricator.wikimedia.org/T437279) (owner: 10Marostegui) [05:18:37] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2067.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:18:56] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2067.codfw.wmnet [05:19:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:20:47] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2067.codfw.wmnet [05:21:01] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2068.codfw.wmnet [05:21:35] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2068.codfw.wmnet [05:21:56] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:24:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:29:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 14.58% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:29:45] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2068.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:30:03] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2068.codfw.wmnet [05:31:00] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2068.codfw.wmnet [05:31:14] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2069.codfw.wmnet [05:31:47] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2069.codfw.wmnet [05:32:07] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:34:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:34:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 6.25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:34:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 20.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:39:25] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2069.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:39:44] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2069.codfw.wmnet [05:39:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 6.25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:39:46] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2069.codfw.wmnet [05:40:00] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2070.codfw.wmnet [05:40:35] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2070.codfw.wmnet [05:40:37] PROBLEM - mysqld processes on db2250 is CRITICAL: PROCS CRITICAL: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [05:40:47] ^ expected [05:40:54] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:41:07] it is a backup source so not critical or production [05:41:09] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning [05:42:50] (03PS1) 10Ayounsi: Revert "drain eqsin-codfw HE link" [homer/public] - 10https://gerrit.wikimedia.org/r/1352345 (https://phabricator.wikimedia.org/T439947) [05:44:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:48:13] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2070.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:48:32] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2070.codfw.wmnet [05:49:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:50:02] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2070.codfw.wmnet [05:50:15] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2071.codfw.wmnet [05:50:53] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2071.codfw.wmnet [05:51:14] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:51:27] 10ops-ulsfo, 06SRE, 06DC-Ops: ulsfo: OOB migration from copper to fiber - https://phabricator.wikimedia.org/T436499#12404654 (10ayounsi) Yup, Papaul is correct, no configuration change should be needed. Because of timezones it makes sens to me that Papaul takes point if there are any complication and netops... [05:54:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:54:14] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2250.codfw.wmnet with reason: cloning [05:59:02] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2071.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:59:21] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2071.codfw.wmnet [05:59:24] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T0600) [06:00:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 8.333% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:00:18] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2071.codfw.wmnet [06:00:19] (03CR) 10Arthur taylor: "this should not be deployed until the community announcement has been made and a week has passed." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347373 (https://phabricator.wikimedia.org/T437054) (owner: 10Sadiya.mohammed13) [06:00:32] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2072.codfw.wmnet [06:01:08] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2072.codfw.wmnet [06:01:28] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:04:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:05:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 16.67% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:09:21] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 12.5% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:09:54] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2072.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:10:12] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2072.codfw.wmnet [06:10:14] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2072.codfw.wmnet [06:10:28] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2073.codfw.wmnet [06:11:03] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2073.codfw.wmnet [06:11:23] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:13:15] 06SRE, 06Infrastructure-Foundations, 10netops: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#12404668 (10ayounsi) ulsfo too, with {T434068} [06:14:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:14:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 12.5% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:20:47] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2073.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:21:16] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2073.codfw.wmnet [06:21:18] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2073.codfw.wmnet [06:21:32] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2074.codfw.wmnet [06:21:42] FIRING: [2x] JobUnavailable: Reduced availability for job cfssl in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:22:07] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2074.codfw.wmnet [06:22:27] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:24:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:24:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 20.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:27:14] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2074.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:27:27] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1352166 (https://phabricator.wikimedia.org/T439143) (owner: 10Herron) [06:27:38] RECOVERY - mysqld processes on db2250 is OK: PROCS OK: 2 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [06:28:29] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1352258 (https://phabricator.wikimedia.org/T437741) (owner: 10Herron) [06:29:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 12.5% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:30:45] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 14.58% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:35:26] !log marostegui@cumin1004 START - Cookbook sre.mysql.pool pool db2245: Pooling back after cloning db2250 [06:36:26] (03PS1) 10Marostegui: site.pp: Update backups on db2250 [puppet] - 10https://gerrit.wikimedia.org/r/1352352 (https://phabricator.wikimedia.org/T437279) [06:36:40] (03CR) 10Marostegui: "This is a noop" [puppet] - 10https://gerrit.wikimedia.org/r/1352352 (https://phabricator.wikimedia.org/T437279) (owner: 10Marostegui) [06:38:01] (03CR) 10Marostegui: [C:03+2] site.pp: Update backups on db2250 [puppet] - 10https://gerrit.wikimedia.org/r/1352352 (https://phabricator.wikimedia.org/T437279) (owner: 10Marostegui) [06:38:11] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2074.codfw.wmnet [06:38:13] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2074.codfw.wmnet [06:38:27] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2075.codfw.wmnet [06:39:00] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2075.codfw.wmnet [06:39:09] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:39:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:39:32] (03CR) 10Arnaudb: "you're 100% right, that was going to be my approach." [puppet] - 10https://gerrit.wikimedia.org/r/1352124 (https://phabricator.wikimedia.org/T440177) (owner: 10Arnaudb) [06:40:45] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 6.25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:44:29] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2075.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:45:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:50:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:50:18] (03PS1) 10Muehlenhoff: Drop build2001 from profile::docker_registry::image_builders [puppet] - 10https://gerrit.wikimedia.org/r/1352354 [06:51:37] (03PS1) 10Muehlenhoff: Remove Puppet leftovers for build2001 [puppet] - 10https://gerrit.wikimedia.org/r/1352356 (https://phabricator.wikimedia.org/T417389) [06:55:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 18.75% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [06:55:27] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2075.codfw.wmnet [06:55:29] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2075.codfw.wmnet [06:55:43] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2076.codfw.wmnet [06:56:17] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2076.codfw.wmnet [06:56:28] !log installing bird security updates [06:56:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:56:37] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:56:42] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:59:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:00:04] Amir1, urbanecm, and awight: May I have your attention please! UTC morning backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T0700) [07:00:04] No Gerrit patches in the queue for this window AFAICS. [07:00:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 16.67% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:00:23] nicksbro_wmde and I will deploy a small patch, coming in a minute now... [07:01:01] (03PS1) 10Nicholusmuwonge: Adds metrics for entity usage limit baseline [extensions/Wikibase] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352357 (https://phabricator.wikimedia.org/T439125) [07:01:42] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:02:12] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, October 07 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/Wikibase] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352357 (https://phabricator.wikimedia.org/T439125) (owner: 10Nicholusmuwonge) [07:02:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 22.92% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:02:37] (03CR) 10Elukey: [C:03+1] ferm: Absent the NRPE check when migrating from ferm to nftables (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1283620 (owner: 10Muehlenhoff) [07:04:27] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2076.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:04:33] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2076.codfw.wmnet [07:05:37] !log elukey@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. [07:06:21] !log elukey@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. [07:06:47] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2076.codfw.wmnet [07:07:01] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2077.codfw.wmnet [07:07:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:08:07] !log elukey@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. [07:08:11] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2077.codfw.wmnet [07:08:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 20.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:08:28] !log elukey@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. [07:08:34] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:09:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:09:30] (03CR) 10DCausse: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352186 (https://phabricator.wikimedia.org/T440376) (owner: 10Ebernhardson) [07:11:43] !log elukey@deploy1003 helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. [07:12:29] !log elukey@deploy1003 helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. [07:12:51] Anyone around to kick a request for nicksbro_wmde to have spiderpig-access? [07:14:33] !log elukey@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. [07:14:57] !log elukey@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. [07:16:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2077.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:16:29] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2077.codfw.wmnet [07:16:32] !log elukey@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [07:17:18] !log elukey@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [07:18:10] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2077.codfw.wmnet [07:18:24] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2078.codfw.wmnet [07:18:27] !log elukey@deploy1003 helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [07:19:02] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2078.codfw.wmnet [07:19:12] !log elukey@deploy1003 helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [07:19:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:19:22] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:19:24] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:20:03] 06SRE, 06Collaboration-Services, 06Traffic: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12404758 (10LSobanski) [07:20:16] (03CR) 10DCausse: [C:03+1] cirrus: Attach X-Cluster-Name header to responses [puppet] - 10https://gerrit.wikimedia.org/r/1352186 (https://phabricator.wikimedia.org/T440376) (owner: 10Ebernhardson) [07:20:20] (03CR) 10TrainBranchBot: [C:03+2] "Approved by awight@deploy1003 using scap backport" [extensions/Wikibase] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352357 (https://phabricator.wikimedia.org/T439125) (owner: 10Nicholusmuwonge) [07:20:30] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2245: Pooling back after cloning db2250 [07:22:01] !log elukey@deploy1003 helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [07:22:22] !log elukey@deploy1003 helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [07:23:49] (03Merged) 10jenkins-bot: Adds metrics for entity usage limit baseline [extensions/Wikibase] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352357 (https://phabricator.wikimedia.org/T439125) (owner: 10Nicholusmuwonge) [07:24:13] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [07:24:24] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [07:24:39] 06SRE, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Site: EQIAD VM request for Kerberos - https://phabricator.wikimedia.org/T438229#12404771 (10LSobanski) Let's go ahead and create a VM, @bking are you OK to do that yourself? Please also release the bare metal host into DC Ops... [07:24:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [07:24:56] !log elukey@deploy1003 helmfile [eqiad] START helmfile.d/admin 'sync'. [07:25:07] !log awight@deploy1003 Started scap sync-world: Backport for [[gerrit:1352357|Adds metrics for entity usage limit baseline (T439125)]] [07:25:11] T439125: Track entity usage count when it exceeds the bundling limit - https://phabricator.wikimedia.org/T439125 [07:25:12] !log elukey@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'sync'. [07:26:00] !log elukey@deploy1003 helmfile [codfw] START helmfile.d/admin 'sync'. [07:26:37] !log elukey@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'sync'. [07:27:11] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2078.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:27:29] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2078.codfw.wmnet [07:27:31] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2078.codfw.wmnet [07:27:45] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2087.codfw.wmnet [07:28:19] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2087.codfw.wmnet [07:28:28] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:29:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:29:31] (03CR) 10Elukey: [C:03+2] admin_ng: move cfssl-issuer's settings to pki.discovery.wmnet [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351901 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [07:29:36] !log awight@deploy1003 nicholusmuwonge, awight: Backport for [[gerrit:1352357|Adds metrics for entity usage limit baseline (T439125)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:29:39] RESOLVED: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [07:29:59] !log uploaded openjdk-21 21.0.12.1+1-1~deb12u1 to component/jdk21 for bookworm-wikimedia T392465 [07:30:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:30:02] T392465: Switch Gerrit from Java 17 to Java 21 - https://phabricator.wikimedia.org/T392465 [07:33:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 22.92% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:33:50] !log elukey@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [07:33:56] !log elukey@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [07:34:28] !log elukey@deploy1003 helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. [07:34:33] !log elukey@deploy1003 helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. [07:35:21] !log elukey@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/admin 'sync'. [07:35:23] !log elukey@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'sync'. [07:36:54] !log elukey@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'sync'. [07:36:59] !log elukey@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'sync'. [07:37:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 22.92% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:37:33] !log elukey@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. [07:37:36] !log elukey@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. [07:38:27] !log elukey@deploy1003 helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [07:38:30] !log elukey@deploy1003 helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [07:39:02] !log elukey@deploy1003 helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [07:39:03] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2087.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:39:08] !log elukey@deploy1003 helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [07:39:23] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2087.codfw.wmnet [07:39:24] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2087.codfw.wmnet [07:39:24] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:39:38] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2088.codfw.wmnet [07:39:39] !log elukey@deploy1003 helmfile [codfw] START helmfile.d/admin 'sync'. [07:39:43] !log elukey@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'sync'. [07:40:12] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2088.codfw.wmnet [07:40:15] !log elukey@deploy1003 helmfile [eqiad] START helmfile.d/admin 'sync'. [07:40:17] !log elukey@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'sync'. [07:41:49] !log awight@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-experimental: apply [07:42:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 22.92% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:42:26] !log awight@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply [07:44:39] oblivian@cumin1004 enable_virtualization (PID 3369636) is awaiting input [07:45:10] FIRING: [2x] BFDdown: BFD session down between cr1-codfw and fe80::669:8f07:ece:f4e7 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [07:45:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 20.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:47:22] (03CR) 10Cathal Mooney: [C:03+1] "Seems clean enough over past week yep." [homer/public] - 10https://gerrit.wikimedia.org/r/1352345 (https://phabricator.wikimedia.org/T439947) (owner: 10Ayounsi) [07:48:53] (03PS1) 10Elukey: Revert "profile::pki::client: move the signer endpoint to pki[12]002" [puppet] - 10https://gerrit.wikimedia.org/r/1352442 [07:49:11] (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352442 (owner: 10Elukey) [07:50:10] RESOLVED: [2x] BFDdown: BFD session down between cr1-codfw and fe80::669:8f07:ece:f4e7 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [07:50:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 20.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [07:50:18] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2088.codfw.wmnet [07:50:20] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2088.codfw.wmnet [07:50:34] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2089.codfw.wmnet [07:50:44] (03CR) 10Cathal Mooney: [C:03+1] "We had bad loss last Thursday on it, and a bit of loss yesterday. I'd be 50/50 on repooling tbh, given the normal bandwidth usage in magr" [homer/public] - 10https://gerrit.wikimedia.org/r/1348476 (owner: 10Ayounsi) [07:51:08] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2089.codfw.wmnet [07:51:31] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:51:57] (03CR) 10Cathal Mooney: [C:03+1] "This was affected by the same saturated path as codfw -> drmrs it seems, and largely ok for the past week so I guess yeah let's give it an" [homer/public] - 10https://gerrit.wikimedia.org/r/1348471 (owner: 10Ayounsi) [07:56:55] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2089.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:57:08] (03CR) 10Ayounsi: [C:03+2] Revert "drain eqsin-codfw HE link" [homer/public] - 10https://gerrit.wikimedia.org/r/1352345 (https://phabricator.wikimedia.org/T439947) (owner: 10Ayounsi) [07:57:27] !log awight@deploy1003 nicholusmuwonge, awight: Continuing with deployment [07:57:51] Tested metrics change on mw-experimental [08:00:04] andre and brennen: #bothumor I � Unicode. All rise for MediaWiki train - Utc-0+Utc-7 Version deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T0800). [08:00:06] o/ [08:00:26] andre: Hi! We're just finishing a scap, sorry for the interference [08:00:40] awight: ah, no worries, I'll wait! :) [08:01:15] It was my first experience testing a jobrunner on mw-experimental... quite nice once you know what to do but the learning curve was steep for me [08:02:09] (03PS1) 10Elukey: pki: remove restriction for /metrics in the httpd vhost [puppet] - 10https://gerrit.wikimedia.org/r/1352444 [08:02:49] !log awight@deploy1003 Finished scap sync-world: Backport for [[gerrit:1352357|Adds metrics for entity usage limit baseline (T439125)]] (duration: 37m 41s) [08:02:52] T439125: Track entity usage count when it exceeds the bundling limit - https://phabricator.wikimedia.org/T439125 [08:03:00] ehehe [08:03:14] andre: good luck! [08:03:36] awight: oh, you're done? alright [08:03:51] oh yes, now I see the log entry above [08:04:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:07:52] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2089.codfw.wmnet [08:07:53] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2089.codfw.wmnet [08:08:07] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2090.codfw.wmnet [08:08:42] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2090.codfw.wmnet [08:09:02] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:09:05] (03PS1) 10TrainBranchBot: group1 to 1.47.0-wmf.23 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352447 (https://phabricator.wikimedia.org/T438219) [08:09:06] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by aklapper@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352447 (https://phabricator.wikimedia.org/T438219) (owner: 10TrainBranchBot) [08:09:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:09:25] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:10:00] (03Merged) 10jenkins-bot: group1 to 1.47.0-wmf.23 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352447 (https://phabricator.wikimedia.org/T438219) (owner: 10TrainBranchBot) [08:14:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 22.92% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:14:19] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2090.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:17:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:17:39] FIRING: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [08:19:13] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [08:20:34] !log aklapper@deploy1003 Rolling back deployment [08:21:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-jobrunner - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [08:21:45] Train blows up, so I'm rolling back [08:22:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 19.64% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:22:35] andre: eek, I was just going to ask if I could squeeze in a backport for T440377.... what blows up? [08:22:36] T440377: InvalidArgumentException: Scalar parameter must be a string, number, Stringable, or MessageSpecifier; got array - https://phabricator.wikimedia.org/T440377 [08:22:39] FIRING: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [08:26:15] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [08:26:49] (03PS3) 10Elukey: kube-reload-certs: allow different labels then "release" [puppet] - 10https://gerrit.wikimedia.org/r/1351965 (https://phabricator.wikimedia.org/T439409) [08:27:27] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2090.codfw.wmnet [08:27:29] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2090.codfw.wmnet [08:27:37] (03PS4) 10Elukey: kube-reload-certs: allow different labels then "release" [puppet] - 10https://gerrit.wikimedia.org/r/1351965 (https://phabricator.wikimedia.org/T439409) [08:27:43] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2091.codfw.wmnet [08:28:17] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2091.codfw.wmnet [08:28:37] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:29:12] !log aklapper@deploy1003 rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs T438219 [08:29:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:29:15] T438219: 1.47.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T438219 [08:29:54] (03CR) 10CI reject: [V:04-1] kube-reload-certs: allow different labels then "release" [puppet] - 10https://gerrit.wikimedia.org/r/1351965 (https://phabricator.wikimedia.org/T439409) (owner: 10Elukey) [08:30:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 16.67% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:30:36] (03PS5) 10Elukey: kube-reload-certs: allow different labels then "release" [puppet] - 10https://gerrit.wikimedia.org/r/1351965 (https://phabricator.wikimedia.org/T439409) [08:31:15] RESOLVED: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [08:33:33] duesen: you can soon backport, give me a few more minutes to sort out things. thanks [08:33:34] (03CR) 10Ayounsi: [C:03+2] un-drain esams-codfw HE link [homer/public] - 10https://gerrit.wikimedia.org/r/1348471 (owner: 10Ayounsi) [08:33:52] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2091.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:34:44] (03Merged) 10jenkins-bot: un-drain esams-codfw HE link [homer/public] - 10https://gerrit.wikimedia.org/r/1348471 (owner: 10Ayounsi) [08:38:10] (03CR) 10Elukey: kube-reload-certs: allow different labels then "release" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1351965 (https://phabricator.wikimedia.org/T439409) (owner: 10Elukey) [08:39:41] (03PS2) 10Arnaudb: gerrit: limit the volume of returnable pages [puppet] - 10https://gerrit.wikimedia.org/r/1308081 (https://phabricator.wikimedia.org/T431398) [08:40:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 8.333% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:41:36] duesen: Logstash calm again after rollback; train currently blocked on T440429. So you can go ahead and run a backport [08:41:37] T440429: Error: Call to a member function getEntityId() on null - https://phabricator.wikimedia.org/T440429 [08:43:06] (03PS1) 10Filippo Giunchedi: openstack: enable performance governor in nova::common [puppet] - 10https://gerrit.wikimedia.org/r/1352452 (https://phabricator.wikimedia.org/T439729) [08:43:50] (03CR) 10Filippo Giunchedi: "My best guess at which profile makes sense, please let me know if I am off base here" [puppet] - 10https://gerrit.wikimedia.org/r/1352452 (https://phabricator.wikimedia.org/T439729) (owner: 10Filippo Giunchedi) [08:44:14] (03CR) 10Jgiannelos: [C:03+2] mobileapps: Disable debugging on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346017 (owner: 10Jgiannelos) [08:45:16] (03CR) 10Ayounsi: [C:04-1] "I'm fine pushing it to later" [homer/public] - 10https://gerrit.wikimedia.org/r/1348476 (owner: 10Ayounsi) [08:45:19] (03Abandoned) 10Jgiannelos: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342233 (owner: 10PipelineBot) [08:45:19] (03Abandoned) 10Jgiannelos: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320213 (owner: 10PipelineBot) [08:45:20] (03Abandoned) 10Jgiannelos: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327560 (owner: 10PipelineBot) [08:45:34] (03CR) 10Dpogorzelski: [C:03+2] logstash: map Hummingbird logs to ECS [puppet] - 10https://gerrit.wikimedia.org/r/1352072 (https://phabricator.wikimedia.org/T440198) (owner: 10Dpogorzelski) [08:45:45] (03Merged) 10jenkins-bot: mobileapps: Disable debugging on staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346017 (owner: 10Jgiannelos) [08:46:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:49:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:49:24] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2091.codfw.wmnet [08:49:25] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2091.codfw.wmnet [08:49:39] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2092.codfw.wmnet [08:50:14] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2092.codfw.wmnet [08:50:37] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:50:38] (03CR) 10Giuseppe Lavagetto: [C:03+1] kube-reload-certs: allow different labels then "release" [puppet] - 10https://gerrit.wikimedia.org/r/1351965 (https://phabricator.wikimedia.org/T439409) (owner: 10Elukey) [08:51:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-wikifunctions releases routed via group1 at eqiad: 25% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-wikifunctions&var-container_name=All&var-release=group1 - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [08:52:17] (03CR) 10Elukey: [C:03+2] kube-reload-certs: allow different labels then "release" [puppet] - 10https://gerrit.wikimedia.org/r/1351965 (https://phabricator.wikimedia.org/T439409) (owner: 10Elukey) [08:53:12] andre: ok, thanks, I'll do that! I'll have a passenger, so it may take a bit longer. And it'S a mediawiki patch, so we have to account for CI... I'll get started. [08:53:32] duesen, do it. good luck :D [08:55:35] !log aqu@deploy1003 Started deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33] [08:55:54] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2092.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:56:19] !log aqu@deploy1003 Finished deploy [analytics/refinery@43a90b3] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@43a90b33] (duration: 00m 44s) [08:56:51] (03CR) 10Btullis: [C:03+2] dumps: web: Add a brownout for v1 meta-current and meta-history [puppet] - 10https://gerrit.wikimedia.org/r/1346007 (https://phabricator.wikimedia.org/T437461) (owner: 10Btullis) [08:57:31] !log aqu@deploy1003 Started deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33] [09:02:13] !log aqu@deploy1003 Finished deploy [analytics/refinery@43a90b3]: Regular analytics weekly train [analytics/refinery@43a90b33] (duration: 04m 42s) [09:02:39] RESOLVED: [2x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [09:04:13] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [09:04:18] (03PS1) 10Daniel Kinzler: REST: Accept message objects as parameter help [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352455 (https://phabricator.wikimedia.org/T433534) [09:04:37] (03PS3) 10Arnaudb: gerrit: cap index query results at 5000 [puppet] - 10https://gerrit.wikimedia.org/r/1308081 (https://phabricator.wikimedia.org/T431398) [09:04:58] (03PS1) 10Seanleong-wmde: Group fix for incorrect group mapping. [extensions/SiteMatrix] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352456 (https://phabricator.wikimedia.org/T439067) [09:05:09] (03CR) 10Arnaudb: "Done" [puppet] - 10https://gerrit.wikimedia.org/r/1308081 (https://phabricator.wikimedia.org/T431398) (owner: 10Arnaudb) [09:05:18] (03CR) 10Arnaudb: "Done" [puppet] - 10https://gerrit.wikimedia.org/r/1308081 (https://phabricator.wikimedia.org/T431398) (owner: 10Arnaudb) [09:05:48] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, October 07 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [extensions/SiteMatrix] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352456 (https://phabricator.wikimedia.org/T439067) (owner: 10Seanleong-wmde) [09:06:00] (03CR) 10TrainBranchBot: [C:03+2] "Approved by daniel@deploy1003 using scap backport" [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352455 (https://phabricator.wikimedia.org/T433534) (owner: 10Daniel Kinzler) [09:07:28] (03CR) 10Muehlenhoff: [C:03+1] "Looks good. If anyone is equally confused as I initially was; multirootca only listens on localhost, but there's another proxy pass config" [puppet] - 10https://gerrit.wikimedia.org/r/1352444 (owner: 10Elukey) [09:09:18] (03CR) 10Elukey: [C:03+2] pki: remove restriction for /metrics in the httpd vhost [puppet] - 10https://gerrit.wikimedia.org/r/1352444 (owner: 10Elukey) [09:12:43] (03Merged) 10jenkins-bot: REST: Accept message objects as parameter help [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352455 (https://phabricator.wikimedia.org/T433534) (owner: 10Daniel Kinzler) [09:13:05] !log daniel@deploy1003 Started scap sync-world: Backport for [[gerrit:1352455|REST: Accept message objects as parameter help (T433534 T440377)]] [09:13:09] T433534: [Hypothesis] PP3.5.2 REST API v2 Beta Launch - https://phabricator.wikimedia.org/T433534 [09:13:10] T440377: InvalidArgumentException: Scalar parameter must be a string, number, Stringable, or MessageSpecifier; got array - https://phabricator.wikimedia.org/T440377 [09:14:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [09:14:20] oblivian@cumin1004 enable_virtualization (PID 3369636) is awaiting input [09:17:20] !log daniel@deploy1003 daniel: Backport for [[gerrit:1352455|REST: Accept message objects as parameter help (T433534 T440377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [09:19:49] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2092.codfw.wmnet [09:19:50] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2092.codfw.wmnet [09:20:05] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2093.codfw.wmnet [09:20:39] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2093.codfw.wmnet [09:20:59] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:22:34] andre: I'm trying to test my patch using the debug extension, but I don't see the effect. I had the same issue yesterday. I noticed that we are supposed to use mw-experimental instead of wmdebug now... but that doesn't seem to help. [09:22:47] Do you know what the problem may be? [09:23:07] Point in case, https://test.wikipedia.org/w/rest.php/specs/v0/module/content/v2-beta should no longer fail on the test server. [09:23:50] RECOVERY - Confd vcl based reload on cp6010 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [09:26:16] hm, the server response header on that URL says mw-api-ext.eqiad.main-6bc8b85d69-94w89 for me, which is definitely not a k8s-mwdebug server… [09:26:31] !log restarting pybal on lvs1018 T439730 [09:26:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:26:34] T439730: Add a new LVS ldap-ro-test service - https://phabricator.wikimedia.org/T439730 [09:27:12] RECOVERY - Check if Pybal has been restarted after pybal.conf was changed on lvs1018 is OK: OK: pybal.service was restarted after /etc/pybal/pybal.conf was changed. https://wikitech.wikimedia.org/wiki/PyBal%23Pybal_service_has_not_been_restarted [09:28:45] (03PS1) 10Santiago Faci: Deploy GrowthBook to staging based on a specific commit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352461 (https://phabricator.wikimedia.org/T439990) [09:28:52] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2093.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:29:03] Lucas_WMDE: let me check the details on that [09:29:11] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2093.codfw.wmnet [09:29:24] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [09:29:46] RECOVERY - PyBal connections to etcd on lvs1018 is OK: OK: 26 connections established with conf1007.eqiad.wmnet:4001 (min=26) https://wikitech.wikimedia.org/wiki/PyBal [09:29:46] RECOVERY - PyBal IPVS diff check on lvs1018 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [09:30:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.958% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [09:30:10] targeting mw-debug works for me on https://en.wikipedia.org/wiki/Special:Version though, at least according to the server response header [09:30:26] same on https://test.wikipedia.org/wiki/Special:Version [09:30:33] is /w/rest.php routed differently? [09:31:13] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2093.codfw.wmnet [09:31:27] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2094.codfw.wmnet [09:31:33] (03CR) 10Blake: [C:03+2] mw-experimental: enable native_sidecars on mw-experimental [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333740 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [09:32:01] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2094.codfw.wmnet [09:32:03] I get server: mw-experimental.eqiad.pinkllama-5c957f9ff7-kkqfh if I send X-Wikimedia-Debug: backend=k8s-mw-experimental-eqiad [09:32:09] Lucas_WMDE: --^ [09:32:21] So that should have the patch, right? [09:32:22] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:32:31] no, mw-experimental is totally different from mwdebug afaik [09:33:02] (03Merged) 10jenkins-bot: mw-experimental: enable native_sidecars on mw-experimental [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333740 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [09:33:08] it’s updated from the other deployment servers hourly (I think), and beyond that only has changes you make manually [09:33:31] Lucas_WMDE: https://wikitech.wikimedia.org/wiki/Mwdebug says the mwdenug is deprecated, and mw-experimental should be used instead. [09:34:03] duesen: that’s specifically for the purpose of ad-hoc testing [09:34:09] (should probably be phrased better on-wiki) [09:34:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [09:34:23] and it’s also referring to the bare-metal mwdebug hosts (that no longer exist) [09:34:38] for deployment with scap, you use k8s-mwdebug, which replaced bare-metal mwdebug for that purpose [09:34:39] Lucas_WMDE: that's REALLY confusing :) [09:35:20] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-experimental: apply [09:35:43] Anyway, If I pick mwdebug in the browser extension, i get server: mw-api-ext.eqiad.main-6bc8b85d69-fp46w, as you said. I guess the debug routing just doesn't work for APIs? [09:35:50] That's not great... [09:35:57] it definitely used to work for the action API [09:36:13] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply [09:36:17] I'll go ahead with the deployment, I'm confident enough that it doesn't break anythign else. It's an a class that's only used in a beta feature that's only enabled on testwiki [09:36:35] (still works for api.php, actually idk if it ever worked for rest.php or if this is a recent regression) [09:36:36] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: log JSON for ECS [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352073 (https://phabricator.wikimedia.org/T440198) (owner: 10Dpogorzelski) [09:36:36] !log daniel@deploy1003 daniel: Continuing with deployment [09:37:22] (that was supposed to be two sentences, oops) [09:37:47] Lucas_WMDE: huh, indeed - w/api.php gives me mw-debug.eqiad.pinkunicorn-64784bf8fc-6bmzt [09:37:59] (03Merged) 10jenkins-bot: liftwing-studio: log JSON for ECS [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352073 (https://phabricator.wikimedia.org/T440198) (owner: 10Dpogorzelski) [09:38:47] index.php also works right [09:39:16] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [09:39:50] 10ops-eqiad, 06SRE, 06DC-Ops: krb1002 booting up extremely slow - https://phabricator.wikimedia.org/T435354#12405266 (10MoritzMuehlenhoff) a:05VRiley-WMF→03bking [09:40:33] claime: do you happen to know if “x-wikimedia-debug has no effect on rest.php” is a known issue? ^^ (closest I found on phab is T428909) [09:40:34] T428909: Investigate X-Wikimedia-Debug routing for APIs going through the rest-gateway - https://phabricator.wikimedia.org/T428909 [09:40:34] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [09:41:29] 06SRE, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Site: EQIAD VM request for Kerberos - https://phabricator.wikimedia.org/T438229#12405275 (10MoritzMuehlenhoff) >>! In T438229#12404771, @LSobanski wrote: > Please also release the bare metal host into DC Ops inventory if not... [09:42:16] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2094.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:42:34] (03CR) 10CWilliams: mediabackups: add mediabackup_worker metrics (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1346031 (https://phabricator.wikimedia.org/T439526) (owner: 10CWilliams) [09:42:46] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2094.codfw.wmnet [09:42:48] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2094.codfw.wmnet [09:42:48] PROBLEM - Confd vcl based reload on cp6010 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [09:43:02] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2095.codfw.wmnet [09:43:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-jobrunner - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [09:43:36] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2095.codfw.wmnet [09:43:56] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:44:01] (03PS5) 10CWilliams: mediabackups: add mediabackup_worker metrics [puppet] - 10https://gerrit.wikimedia.org/r/1346031 (https://phabricator.wikimedia.org/T439526) [09:44:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [09:44:45] (03CR) 10CWilliams: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346031 (https://phabricator.wikimedia.org/T439526) (owner: 10CWilliams) [09:46:54] 06SRE, 06Traffic, 10WikimediaDebug: Debug server routing does not work for /w/rest.php - https://phabricator.wikimedia.org/T440439 (10daniel) 03NEW [09:47:22] Lucas_WMDE: I filed https://phabricator.wikimedia.org/T440439 [09:47:26] Hitting a different issue now [09:47:36] duesen: thanks! [09:47:41] (03CR) 10Ozge: "Kindly reminder" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352020 (https://phabricator.wikimedia.org/T440266) (owner: 10Ozge) [09:48:15] FIRING: [5x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [09:49:50] andre: My patch is hitting the getEntityId on null error. I will ignore that and push on with the backport. [09:50:24] !log daniel@deploy1003 Finished scap sync-world: Backport for [[gerrit:1352455|REST: Accept message objects as parameter help (T433534 T440377)]] (duration: 37m 19s) [09:50:28] T433534: [Hypothesis] PP3.5.2 REST API v2 Beta Launch - https://phabricator.wikimedia.org/T433534 [09:50:28] T440377: InvalidArgumentException: Scalar parameter must be a string, number, Stringable, or MessageSpecifier; got array - https://phabricator.wikimedia.org/T440377 [09:50:41] ack [09:51:45] (03CR) 10Ilias Sarantopoulos: [C:03+1] ml-services: Add qwen36-27b-experiments and reduce staging resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352020 (https://phabricator.wikimedia.org/T440266) (owner: 10Ozge) [09:53:09] andre: confirmed working, backport done [09:53:15] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [09:53:21] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2095.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:53:27] (03CR) 10Ozge: [C:03+2] ml-services: Add qwen36-27b-experiments and reduce staging resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352020 (https://phabricator.wikimedia.org/T440266) (owner: 10Ozge) [09:53:36] duesen, thanks! [09:53:40] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2095.codfw.wmnet [09:54:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [09:54:16] !log just a test for SAL [09:54:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:54:56] (03Merged) 10jenkins-bot: ml-services: Add qwen36-27b-experiments and reduce staging resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352020 (https://phabricator.wikimedia.org/T440266) (owner: 10Ozge) [09:55:20] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2095.codfw.wmnet [09:55:34] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2102.codfw.wmnet [09:56:08] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2102.codfw.wmnet [09:56:28] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:58:15] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [09:59:01] (03PS1) 10Blake: mediawiki: update module versions to support native sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352468 (https://phabricator.wikimedia.org/T417800) [09:59:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [09:59:48] RECOVERY - Confd vcl based reload on cp6010 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [10:00:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.333% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [10:00:04] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T1000) [10:00:50] (03PS1) 10Dpogorzelski: liftwing-studio: log LiteLLM warnings and above [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352469 (https://phabricator.wikimedia.org/T440198) [10:01:16] (03CR) 10Clément Goubert: [C:03+2] wikikube: Add lac-lambdas namespace [puppet] - 10https://gerrit.wikimedia.org/r/1347649 (https://phabricator.wikimedia.org/T439879) (owner: 10Clément Goubert) [10:02:22] (03PS1) 10Giuseppe Lavagetto: jaeger: add release label to query pod and certificate [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352471 [10:05:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2102.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:05:41] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2102.codfw.wmnet [10:07:02] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: log LiteLLM warnings and above [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352469 (https://phabricator.wikimedia.org/T440198) (owner: 10Dpogorzelski) [10:07:14] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [10:07:41] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [10:07:54] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2102.codfw.wmnet [10:08:08] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2103.codfw.wmnet [10:08:34] (03CR) 10Phuedx: [C:03+1] Deploy GrowthBook to staging based on a specific commit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352461 (https://phabricator.wikimedia.org/T439990) (owner: 10Santiago Faci) [10:09:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:09:17] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2103.codfw.wmnet [10:09:37] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:09:40] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:11:00] FIRING: PingLossPercent: Blackbox probe packet loss 6.25% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [10:12:43] (03CR) 10Santiago Faci: [C:03+2] Deploy GrowthBook to staging based on a specific commit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352461 (https://phabricator.wikimedia.org/T439990) (owner: 10Santiago Faci) [10:14:10] (03Merged) 10jenkins-bot: Deploy GrowthBook to staging based on a specific commit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352461 (https://phabricator.wikimedia.org/T439990) (owner: 10Santiago Faci) [10:14:58] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2103.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:15:06] (03CR) 10Federico Ceratto: team-data-persistence: Alarm on stale data in Zarcillo (032 comments) [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) (owner: 10Federico Ceratto) [10:16:24] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [10:16:34] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply [10:16:56] (03PS1) 10Muehlenhoff: service catalogue: Set ldap-ro-test/ldap-ro-test-ssl to production [puppet] - 10https://gerrit.wikimedia.org/r/1352474 (https://phabricator.wikimedia.org/T439730) [10:17:26] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply [10:19:29] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [10:20:52] (03PS1) 10Vgutierrez: varnish: Set a explicit timeout for varnishadm on reload-vcl [puppet] - 10https://gerrit.wikimedia.org/r/1352477 (https://phabricator.wikimedia.org/T440441) [10:21:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.542% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [10:24:17] (03PS1) 10Muehlenhoff: Revert "Add explicit replica hostnames to LDAP cert SNIs" [puppet] - 10https://gerrit.wikimedia.org/r/1352478 [10:24:34] (03PS2) 10Muehlenhoff: Revert "Add explicit replica hostnames to LDAP cert SNIs" [puppet] - 10https://gerrit.wikimedia.org/r/1352478 [10:25:12] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [10:26:06] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [10:26:24] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [10:26:34] (03CR) 10Clément Goubert: [C:03+1] mediawiki: update module versions to support native sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352468 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:27:00] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2103.codfw.wmnet [10:27:01] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2103.codfw.wmnet [10:27:15] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2104.codfw.wmnet [10:27:49] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2104.codfw.wmnet [10:28:09] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:28:58] (03CR) 10Blake: [C:03+2] mediawiki: update module versions to support native sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352468 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:29:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:30:16] (03CR) 10Fabfur: [C:03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1352477 (https://phabricator.wikimedia.org/T440441) (owner: 10Vgutierrez) [10:30:31] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [10:30:53] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [10:31:14] (03Merged) 10jenkins-bot: mediawiki: update module versions to support native sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352468 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:32:38] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [10:33:24] !log blake@deploy1003 Started scap sync-world: update vendored modules for T417800 [10:33:27] T417800: Implement proper sidecar container support in mediawiki pods - https://phabricator.wikimedia.org/T417800 [10:33:33] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2104.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:35:09] (03PS4) 10Blake: mw-experimental: declare mesh, mcrouter, statsd as native k8s sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352466 (https://phabricator.wikimedia.org/T417800) [10:39:55] !log blake@deploy1003 Finished scap sync-world: update vendored modules for T417800 (duration: 07m 32s) [10:39:58] T417800: Implement proper sidecar container support in mediawiki pods - https://phabricator.wikimedia.org/T417800 [10:44:30] (03CR) 10Blake: "recheck" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352466 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:44:30] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2104.codfw.wmnet [10:44:32] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2104.codfw.wmnet [10:44:46] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2105.codfw.wmnet [10:45:20] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2105.codfw.wmnet [10:45:28] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:47:26] (03PS5) 10Blake: mw-experimental: declare mesh, mcrouter, statsd as native k8s sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352466 (https://phabricator.wikimedia.org/T417800) [10:48:06] (03CR) 10Hashar: "recheck with new CI image (Trixie/Bazelisk upgraded etc)" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1350527 (https://phabricator.wikimedia.org/T437620) (owner: 10Hashar) [10:48:15] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [10:48:16] (03CR) 10Vgutierrez: [C:03+2] varnish: Set a explicit timeout for varnishadm on reload-vcl [puppet] - 10https://gerrit.wikimedia.org/r/1352477 (https://phabricator.wikimedia.org/T440441) (owner: 10Vgutierrez) [10:49:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:49:27] 06SRE, 10Observability-Alerting, 13Patch-For-Review, 07Sustainability (Incident Followup): Paging alert on combination of hosts down and a BGP outage - https://phabricator.wikimedia.org/T417051#12405513 (10hnowlan) 05Open→03Declined Seems like the work described in this ticket as a wontfix given th... [10:49:41] (03Abandoned) 10Hnowlan: team-netops: create alert for combined impact of BGP impact and probe failure [alerts] - 10https://gerrit.wikimedia.org/r/1352138 (https://phabricator.wikimedia.org/T417051) (owner: 10Hnowlan) [10:50:54] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2105.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:50:55] (03CR) 10Blake: "My understanding from conversations with Reuven is that we might want to be cautious about when we perform maintenance operations on memca" [puppet] - 10https://gerrit.wikimedia.org/r/1352155 (owner: 10Muehlenhoff) [10:53:15] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [10:55:11] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.upgrade restart P{lvs7003.magru.wmnet} and A:liberica [10:55:26] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin depooling P{lvs7003.magru.wmnet} and A:liberica [10:55:37] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7003.magru.wmnet} and A:liberica [10:55:50] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin pooling P{lvs7003.magru.wmnet} and A:liberica [10:55:57] !log vgutierrez@cumin1004 END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P{lvs7003.magru.wmnet} and A:liberica [10:56:10] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7003.magru.wmnet} and A:liberica [10:58:15] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [10:59:38] (03PS1) 10Btullis: apereo_cas: Add a client_roles parameter for services [puppet] - 10https://gerrit.wikimedia.org/r/1352484 (https://phabricator.wikimedia.org/T435602) [10:59:40] (03PS1) 10Btullis: idp_test: Accept wmf_s3_login requests only from the Ceph admin hosts [puppet] - 10https://gerrit.wikimedia.org/r/1352485 (https://phabricator.wikimedia.org/T435602) [10:59:41] (03PS1) 10Btullis: idp_test: Remove required_groups from the wmf_s3_login client [puppet] - 10https://gerrit.wikimedia.org/r/1352486 (https://phabricator.wikimedia.org/T435602) [11:00:05] mvolz: That opportune time for a Services – Citoid / Zotero deploy is upon us again. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T1100). [11:00:48] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352484 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [11:01:42] FIRING: [2x] JobUnavailable: Reduced availability for job cfssl in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:02:04] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs7003.magru.wmnet} and A:liberica [11:02:23] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs7003.magru.wmnet} and A:liberica [11:02:52] andre, duesen: there's a significant spike in getEntityId errors on mw-api-ext after your change rollout - is there context on that error? I see you talked about it earlier [11:03:12] https://logstash.wikimedia.org/goto/c7db2430684221844f9e00784109ce34 [11:03:13] duesen: I guess I misinterpreted your line above. :( Error rate is pretty bad again now. [11:04:02] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2105.codfw.wmnet [11:04:04] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2105.codfw.wmnet [11:04:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:04:17] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2106.codfw.wmnet [11:04:47] hnowlan: yeah I am not sure what exactly is going on. I did a rollback for group1 from .23 to .22 in https://phabricator.wikimedia.org/T438219#12405010 which was bumpy, then daniel did a backport of https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1352455, and now https://versions.toolforge.org/ says that group1 is back in .23 [11:04:51] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2106.codfw.wmnet [11:05:12] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:06:56] duesen: around? [11:08:15] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [11:08:17] I think I'm going to roll back yet again [11:08:23] let me do that now [11:08:46] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.upgrade restart P{lvs7003.magru.wmnet} and A:liberica [11:09:01] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin depooling P{lvs7003.magru.wmnet} and A:liberica [11:09:13] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7003.magru.wmnet} and A:liberica [11:09:26] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin pooling P{lvs7003.magru.wmnet} and A:liberica [11:09:33] !log Rolling back again wmf.23 from group1 to group0 - T438219 [11:09:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:09:35] T438219: 1.47.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T438219 [11:09:48] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7003.magru.wmnet} and A:liberica [11:09:51] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7003.magru.wmnet} and A:liberica [11:09:53] (03PS1) 10TrainBranchBot: group0 to 1.47.0-wmf.23 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352488 (https://phabricator.wikimedia.org/T438219) [11:09:54] (03CR) 10Muehlenhoff: "Then the memcached-all alias should not be used in that situation :-) But there are definitely use cases (e.g. with debdeploy where it mak" [puppet] - 10https://gerrit.wikimedia.org/r/1352155 (owner: 10Muehlenhoff) [11:09:55] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by aklapper@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352488 (https://phabricator.wikimedia.org/T438219) (owner: 10TrainBranchBot) [11:10:32] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2106.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:10:43] (03Merged) 10jenkins-bot: group0 to 1.47.0-wmf.23 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352488 (https://phabricator.wikimedia.org/T438219) (owner: 10TrainBranchBot) [11:13:15] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [11:15:08] (03PS1) 10Cathal Mooney: LVS Realserver IPIP: add default values for source IPIP ranges [puppet] - 10https://gerrit.wikimedia.org/r/1352489 (https://phabricator.wikimedia.org/T440201) [11:16:44] (03CR) 10Vgutierrez: "the alternative would be using `hieradata/cloud.yaml`" [puppet] - 10https://gerrit.wikimedia.org/r/1352489 (https://phabricator.wikimedia.org/T440201) (owner: 10Cathal Mooney) [11:18:24] PROBLEM - Host wikikube-worker1034 is DOWN: PING CRITICAL - Packet loss = 50%, RTA = 3184.63 ms [11:18:28] RECOVERY - Host wikikube-worker1034 is UP: PING WARNING - Packet loss = 33%, RTA = 911.40 ms [11:19:04] (03PS1) 10Santiago Faci: Deploy GrowthBook to production based on a specific commit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352490 (https://phabricator.wikimedia.org/T439990) [11:21:04] !log aklapper@deploy1003 rebuilt and synchronized wikiversions files: group0 to 1.47.0-wmf.23 refs T438219 [11:21:07] T438219: 1.47.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T438219 [11:21:30] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2106.codfw.wmnet [11:21:32] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2106.codfw.wmnet [11:21:46] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2107.codfw.wmnet [11:22:19] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2107.codfw.wmnet [11:22:28] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:23:15] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [11:23:44] (03CR) 10Aklapper: "After this got backported, group1 got back to wmf.23 and its high error rate, so I rolled back the train a second time on group0 to wmf.22" [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352455 (https://phabricator.wikimedia.org/T433534) (owner: 10Daniel Kinzler) [11:24:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:24:47] (03CR) 10Blake: [C:03+1] "Understood, thanks for the additional context :)" [puppet] - 10https://gerrit.wikimedia.org/r/1352155 (owner: 10Muehlenhoff) [11:26:26] hnowlan, duesen: FYI I rolled back the train a second time now, error rate is low again. Sorry (for the noise) and thanks (for the heads-up) [11:28:15] RESOLVED: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [11:28:45] FIRING: [3x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [11:30:17] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2107.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:30:45] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2107.codfw.wmnet [11:30:47] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2107.codfw.wmnet [11:31:01] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2108.codfw.wmnet [11:31:15] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [11:31:43] andre: thanks! [11:33:08] (03CR) 10Muehlenhoff: [C:03+2] Remove Puppet leftovers for build2001 [puppet] - 10https://gerrit.wikimedia.org/r/1352356 (https://phabricator.wikimedia.org/T417389) (owner: 10Muehlenhoff) [11:33:30] RESOLVED: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [11:34:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:34:29] (03PS2) 10Cathal Mooney: hieradata/cloud/eqiad1.yaml: add default IPIP source IP ranges [puppet] - 10https://gerrit.wikimedia.org/r/1352489 (https://phabricator.wikimedia.org/T440201) [11:35:51] (03CR) 10Marostegui: [C:03+1] "Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1346031 (https://phabricator.wikimedia.org/T439526) (owner: 10CWilliams) [11:36:05] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2108.codfw.wmnet [11:36:26] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:37:54] jouncebot: nowandnext [11:37:54] For the next 0 hour(s) and 22 minute(s): Services – Citoid / Zotero (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T1100) [11:37:54] In 1 hour(s) and 22 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T1300) [11:38:51] (03CR) 10Vgutierrez: [C:03+1] hieradata/cloud/eqiad1.yaml: add default IPIP source IP ranges [puppet] - 10https://gerrit.wikimedia.org/r/1352489 (https://phabricator.wikimedia.org/T440201) (owner: 10Cathal Mooney) [11:39:24] (03PS4) 10Hnowlan: graphite: remove module, references, config [puppet] - 10https://gerrit.wikimedia.org/r/1332767 (https://phabricator.wikimedia.org/T435340) [11:40:15] (03CR) 10Elukey: [C:03+1] jaeger: add release label to query pod and certificate [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352471 (owner: 10Giuseppe Lavagetto) [11:42:58] (03CR) 10Hnowlan: [C:03+2] graphite: remove module, references, config [puppet] - 10https://gerrit.wikimedia.org/r/1332767 (https://phabricator.wikimedia.org/T435340) (owner: 10Hnowlan) [11:43:31] 06SRE, 06Traffic, 10WikimediaDebug: Debug server routing does not work for /w/rest.php - https://phabricator.wikimedia.org/T440439#12405731 (10Vgutierrez) `backend=k8s-mwdebug` isn't currently allowed for `/w/rest.php`: `lang=lua -- For rest.php, only allow debug remapping for mw-experimental and mw-pars... [11:43:49] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2108.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:44:09] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2108.codfw.wmnet [11:44:24] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:45:25] !log remove graphite config from puppet, moved graphite hosts to insetup [11:45:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:45:50] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2108.codfw.wmnet [11:46:04] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2109.codfw.wmnet [11:46:42] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2109.codfw.wmnet [11:47:02] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:49:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:49:45] FIRING: WidespreadPuppetFailure: Puppet has failed in magru - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [11:51:32] (03CR) 10Muehlenhoff: [C:03+1] "Looks good!" [puppet] - 10https://gerrit.wikimedia.org/r/1352442 (owner: 10Elukey) [11:52:41] (03CR) 10Marostegui: team-data-persistence: Alarm on stale data in Zarcillo (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) (owner: 10Federico Ceratto) [11:54:45] FIRING: [3x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [11:54:50] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2109.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:55:04] (03PS6) 10Blake: mw-experimental: declare mesh, mcrouter, statsd as native k8s sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352466 (https://phabricator.wikimedia.org/T417800) [11:55:09] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2109.codfw.wmnet [11:56:28] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2109.codfw.wmnet [11:56:42] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2110.codfw.wmnet [11:57:19] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2110.codfw.wmnet [11:57:39] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:59:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:59:16] (03PS1) 10Daphne Smit: PublishDialog: Use one slot for the summary field warning [extensions/WikiLambda] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352493 (https://phabricator.wikimedia.org/T372149) [11:59:45] FIRING: [5x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [12:03:08] (03PS7) 10Clément Goubert: mw-experimental: declare mesh, mcrouter as native k8s sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352466 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [12:03:09] (03PS1) 10Hnowlan: Revert "graphite: remove module, references, config" [puppet] - 10https://gerrit.wikimedia.org/r/1352494 [12:04:13] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker2108.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:05:28] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2110.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:05:46] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2110.codfw.wmnet [12:07:07] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, October 07 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [extensions/WikiLambda] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352493 (https://phabricator.wikimedia.org/T372149) (owner: 10Daphne Smit) [12:07:49] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2110.codfw.wmnet [12:08:04] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2111.codfw.wmnet [12:09:13] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2111.codfw.wmnet [12:09:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker2109.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:09:13] FIRING: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:09:24] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker2109.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:09:29] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:09:33] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:09:35] (03CR) 10CI reject: [V:04-1] Revert "graphite: remove module, references, config" [puppet] - 10https://gerrit.wikimedia.org/r/1352494 (owner: 10Hnowlan) [12:09:59] (03CR) 10Cathal Mooney: [C:03+2] hieradata/cloud/eqiad1.yaml: add default IPIP source IP ranges [puppet] - 10https://gerrit.wikimedia.org/r/1352489 (https://phabricator.wikimedia.org/T440201) (owner: 10Cathal Mooney) [12:11:35] (03PS1) 10Hnowlan: varnish: explicitly cast timeout in reload-vcl [puppet] - 10https://gerrit.wikimedia.org/r/1352495 (https://phabricator.wikimedia.org/T440441) [12:11:41] (03CR) 10Muehlenhoff: [C:03+2] Remove profile::base::cuminunpriv from apt servers [puppet] - 10https://gerrit.wikimedia.org/r/1351375 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [12:12:48] (03CR) 10Slyngshede: [C:03+1] varnish: explicitly cast timeout in reload-vcl [puppet] - 10https://gerrit.wikimedia.org/r/1352495 (https://phabricator.wikimedia.org/T440441) (owner: 10Hnowlan) [12:13:20] (03CR) 10Hnowlan: [C:03+2] varnish: explicitly cast timeout in reload-vcl [puppet] - 10https://gerrit.wikimedia.org/r/1352495 (https://phabricator.wikimedia.org/T440441) (owner: 10Hnowlan) [12:14:04] (03CR) 10Giuseppe Lavagetto: [C:03+2] jaeger: add release label to query pod and certificate [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352471 (owner: 10Giuseppe Lavagetto) [12:14:09] moritzm: safe to merge your change? [12:15:28] (03Merged) 10jenkins-bot: jaeger: add release label to query pod and certificate [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352471 (owner: 10Giuseppe Lavagetto) [12:15:34] (03PS6) 10CWilliams: mediabackups: add mediabackup_worker metrics [puppet] - 10https://gerrit.wikimedia.org/r/1346031 (https://phabricator.wikimedia.org/T439526) [12:16:35] I have broken a lot of stuff so I'll merge [12:16:38] hnowlan: I was about to, please merge along [12:16:41] ah cool :) [12:16:52] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2111.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:17:10] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2111.codfw.wmnet [12:18:09] (03CR) 10CWilliams: [C:03+2] mediabackups: add mediabackup_worker metrics [puppet] - 10https://gerrit.wikimedia.org/r/1346031 (https://phabricator.wikimedia.org/T439526) (owner: 10CWilliams) [12:19:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker2110.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:19:24] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2111.codfw.wmnet [12:19:38] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2112.codfw.wmnet [12:20:10] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2112.codfw.wmnet [12:20:31] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:24:13] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker2110.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:25:17] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: ms-be1074: /dev/sdn issue - https://phabricator.wikimedia.org/T440281#12405826 (10Jclark-ctr) @CWilliams-WMF I do have a spare drive let me know when i can swap disk Failed disk Physical Disk 0:2:12 Serial Number WWZ947C8 [12:25:31] !log oblivian@deploy1003 helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: apply [12:25:45] !log oblivian@deploy1003 helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: apply [12:26:29] !log oblivian@deploy1003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: apply [12:26:37] !log oblivian@deploy1003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: apply [12:27:39] (03Abandoned) 10Hnowlan: Revert "graphite: remove module, references, config" [puppet] - 10https://gerrit.wikimedia.org/r/1352494 (owner: 10Hnowlan) [12:27:49] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2112.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:28:09] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2112.codfw.wmnet [12:28:10] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2112.codfw.wmnet [12:28:25] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2113.codfw.wmnet [12:28:59] (03CR) 10Daniel Kertesz: [C:03+1] service catalogue: Set ldap-ro-test/ldap-ro-test-ssl to production [puppet] - 10https://gerrit.wikimedia.org/r/1352474 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [12:29:00] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2113.codfw.wmnet [12:29:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker2111.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:29:20] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:29:24] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker2111.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:31:23] PROBLEM - Host wikikube-worker1016 is DOWN: PING CRITICAL - Packet loss = 60%, RTA = 3566.27 ms [12:31:59] RECOVERY - Host wikikube-worker1016 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms [12:32:09] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352485 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [12:33:13] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352486 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [12:33:35] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352486 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [12:37:09] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2113.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:37:27] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2113.codfw.wmnet [12:37:29] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2113.codfw.wmnet [12:37:43] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2114.codfw.wmnet [12:38:20] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2114.codfw.wmnet [12:38:40] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:43:34] (03CR) 10DCausse: [C:03+1] Data-platform: migrate masters-eligible check to Prometheus [alerts] - 10https://gerrit.wikimedia.org/r/1352282 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [12:44:45] FIRING: [5x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [12:45:43] !log uploaded librsvg 2.60.0+dfsg-1+wmf13u2 to component/thumbor (backport of RUSTSEC-2026-0305 to our internal build) [12:45:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:46:54] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device ssw1-a8-eqiad [12:47:21] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad [12:47:31] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a8-eqiad [12:47:50] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad [12:48:05] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2114.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:48:34] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2114.codfw.wmnet [12:49:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker2113.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:49:30] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2114.codfw.wmnet [12:49:44] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2115.codfw.wmnet [12:49:45] RESOLVED: [5x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [12:50:53] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2115.codfw.wmnet [12:50:59] (03CR) 10Clément Goubert: [C:03+1] mw-experimental: declare mesh, mcrouter as native k8s sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352466 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [12:51:14] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:52:52] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for krb1002.mgmt:22 - https://phabricator.wikimedia.org/T439280#12405948 (10Jclark-ctr) Supermicro ticket opened #CS-00211329 [12:53:22] (03PS1) 10Muehlenhoff: thumbor-plugins: Rebuild to include latest librsvg backport [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1352505 [12:54:13] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker2113.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:54:25] !log oblivian@deploy1003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:54:32] !log oblivian@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:54:35] !log oblivian@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:54:43] !log oblivian@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:55:48] (03CR) 10Federico Ceratto: team-data-persistence: Alarm on stale data in Zarcillo (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) (owner: 10Federico Ceratto) [12:56:01] (03PS2) 10Scott French: rest-gateway: Enable diversion of testwiki traffic to mw-pretrain [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352503 (https://phabricator.wikimedia.org/T433752) [12:56:28] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T440195#12405974 (10Jclark-ctr) 05Open→03Resolved [12:58:47] (03CR) 10Clément Goubert: [C:03+1] REST Gateway: change link syntax used in ratee limit responses. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352145 (owner: 10Daniel Kinzler) [12:58:50] (03PS1) 10Seanleong-wmde: UsageDeduplicator: Re-index C usages after filtering by CQR modifier [extensions/Wikibase] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352506 (https://phabricator.wikimedia.org/T440429) [12:59:24] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-eqiad:et-0/1/0 (DISABLED) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [13:00:05] urbanecm and TheresNoTime: Time to snap out of that daydream and deploy UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T1300). [13:00:05] seanleong-wmde and DSmit-WMF: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:10] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2115.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:00:23] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2115.codfw.wmnet [13:01:04] I’m here too, in case DSmit-WMF isn't. [13:01:13] I am! [13:01:18] Excellent. [13:01:25] Hi o7, I can start with a backport of a unreak now ticket, if possible? [13:02:36] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2115.codfw.wmnet [13:02:37] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2067-2078,2087-2095,2102-2115].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [13:03:20] Okay, I will start now [13:03:30] (03PS1) 10Giuseppe Lavagetto: deployment_server: fix jaeger label name [puppet] - 10https://gerrit.wikimedia.org/r/1352507 [13:04:21] (03CR) 10TrainBranchBot: [C:03+2] "Approved by seanleong-wmde@deploy1003 using scap backport" [extensions/Wikibase] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352506 (https://phabricator.wikimedia.org/T440429) (owner: 10Seanleong-wmde) [13:04:25] RESOLVED: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:06:40] (03CR) 10Giuseppe Lavagetto: [C:03+2] deployment_server: fix jaeger label name [puppet] - 10https://gerrit.wikimedia.org/r/1352507 (owner: 10Giuseppe Lavagetto) [13:07:49] (03Merged) 10jenkins-bot: UsageDeduplicator: Re-index C usages after filtering by CQR modifier [extensions/Wikibase] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352506 (https://phabricator.wikimedia.org/T440429) (owner: 10Seanleong-wmde) [13:08:14] !log seanleong-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1352506|UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] [13:08:17] T440429: Error: Call to a member function getEntityId() on null - https://phabricator.wikimedia.org/T440429 [13:12:11] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a8-eqiad [13:12:28] !log seanleong-wmde@deploy1003 seanleong-wmde: Backport for [[gerrit:1352506|UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:12:54] !log cmooney@cumin1004 END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad [13:13:11] !log installing mbedtls security updates [13:13:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:14:02] !log seanleong-wmde@deploy1003 seanleong-wmde: Continuing with deployment [13:15:20] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a8-eqiad [13:15:32] !log cmooney@cumin1004 END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-a8-eqiad [13:16:27] RECOVERY - Check unit status of kube-reload-certs on deploy1003 is OK: OK: Status of the systemd unit kube-reload-certs https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [13:19:21] !log seanleong-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1352506|UsageDeduplicator: Re-index C usages after filtering by CQR modifier (T440429)]] (duration: 11m 07s) [13:19:24] T440429: Error: Call to a member function getEntityId() on null - https://phabricator.wikimedia.org/T440429 [13:20:20] (03CR) 10TrainBranchBot: [C:03+2] "Approved by seanleong-wmde@deploy1003 using scap backport" [extensions/SiteMatrix] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352456 (https://phabricator.wikimedia.org/T439067) (owner: 10Seanleong-wmde) [13:21:08] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a8-eqiad [13:22:04] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-eqiad [13:22:11] oblivian@cumin1004 enable_virtualization (PID 3438579) is awaiting input [13:23:41] (03PS4) 10Milazg: WIP: change config [puppet] - 10https://gerrit.wikimedia.org/r/1352263 [13:23:52] (03CR) 10Milazg: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352263 (owner: 10Milazg) [13:25:15] (03PS1) 10Elukey: profile::prometheus::ops: change port for cfssl [puppet] - 10https://gerrit.wikimedia.org/r/1352509 [13:26:52] (03Merged) 10jenkins-bot: Group fix for incorrect group mapping. [extensions/SiteMatrix] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352456 (https://phabricator.wikimedia.org/T439067) (owner: 10Seanleong-wmde) [13:27:00] (03CR) 10Elukey: [C:03+1] Drop build2001 from profile::docker_registry::image_builders [puppet] - 10https://gerrit.wikimedia.org/r/1352354 (owner: 10Muehlenhoff) [13:27:06] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device ssw1-a8-eqiad [13:27:10] !log seanleong-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1352456|Group fix for incorrect group mapping. (T439067)]] [13:27:13] T439067: Fix special wiki "group" in SiteMatrixLookup - https://phabricator.wikimedia.org/T439067 [13:27:32] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-eqiad [13:27:53] (03CR) 10Elukey: [C:03+2] profile::prometheus::ops: change port for cfssl [puppet] - 10https://gerrit.wikimedia.org/r/1352509 (owner: 10Elukey) [13:30:30] (03PS1) 10Muehlenhoff: Remove unpriv_cumin_masters Hiera key and alertmanager config [puppet] - 10https://gerrit.wikimedia.org/r/1352510 (https://phabricator.wikimedia.org/T439522) [13:31:07] !log oblivian@cumin1004 START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2124-2140].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [13:31:10] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2124.codfw.wmnet [13:31:26] !log seanleong-wmde@deploy1003 seanleong-wmde: Backport for [[gerrit:1352456|Group fix for incorrect group mapping. (T439067)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:31:43] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2124.codfw.wmnet [13:32:04] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:34:45] (03CR) 10Bking: [C:03+2] Data-platform: migrate masters-eligible check to Prometheus [alerts] - 10https://gerrit.wikimedia.org/r/1352282 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [13:34:55] 10SRE-SLO, 06Collaboration-Services: Implement service level indicator measurement for Gerrit - https://phabricator.wikimedia.org/T396979#12406162 (10ABran-WMF) 05Stalled→03Open [13:36:13] (03CR) 10Muehlenhoff: [C:03+2] Drop build2001 from profile::docker_registry::image_builders [puppet] - 10https://gerrit.wikimedia.org/r/1352354 (owner: 10Muehlenhoff) [13:37:18] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2124.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:37:24] !log seanleong-wmde@deploy1003 seanleong-wmde: Continuing with deployment [13:39:44] (03CR) 10Elukey: [C:03+2] Revert "profile::pki::client: move the signer endpoint to pki[12]002" [puppet] - 10https://gerrit.wikimedia.org/r/1352442 (owner: 10Elukey) [13:39:44] seanleong-wmde: uh, thank you so much for backporting the Wikibase issue! <3 Just got back and was about to do the same (and then try to move the train forward again, maybe later in this window, if some time is left) :) [13:42:42] !log seanleong-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1352456|Group fix for incorrect group mapping. (T439067)]] (duration: 15m 31s) [13:42:45] T439067: Fix special wiki "group" in SiteMatrixLookup - https://phabricator.wikimedia.org/T439067 [13:43:05] andre: No prob! thanks for notifying us! [13:43:11] <3 [13:43:35] seanleong-wmde, yay. Please tell me when all backports are done for this window, maybe there will be some time left [13:43:43] DSmit-WMF: I'm done now! [13:43:44] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad A/B switch cabling documentation - https://phabricator.wikimedia.org/T418018#12406199 (10cmooney) >>! In T418018#12403321, @Jclark-ctr wrote: > @cmooney @RobH Finished A8 LSW, spine management, and SCS cabling. Updated the SCS... [13:43:57] Sorry for the delay [13:44:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:44:27] (03PS1) 10EMcFarland: Home Experience: Feature flag to enable Home [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352515 (https://phabricator.wikimedia.org/T439738) [13:44:42] !log start maintenance on cr2-codfw [13:44:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:45:24] I dont see wmf 23 yet here https://www.wikifunctions.org/wiki/Special:Version [13:45:26] !log pt1979@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade [13:46:33] DSmit-WMF: because the train was rolled back because of that issue and another one, see https://versions.toolforge.org/ and https://phabricator.wikimedia.org/T438219 [13:46:36] I think there's a blocker in the train for group1, so wikifunctions is still in wmf22, but maybe Andre will have more info [13:46:55] yeah, see the task graph in https://phabricator.wikimedia.org/T438219 [13:47:22] ah ok thank you. The fix only impacts Wikifunctions [13:47:33] I think https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/1352493 for https://phabricator.wikimedia.org/T440446 still needs backporting and deploying? [13:47:43] * andre trying to get an overview, somehow [13:48:15] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2124.codfw.wmnet [13:48:17] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2124.codfw.wmnet [13:48:31] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2125.codfw.wmnet [13:48:57] It was scheduled https://schedule-deployment.toolforge.org/window/1791378000 here. But if theres a blocker then nothing I can do for now? [13:49:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:49:04] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2125.codfw.wmnet [13:49:13] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:49:53] (03PS1) 10Jforrester: wikifunctions: Upgrade evaluators from 2026-09-30-134115 to 2026-10-07-130343 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352518 (https://phabricator.wikimedia.org/T404980) [13:50:04] (03PS1) 10Jforrester: wikifunctions: Upgrade orchestrator from 2026-09-30-134715 to 2026-10-07-060824 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352519 (https://phabricator.wikimedia.org/T370472) [13:50:14] DSmit-WMF: No, we *are* the blocker. [13:50:20] Or one of them. [13:50:30] (03CR) 10Clément Goubert: [C:03+1] "LGTM, will require a puppet run on the deployment server as well as a helmfile-only mediawiki deployment to take effect." [puppet] - 10https://gerrit.wikimedia.org/r/1351442 (https://phabricator.wikimedia.org/T432733) (owner: 10DLynch) [13:50:33] DSmit-WMF: I can deploy it now? [13:50:35] (03PS1) 10Ozge: ml-services: moves llm-qwen3-14b to ml-serve1013 or ml-serve1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352520 [13:50:52] yes [13:50:54] heh, James' answer is shorter than what I was about to answer :D [13:50:54] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jforrester@deploy1003 using scap backport" [extensions/WikiLambda] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352493 (https://phabricator.wikimedia.org/T372149) (owner: 10Daphne Smit) [13:51:00] :-) [13:51:17] (03CR) 10Gkyziridis: [C:03+1] ml-services: moves llm-qwen3-14b to ml-serve1013 or ml-serve1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352520 (owner: 10Ozge) [13:51:37] (03CR) 10Kamila Součková: [C:03+1] Test new rsyslog image on mw-experimental [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352121 (https://phabricator.wikimedia.org/T438375) (owner: 10Herron) [13:51:47] sorry I was confused. I didnt know I needed to press the button;) [13:51:56] thank you James_F [13:52:00] (03CR) 10Muehlenhoff: [C:03+2] service catalogue: Set ldap-ro-test/ldap-ro-test-ssl to production [puppet] - 10https://gerrit.wikimedia.org/r/1352474 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [13:53:14] DSmit-WMF: No worries at all. :-) [13:53:38] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.upgrade restart P{lvs7003.magru.wmnet} and A:liberica [13:53:52] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin depooling P{lvs7003.magru.wmnet} and A:liberica [13:54:04] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7003.magru.wmnet} and A:liberica [13:54:13] RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr2-eqiad:et-0/1/0 (DISABLED) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [13:54:15] (03PS3) 10Clément Goubert: Add revision-feature-counts ingress CNAME [dns] - 10https://gerrit.wikimedia.org/r/1347696 (https://phabricator.wikimedia.org/T437000) [13:54:17] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin pooling P{lvs7003.magru.wmnet} and A:liberica [13:54:32] (03Merged) 10jenkins-bot: PublishDialog: Use one slot for the summary field warning [extensions/WikiLambda] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352493 (https://phabricator.wikimedia.org/T372149) (owner: 10Daphne Smit) [13:54:33] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2125.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:54:36] !log vgutierrez@cumin1004 END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling P{lvs7003.magru.wmnet} and A:liberica [13:54:50] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7003.magru.wmnet} and A:liberica [13:54:54] !log jforrester@deploy1003 Started scap sync-world: Backport for [[gerrit:1352493|PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] [13:54:59] T372149: Add support for the "Add pages I create and files I upload to my watchlist" preference to WikiLambda API edit code - https://phabricator.wikimedia.org/T372149 [13:55:04] T440446: Function pages on Wikifunctions are blank since 1.47.0-wmf.23 (Vue compiler error 30 in PublishDialog) - https://phabricator.wikimedia.org/T440446 [13:55:59] (03CR) 10Kamila Součková: [C:03+1] scap: Remove unused php7_admin_port option [puppet] - 10https://gerrit.wikimedia.org/r/1219164 (https://phabricator.wikimedia.org/T224491) (owner: 10Krinkle) [13:57:15] (03PS1) 10Bking: cirrussearch: Absent icinga-based masters-eligible checks [puppet] - 10https://gerrit.wikimedia.org/r/1352523 (https://phabricator.wikimedia.org/T431304) [13:58:21] (03PS1) 10Muehlenhoff: idp-test: Configure to use the new Trixie/MDB LDAP servers for tests [puppet] - 10https://gerrit.wikimedia.org/r/1352525 (https://phabricator.wikimedia.org/T331699) [13:58:30] (03CR) 10CI reject: [V:04-1] cirrussearch: Absent icinga-based masters-eligible checks [puppet] - 10https://gerrit.wikimedia.org/r/1352523 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [13:59:18] !log jforrester@deploy1003 daphnesmit, jforrester: Backport for [[gerrit:1352493|PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:00:05] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T1400) [14:00:16] (03PS2) 10Bking: cirrussearch: Absent icinga-based masters-eligible checks [puppet] - 10https://gerrit.wikimedia.org/r/1352523 (https://phabricator.wikimedia.org/T431304) [14:00:30] !log jforrester@deploy1003 daphnesmit, jforrester: Continuing with deployment [14:01:28] (03CR) 10Kamila Součková: [C:03+1] "For my information, what's the plan for checking that things are working (both this patch and previous), just check that logs are turning " [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346712 (https://phabricator.wikimedia.org/T438375) (owner: 10Herron) [14:02:00] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352525 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [14:02:04] (03CR) 10Ilias Sarantopoulos: [C:03+2] ml-services: moves llm-qwen3-14b to ml-serve1013 or ml-serve1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352520 (owner: 10Ozge) [14:03:10] (03PS3) 10Clément Goubert: revision-feature-counts: service catalog, listener [puppet] - 10https://gerrit.wikimedia.org/r/1347695 (https://phabricator.wikimedia.org/T439879) [14:03:24] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.upgrade restart P{lvs7003.magru.wmnet} and A:liberica [14:03:36] (03Merged) 10jenkins-bot: ml-services: moves llm-qwen3-14b to ml-serve1013 or ml-serve1014 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352520 (owner: 10Ozge) [14:03:39] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin depooling P{lvs7003.magru.wmnet} and A:liberica [14:03:51] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7003.magru.wmnet} and A:liberica [14:04:04] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin pooling P{lvs7003.magru.wmnet} and A:liberica [14:04:26] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7003.magru.wmnet} and A:liberica [14:04:28] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart P{lvs7003.magru.wmnet} and A:liberica [14:05:17] (03CR) 10Cory Massaro: [C:03+2] wikifunctions: Upgrade orchestrator from 2026-09-30-134715 to 2026-10-07-060824 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352519 (https://phabricator.wikimedia.org/T370472) (owner: 10Jforrester) [14:05:29] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2125.codfw.wmnet [14:05:31] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2125.codfw.wmnet [14:05:44] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2126.codfw.wmnet [14:05:46] (03CR) 10Cory Massaro: wikifunctions: Upgrade orchestrator from 2026-09-30-134715 to 2026-10-07-060824 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352519 (https://phabricator.wikimedia.org/T370472) (owner: 10Jforrester) [14:05:50] (03CR) 10Cory Massaro: [C:03+2] wikifunctions: Upgrade evaluators from 2026-09-30-134115 to 2026-10-07-130343 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352518 (https://phabricator.wikimedia.org/T404980) (owner: 10Jforrester) [14:05:54] !log jforrester@deploy1003 Finished scap sync-world: Backport for [[gerrit:1352493|PublishDialog: Use one slot for the summary field warning (T372149 T440446)]] (duration: 10m 59s) [14:05:59] T372149: Add support for the "Add pages I create and files I upload to my watchlist" preference to WikiLambda API edit code - https://phabricator.wikimedia.org/T372149 [14:05:59] T440446: Function pages on Wikifunctions are blank since 1.47.0-wmf.23 (Vue compiler error 30 in PublishDialog) - https://phabricator.wikimedia.org/T440446 [14:06:07] DSmit-WMF: Done! [14:06:09] James_F: Thanks! Do you by any chance know if the "Wikifunctions Services UTC Afternoon" window listed on https://wikitech.wikimedia.org/wiki/Deployments#Wednesday,_October_7 is being used, or could I use the remaining 20min to try another time to move forward the train? [14:06:17] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2126.codfw.wmnet [14:06:38] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:06:45] andre: Yes, always used. [14:06:52] andre: It’s 54 minutes left. [14:07:05] andre: But you can push the train now, I’ll hold back on the MW side. :-) [14:07:22] James_F: oh, thank you a lot, I'll push the train now then [14:07:28] (03Merged) 10jenkins-bot: wikifunctions: Upgrade evaluators from 2026-09-30-134115 to 2026-10-07-130343 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352518 (https://phabricator.wikimedia.org/T404980) (owner: 10Jforrester) [14:07:30] 06SRE, 10SRE-Access-Requests: Requesting access to for - https://phabricator.wikimedia.org/T440464 (10OBenhmida) 03NEW [14:08:35] (03PS2) 10Vgutierrez: liberica: Sync with liberica 0.25 idletcpconnectioncheck config [puppet] - 10https://gerrit.wikimedia.org/r/1351400 (https://phabricator.wikimedia.org/T439840) [14:09:08] !log apine@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:09:53] (03PS1) 10TrainBranchBot: group1 to 1.47.0-wmf.23 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352531 (https://phabricator.wikimedia.org/T438219) [14:09:55] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by aklapper@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352531 (https://phabricator.wikimedia.org/T438219) (owner: 10TrainBranchBot) [14:10:21] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [14:10:30] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users + Kerberos for OBenhmida - https://phabricator.wikimedia.org/T440464#12406408 (10Aklapper) [14:10:41] (03Merged) 10jenkins-bot: group1 to 1.47.0-wmf.23 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352531 (https://phabricator.wikimedia.org/T438219) (owner: 10TrainBranchBot) [14:10:46] !log apine@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:11:00] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users + Kerberos for OBenhmida - https://phabricator.wikimedia.org/T440464#12406410 (10Aklapper) @OBenhmida: Please also [link your LDAP account to your Phabricator account](https://phabricator.wikimedia.org/settings/panel/external/), so... [14:11:03] !log apine@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:11:54] !log apine@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:12:00] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2126.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:12:07] !log apine@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:12:36] !log apine@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:13:18] (03CR) 10Cory Massaro: [C:03+2] wikifunctions: Upgrade orchestrator from 2026-09-30-134715 to 2026-10-07-060824 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352519 (https://phabricator.wikimedia.org/T370472) (owner: 10Jforrester) [14:14:06] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352523 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [14:14:40] !log upload liberica 0.26 to bookworm-wikimedia (apt.wm.o) - T439840 [14:14:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:14:42] T439840: Get liberica ready to be deployed on low-traffic loadbalancers - https://phabricator.wikimedia.org/T439840 [14:14:48] (03Merged) 10jenkins-bot: wikifunctions: Upgrade orchestrator from 2026-09-30-134715 to 2026-10-07-060824 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352519 (https://phabricator.wikimedia.org/T370472) (owner: 10Jforrester) [14:15:29] Confirm the bug T440446 is fixed [14:15:29] T440446: Function pages on Wikifunctions are blank since 1.47.0-wmf.23 (Vue compiler error 30 in PublishDialog) - https://phabricator.wikimedia.org/T440446 [14:16:11] 06SRE, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Site: EQIAD VM request for Kerberos - https://phabricator.wikimedia.org/T438229#12406444 (10bking) Thanks Lukasz and Moritz. I'll create the VM myself and release the host into DC Ops inventory as you suggested. [14:16:31] (03CR) 10Vgutierrez: [C:03+2] liberica: Sync with liberica 0.25 idletcpconnectioncheck config [puppet] - 10https://gerrit.wikimedia.org/r/1351400 (https://phabricator.wikimedia.org/T439840) (owner: 10Vgutierrez) [14:16:32] !log apine@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:16:37] 06SRE, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Site: EQIAD VM request for Kerberos - https://phabricator.wikimedia.org/T438229#12406446 (10bking) 05Open→03In progress p:05Triage→03Medium a:03bking [14:16:44] 10ops-eqiad, 06DC-Ops, 10Icinga, 10Observability-Alerting, 07Sustainability (Incident Followup): Check notification status for ps1-d5-eqiad - https://phabricator.wikimedia.org/T439233#12406450 (10hnowlan) [14:16:50] (03PS1) 10Eevans: linked_artifacts: deploy Hoarde v1.5.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352533 (https://phabricator.wikimedia.org/T440395) [14:16:58] !log apine@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:17:18] !log apine@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:17:32] (03CR) 10Elukey: [C:03+1] Remove unpriv_cumin_masters Hiera key and alertmanager config [puppet] - 10https://gerrit.wikimedia.org/r/1352510 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [14:18:01] !log apine@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:18:10] !log apine@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:18:34] (03CR) 10Eevans: [C:03+2] linked_artifacts: deploy Hoarde v1.5.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352533 (https://phabricator.wikimedia.org/T440395) (owner: 10Eevans) [14:18:46] !log apine@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:18:57] andre: Looking good from the WF side, at least. :-) [14:19:05] James_F: Hmm, https://phabricator.wikimedia.org/T440432 is showing high levels in Logstash again, and maybe my assessment there was wrong [14:19:12] nearly finished with train [14:19:55] andre: If Wikidata injected a bunch of bad content into the job queue they’ll take a while to burn out. [14:19:56] !log installing postgresql-15 security updates [14:19:57] (03Merged) 10jenkins-bot: linked_artifacts: deploy Hoarde v1.5.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352533 (https://phabricator.wikimedia.org/T440395) (owner: 10Eevans) [14:19:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:19:58] But let me look. [14:20:12] !log aklapper@deploy1003 rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.23 refs T438219 [14:20:16] T438219: 1.47.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T438219 [14:20:16] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for dhardy - https://phabricator.wikimedia.org/T440467 (10Dillon) 03NEW [14:20:20] !log eevans@deploy1003 helmfile [staging] START helmfile.d/services/linked-artifacts: apply [14:20:24] James_F: Thanks. Done with train. And yeah https://phabricator.wikimedia.org/T440432 is going up again but that [14:20:31] 's all .22, previous version. [14:20:33] !log eevans@deploy1003 helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply [14:20:47] Ack., [14:20:59] James_F: I guess your judgement is very welcome whether I should roll back a third time, or if this is fine, or fixable soon. TIA [14:21:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jforrester@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346643 (owner: 10Jforrester) [14:21:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jforrester@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351531 (https://phabricator.wikimedia.org/T400015) (owner: 10Jforrester) [14:22:01] (03Merged) 10jenkins-bot: Factor out Abstract Wikipedia client mode into a dblist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346643 (owner: 10Jforrester) [14:22:03] (03Merged) 10jenkins-bot: Wikifunctions: Switch the editor from Ace to CodeMirror [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351531 (https://phabricator.wikimedia.org/T400015) (owner: 10Jforrester) [14:22:20] !log jforrester@deploy1003 Started scap sync-world: Backport for [[gerrit:1346643|Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531|Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] [14:22:24] T400015: Add CodeMirror integration in WikiLambda as an alternative to CodeEditor - https://phabricator.wikimedia.org/T400015 [14:24:01] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2126.codfw.wmnet [14:24:03] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2126.codfw.wmnet [14:24:17] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2127.codfw.wmnet [14:24:38] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-magru [14:24:51] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2127.codfw.wmnet [14:25:11] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:26:34] !log jforrester@deploy1003 jforrester: Backport for [[gerrit:1346643|Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531|Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:27:11] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-magru [14:27:19] !log installing PHP 8.2 security updates [14:27:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:30:05] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T1400) [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T1430) [14:30:14] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users + Kerberos for OBenhmida - https://phabricator.wikimedia.org/T440464#12406557 (10OBenhmida) > @OBenhmida: Please also link your LDAP account to your Phabricator account, so your 'LDAP User' account is shown on your Phabricator profi... [14:30:27] PROBLEM - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [14:31:03] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:32:23] (03PS1) 10Jforrester: WikifunctionsClientUsageUpdateJob: Accept jobs from a later version [extensions/WikiLambda] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352539 (https://phabricator.wikimedia.org/T440432) [14:32:35] andre: I think that will quiet it. Want me to deploy? [14:33:55] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-ulsfo [14:34:03] oblivian@cumin1004 enable_virtualization (PID 3438579) is awaiting input [14:34:09] (03PS4) 10Clément Goubert: admin_ng: Add lac-lambdas namespace to wikikube [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347653 (https://phabricator.wikimedia.org/T439879) [14:34:09] (03PS8) 10Clément Goubert: revision-feature-counts: add deployment chart files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) [14:34:33] James_F: I think it was bad content in the job queue, spike in Logstash is mostly gone, but feel free to :) (sorry, in a meeting right now) [14:34:39] Ack. [14:34:52] !log jforrester@deploy1003 jforrester: Continuing with deployment [14:36:12] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-ulsfo [14:36:53] (03PS1) 10Elukey: profile::pki::multirootca: update httpd's SANs [puppet] - 10https://gerrit.wikimedia.org/r/1352541 [14:37:13] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-eqsin [14:37:35] (03PS2) 10Elukey: profile::pki::multirootca: update httpd's SANs [puppet] - 10https://gerrit.wikimedia.org/r/1352541 [14:37:50] (03PS3) 10Bking: cirrussearch: Absent icinga-based masters-eligible checks [puppet] - 10https://gerrit.wikimedia.org/r/1352523 (https://phabricator.wikimedia.org/T431304) [14:38:02] (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352541 (owner: 10Elukey) [14:38:33] (03CR) 10CI reject: [V:04-1] cirrussearch: Absent icinga-based masters-eligible checks [puppet] - 10https://gerrit.wikimedia.org/r/1352523 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [14:39:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:39:58] (03PS4) 10Bking: cirrussearch: Absent icinga-based masters-eligible checks [puppet] - 10https://gerrit.wikimedia.org/r/1352523 (https://phabricator.wikimedia.org/T431304) [14:40:34] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352523 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [14:40:47] !log jforrester@deploy1003 Finished scap sync-world: Backport for [[gerrit:1346643|Factor out Abstract Wikipedia client mode into a dblist]], [[gerrit:1351531|Wikifunctions: Switch the editor from Ace to CodeMirror (T400015)]] (duration: 18m 26s) [14:40:50] T400015: Add CodeMirror integration in WikiLambda as an alternative to CodeEditor - https://phabricator.wikimedia.org/T400015 [14:41:12] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-eqsin [14:41:27] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-drmrs [14:41:44] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jforrester@deploy1003 using scap backport" [extensions/WikiLambda] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352539 (https://phabricator.wikimedia.org/T440432) (owner: 10Jforrester) [14:43:12] (03Merged) 10jenkins-bot: WikifunctionsClientUsageUpdateJob: Accept jobs from a later version [extensions/WikiLambda] (wmf/1.47.0-wmf.22) - 10https://gerrit.wikimedia.org/r/1352539 (https://phabricator.wikimedia.org/T440432) (owner: 10Jforrester) [14:43:33] !log jforrester@deploy1003 Started scap sync-world: Backport for [[gerrit:1352539|WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] [14:43:37] T440432: TypeError: Cannot assign null to property MediaWiki\Extension\WikiLambda\Jobs\WikifunctionsClientUsageUpdateJob::$targetFunction of type string - https://phabricator.wikimedia.org/T440432 [14:43:41] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-drmrs [14:43:45] (03CR) 10Xcollazo: "PPC LGTM." [puppet] - 10https://gerrit.wikimedia.org/r/1346752 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [14:44:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:44:20] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.upgrade upgrade A:liberica-esams [14:44:25] 06SRE, 10SRE-Access-Requests: Requesting access to dumps-roots for analytics_admins_members (people associated with analytics-admins) - https://phabricator.wikimedia.org/T440471 (10dr0ptp4kt) 03NEW [14:45:10] (03PS1) 10Bking: cirrussearch: Remove icinga-based checks [puppet] - 10https://gerrit.wikimedia.org/r/1352543 (https://phabricator.wikimedia.org/T431304) [14:45:35] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352543 (https://phabricator.wikimedia.org/T431304) (owner: 10Bking) [14:45:45] (03PS2) 10Bking: cirrussearch: Remove icinga-based checks [puppet] - 10https://gerrit.wikimedia.org/r/1352543 (https://phabricator.wikimedia.org/T431304) [14:46:35] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) upgrade A:liberica-esams [14:47:41] !log jforrester@deploy1003 jforrester: Backport for [[gerrit:1352539|WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:48:23] !log jforrester@deploy1003 jforrester: Continuing with deployment [14:49:01] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1352541 (owner: 10Elukey) [14:49:08] (03PS1) 10Dr0ptp4kt: Grant dumps-roots access to analytics_admins_members [puppet] - 10https://gerrit.wikimedia.org/r/1352546 (https://phabricator.wikimedia.org/T440471) [14:49:42] (03CR) 10Elukey: [C:03+2] profile::pki::multirootca: update httpd's SANs [puppet] - 10https://gerrit.wikimedia.org/r/1352541 (owner: 10Elukey) [14:50:08] (03CR) 10Clément Goubert: [C:03+1] "Let's move forwards like this, it's temporary and will (at worst) get cleaned up when we do the changes to support `/api/` paths" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346837 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [14:51:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:52:48] (03CR) 10Clément Goubert: [C:03+1] api-gateway: Support PHP_ENGINE routing in the mw-api plugin [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335270 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [14:53:42] !log jforrester@deploy1003 Finished scap sync-world: Backport for [[gerrit:1352539|WikifunctionsClientUsageUpdateJob: Accept jobs from a later version (T440432)]] (duration: 10m 09s) [14:53:45] T440432: TypeError: Cannot assign null to property MediaWiki\Extension\WikiLambda\Jobs\WikifunctionsClientUsageUpdateJob::$targetFunction of type string - https://phabricator.wikimedia.org/T440432 [14:55:24] Are these elevated levels of logins related to a change, or is something fishy happening? [14:56:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:56:36] (03CR) 10Btullis: [C:03+2] "Yes, all good. Thanks." [puppet] - 10https://gerrit.wikimedia.org/r/1346752 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [14:56:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job cfssl in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:00:53] (03CR) 10Btullis: [C:03+1] "Looks good to me. Thanks." [cookbooks] - 10https://gerrit.wikimedia.org/r/1342235 (https://phabricator.wikimedia.org/T437615) (owner: 10Klausman) [15:01:04] (03CR) 10Klausman: [V:03+2 C:03+2] cookbooks/idm: Add user-cleanup cookbook for DPE SRE hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1342235 (https://phabricator.wikimedia.org/T437615) (owner: 10Klausman) [15:03:34] !log pt1979@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-a8-codfw,ssw1-a8-codfw IPv6,ssw1-a8-codfw.mgmt with reason: Junos upgrade [15:03:57] !log pt1979@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-d8-codfw,ssw1-d8-codfw IPv6,ssw1-d8-codfw.mgmt with reason: Junos upgrade [15:04:02] (03Merged) 10jenkins-bot: cookbooks/idm: Add user-cleanup cookbook for DPE SRE hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1342235 (https://phabricator.wikimedia.org/T437615) (owner: 10Klausman) [15:04:13] !log pt1979@cumin1004 DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ssw1-f8-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade [15:04:55] !log pt1979@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ssw1-f1-codfw,ssw1-f1-codfw IPv6,ssw1-f1-codfw.mgmt with reason: Junos upgrade [15:05:25] !log pt1979@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudsw1-b1-codfw,cloudsw1-b1-codfw IPv6,cloudsw1-b1-codfw.mgmt with reason: Junos upgrade [15:06:19] !log pt1979@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pfw1-codfw with reason: Junos upgrade [15:06:27] !log pt1979@cumin1004 DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ceqiad IPv6,cr2-eqiad,re0.cr2-eqiad.mgmt with reason: Junos upgrade [15:06:35] PROBLEM - Host db2201 is DOWN: PING CRITICAL - Packet loss = 100% [15:06:36] (03PS1) 10Xcollazo: mediawiki-dumps-legacy: Show the sunset date of the XML dumps [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352550 (https://phabricator.wikimedia.org/T440466) [15:07:03] !log pt1979@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqiad,cr2-eqiad IPv6,re0.cr2-eqiad.mgmt with reason: Junos upgrade [15:07:29] !log pt1979@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-esams,cr2-esams IPv6,cr2-esams.mgmt with reason: Junos upgrade [15:07:30] (03CR) 10Elukey: "One consideration:" [cookbooks] - 10https://gerrit.wikimedia.org/r/1342235 (https://phabricator.wikimedia.org/T437615) (owner: 10Klausman) [15:07:38] !log pt1979@cumin1004 DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cr2-ulsfo,cr4-ulsfo IPv6,cr2-ulsfo.mgmt with reason: Junos upgrade [15:07:41] (03CR) 10Phuedx: [C:03+1] Update Test Kitchen instruments intake URL on beta cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352261 (https://phabricator.wikimedia.org/T437720) (owner: 10Clare Ming) [15:08:03] (03PS1) 10Xcollazo: dumps: web: Show the sunset date of the XML dumps [puppet] - 10https://gerrit.wikimedia.org/r/1352551 (https://phabricator.wikimedia.org/T440466) [15:08:19] !log pt1979@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr4-ulsfo,cr4-ulsfo IPv6,cr4-ulsfo.mgmt with reason: Junos upgrade [15:09:38] !log pt1979@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr3-eqsin,cr3-eqsin IPv6,cr3-eqsin.mgmt with reason: Junos upgrade [15:10:39] !log pt1979@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-drmrs,cr2-drmrs IPv6,cr2-drmrs.mgmt with reason: Junos upgrade [15:11:06] !log pt1979@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqdfw,cr2-eqdfw IPv6 with reason: Junos upgrade [15:12:23] (03PS4) 10Ahmon Dancy: mediawiki::deployment::server: Automate scope=pretrain deployments [puppet] - 10https://gerrit.wikimedia.org/r/1329690 (https://phabricator.wikimedia.org/T436178) (owner: 10BryanDavis) [15:14:24] (03CR) 10Blake: [C:03+2] mw-experimental: declare mesh, mcrouter as native k8s sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352466 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [15:14:47] RECOVERY - Host db2201 is UP: PING OK - Packet loss = 0%, RTA = 30.54 ms [15:14:55] (03CR) 10Michael Große: "Looks good, one nit mentioned inline." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352515 (https://phabricator.wikimedia.org/T439738) (owner: 10EMcFarland) [15:15:12] (03PS3) 10Ahmon Dancy: kubernetes.yaml: Revise pretrain logstash thresholds [puppet] - 10https://gerrit.wikimedia.org/r/1345206 (https://phabricator.wikimedia.org/T435419) [15:15:54] (03CR) 10Muehlenhoff: "Just move it under cookbooks.sre.hadoop?" [cookbooks] - 10https://gerrit.wikimedia.org/r/1342235 (https://phabricator.wikimedia.org/T437615) (owner: 10Klausman) [15:16:01] PROBLEM - MariaDB read only x1 on db2201 is CRITICAL: Could not connect to localhost:3320 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [15:16:01] PROBLEM - MariaDB read only s5 on db2201 is CRITICAL: Could not connect to localhost:3315 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [15:16:09] (03Merged) 10jenkins-bot: mw-experimental: declare mesh, mcrouter as native k8s sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352466 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [15:16:09] (03CR) 10Ahmon Dancy: "Acknowledged. Please merge at your leisure, otherwise I can scheduled for the Thursday puppet window." [puppet] - 10https://gerrit.wikimedia.org/r/1345206 (https://phabricator.wikimedia.org/T435419) (owner: 10Ahmon Dancy) [15:16:37] PROBLEM - mysqld processes on db2201 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [15:16:37] PROBLEM - MariaDB Replica SQL: x1 on db2201 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [15:16:37] PROBLEM - MariaDB Replica SQL: s5 on db2201 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [15:16:37] PROBLEM - MariaDB Replica IO: s5 on db2201 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [15:16:37] PROBLEM - MariaDB Replica IO: x1 on db2201 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [15:17:25] 10ops-eqiad, 06SRE, 06DC-Ops, 10Icinga, and 2 others: Check notification status for ps1-d5-eqiad - https://phabricator.wikimedia.org/T439233#12406979 (10Jclark-ctr) DCops is not aware of any disabling notifications. on ps1-d5-eqiad [15:18:02] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-experimental: apply [15:18:30] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply [15:19:03] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 3 others: wikikube-worker1152.eqiad.wmnet networking issue - https://phabricator.wikimedia.org/T438819#12406991 (10Jclark-ctr) a:05Jelto→03Jclark-ctr [15:20:27] RECOVERY - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [15:22:58] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12407022 (10Jclark-ctr) 05Open→03Resolved [15:25:42] 10ops-eqiad, 06DC-Ops: Alert for device ps1-f6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T440479 (10phaultfinder) 03NEW [15:25:50] (03CR) 10Klausman: [V:03+2 C:03+2] "Sure, I can do that. I wasn't sure where to put it initially and just picked a spot." [cookbooks] - 10https://gerrit.wikimedia.org/r/1342235 (https://phabricator.wikimedia.org/T437615) (owner: 10Klausman) [15:32:12] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2201.codfw.wmnet with reason: troubleshooting [15:37:31] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12407131 (10Gehel) [15:37:59] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 06Machine-Learning-Team (Q1 FY2026-27): eqiad row A&B host migration details request for Machine Learning - https://phabricator.wikimedia.org/T432649#12407140 (10Gehel) [15:39:39] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12407188 (10Gehel) [15:40:03] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), and 2 others: eqiad row A&B host migration details request for Machine Learning - https://phabricator.wikimedia.org/T432649#12407213 (10Gehel) [15:40:11] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), and 2 others: eqiad row A&B host migration details request for Machine Learning - https://phabricator.wikimedia.org/T432649#12407217 (10Gehel) 05Resolved→03In progress [15:43:17] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:43:27] (03CR) 10Clare Ming: Deploy GrowthBook to production based on a specific commit (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352490 (https://phabricator.wikimedia.org/T439990) (owner: 10Santiago Faci) [15:43:57] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2127.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:44:03] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2127.codfw.wmnet [15:44:04] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2127.codfw.wmnet [15:44:18] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2128.codfw.wmnet [15:44:24] FIRING: [3x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:xe-1/0/1:1 (Core: cr2-codfw:xe-0/0/1:2) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [15:44:52] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2128.codfw.wmnet [15:45:15] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:45:26] (03PS2) 10Jdlrobson: CentralNotice: Exclude the preferred sources experiment from banners [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344343 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [15:45:32] PROBLEM - Host cr2-codfw is DOWN: PING CRITICAL - Packet loss = 100% [15:45:44] (03PS3) 10Jdlrobson: CentralNotice: Exclude the preferred sources experiment from banners [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344343 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [15:46:17] topranks, papaul: expired downtime for the maintenance? [15:46:22] 10ops-codfw, 06SRE, 06DC-Ops: Enter CyrusOne's Rack Location in Netbox's "Facility ID" Field - https://phabricator.wikimedia.org/T440071#12407291 (10Jhancock.wm) So they're saying they don't have rack IDs but in the recent power incident, they did list the racks by their tile number. So I think that's gonna... [15:46:47] PROBLEM - Host cr2-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [15:46:54] moritzm: I expect so yes, you can ignore it that's the one were working on [15:46:58] (03PS4) 10Jdlrobson: CentralNotice: Exclude the preferred sources experiment from banners [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344343 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [15:47:00] ok! [15:47:00] topranks: ack [15:47:03] thanks [15:47:22] FIRING: CertAlmostExpired: gNMI TLS certificate for cr2-codfw.wikimedia.org is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:47:39] FIRING: [7x] CoreBGPDown: Core BGP session down between cr1-codfw and cr2-codfw (208.80.153.193) - group Confed_codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:47:53] !log pt1979@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: Junos upgrade [15:48:04] RECOVERY - Host cr2-codfw is UP: PING OK - Packet loss = 0%, RTA = 30.48 ms [15:48:06] (03CR) 10Jdlrobson: [C:04-1] "Hey Eric, I made a few tweaks based on the upstream change." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344343 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [15:48:26] (03CR) 10Marostegui: [C:03+1] team-data-persistence: Alarm on stale data in Zarcillo (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) (owner: 10Federico Ceratto) [15:48:26] !log pt1979@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt with reason: Junos upgrade [15:51:10] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr[1-2]-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt with reason: cr2-codfw upgrade [15:51:18] (03PS2) 10Blake: localmemcached,rsyslog: Add restart policy if we should be a sidecar. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334735 (https://phabricator.wikimedia.org/T417800) [15:51:49] RECOVERY - Host cr2-codfw IPv6 is UP: PING OK - Packet loss = 0%, RTA = 30.56 ms [15:52:38] (03PS1) 10Klausman: cookbooks/idm: Move check user leftovers cookbook to under hadoop [cookbooks] - 10https://gerrit.wikimedia.org/r/1352559 (https://phabricator.wikimedia.org/T437615) [15:55:11] 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: ms-be1074: /dev/sdn issue - https://phabricator.wikimedia.org/T440281#12407372 (10CWilliams-WMF) @Jclark-ctr thanks, that is good news! @MatthewVernon should be back on Monday. Given that all drives are showing increased "other error counts", it may be des... [15:55:11] !log elukey@cumin1004 START - Cookbook sre.hosts.reimage for host testvm2001.codfw.wmnet with OS trixie [15:56:05] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (push config only) [15:56:10] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (push config only) [15:56:12] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2128.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:56:23] (03PS1) 10FNegri: Update urldownloader endpoints [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352561 (https://phabricator.wikimedia.org/T439732) [15:56:39] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2128.codfw.wmnet [15:56:41] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2128.codfw.wmnet [15:56:43] (03CR) 10Elukey: [C:03+1] "LGTM thanks!" [cookbooks] - 10https://gerrit.wikimedia.org/r/1352559 (https://phabricator.wikimedia.org/T437615) (owner: 10Klausman) [15:56:55] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2129.codfw.wmnet [15:57:05] (03PS2) 10EMcFarland: Home Experience: Feature flag to enable Home [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352515 (https://phabricator.wikimedia.org/T439738) [15:57:13] (03CR) 10CI reject: [V:04-1] cookbooks/idm: Move check user leftovers cookbook to under hadoop [cookbooks] - 10https://gerrit.wikimedia.org/r/1352559 (https://phabricator.wikimedia.org/T437615) (owner: 10Klausman) [15:57:24] (03CR) 10Herron: "yes exactly, it is meant to be a noop in terms of functionality" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346712 (https://phabricator.wikimedia.org/T438375) (owner: 10Herron) [15:57:29] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2129.codfw.wmnet [15:57:29] (03CR) 10Federico Ceratto: [C:03+2] team-data-persistence: Alarm on stale data in Zarcillo [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) (owner: 10Federico Ceratto) [15:57:48] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for dhardy - https://phabricator.wikimedia.org/T440467#12407396 (10DMburugu) As the manager, I approve. [15:57:49] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:57:57] (03CR) 10FNegri: Update urldownloader endpoints (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352561 (https://phabricator.wikimedia.org/T439732) (owner: 10FNegri) [15:58:02] (03CR) 10EMcFarland: Home Experience: Feature flag to enable Home (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352515 (https://phabricator.wikimedia.org/T439738) (owner: 10EMcFarland) [15:59:46] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [16:00:12] (03Merged) 10jenkins-bot: team-data-persistence: Alarm on stale data in Zarcillo [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) (owner: 10Federico Ceratto) [16:02:25] (03CR) 10Clément Goubert: [C:03+1] localmemcached,rsyslog: Add restart policy if we should be a sidecar. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334735 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [16:02:26] (03PS2) 10FNegri: toolhub: Update urldownloader endpoints [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352561 (https://phabricator.wikimedia.org/T439732) [16:03:09] 10ops-codfw, 06SRE, 10database-backups, 06DBA, 06DC-Ops: db2201 (backup source s5 and x1) crashed - https://phabricator.wikimedia.org/T440283#12407467 (10Jhancock.wm) @Marostegui done [16:03:21] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [16:03:39] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352561 (https://phabricator.wikimedia.org/T439732) (owner: 10FNegri) [16:04:39] 10ops-codfw, 06SRE, 10database-backups, 06DBA, 06DC-Ops: db2201 (backup source s5 and x1) crashed - https://phabricator.wikimedia.org/T440283#12407470 (10Marostegui) Thank you - I am starting mariadb. Next steps I guess would be sending all back to Dell? [16:05:02] RECOVERY - MariaDB read only x1 on db2201 is OK: Version 10.11.18-MariaDB-log, Uptime 59s, read_only: True, event_scheduler: True, 24.32 QPS, connection latency: 0.033522s, query latency: 0.000999s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [16:05:26] (03CR) 10Muehlenhoff: "Make sure to run https://wikitech.wikimedia.org/wiki/Spicerack/Cookbooks#Renaming/Deleting_a_cookbook post deploy" [cookbooks] - 10https://gerrit.wikimedia.org/r/1352559 (https://phabricator.wikimedia.org/T437615) (owner: 10Klausman) [16:06:19] (03CR) 10Clément Goubert: [C:03+2] admin_ng: Add lac-lambdas namespace to wikikube [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347653 (https://phabricator.wikimedia.org/T439879) (owner: 10Clément Goubert) [16:06:38] RECOVERY - mysqld processes on db2201 is OK: PROCS OK: 2 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [16:06:58] (03CR) 10Santiago Faci: Deploy GrowthBook to production based on a specific commit (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352490 (https://phabricator.wikimedia.org/T439990) (owner: 10Santiago Faci) [16:07:01] (03PS9) 10Clément Goubert: revision-feature-counts: add deployment chart files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) [16:07:02] RECOVERY - MariaDB read only s5 on db2201 is OK: Version 10.11.18-MariaDB-log, Uptime 41s, read_only: True, event_scheduler: True, 24.33 QPS, connection latency: 0.031998s, query latency: 0.000598s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [16:08:14] (03CR) 10Blake: [C:03+2] localmemcached,rsyslog: Add restart policy if we should be a sidecar. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334735 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [16:08:14] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2129.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:08:33] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2129.codfw.wmnet [16:09:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker2128.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:09:13] FIRING: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:10:21] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host testvm2001.codfw.wmnet (remove config only) [16:10:24] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2129.codfw.wmnet [16:10:26] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host testvm2001.codfw.wmnet (remove config only) [16:10:37] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2130.codfw.wmnet [16:11:11] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2130.codfw.wmnet [16:11:26] (03Merged) 10jenkins-bot: admin_ng: Add lac-lambdas namespace to wikikube [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347653 (https://phabricator.wikimedia.org/T439879) (owner: 10Clément Goubert) [16:11:32] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:11:42] (03PS1) 10Federico Ceratto: team-data-persistence: Try setting alert only in eqiad [alerts] - 10https://gerrit.wikimedia.org/r/1352562 [16:12:25] (03Abandoned) 10Dduvall: buildx: New driver for building images via Buildx/BuildKit [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1329694 (https://phabricator.wikimedia.org/T434958) (owner: 10Dduvall) [16:12:34] !log elukey@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage [16:13:31] (03PS1) 10Xcollazo: Show the sunset date of the XML dumps [dumps] - 10https://gerrit.wikimedia.org/r/1352564 (https://phabricator.wikimedia.org/T440466) [16:14:13] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker2128.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:14:58] (03Merged) 10jenkins-bot: localmemcached,rsyslog: Add restart policy if we should be a sidecar. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1334735 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [16:15:38] RECOVERY - MariaDB Replica IO: x1 on db2201 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:15:38] RECOVERY - MariaDB Replica SQL: x1 on db2201 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:16:38] RECOVERY - MariaDB Replica IO: s5 on db2201 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:16:38] RECOVERY - MariaDB Replica SQL: s5 on db2201 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:47] (03PS2) 10Klausman: cookbooks/idm: Move check user leftovers cookbook to under hadoop [cookbooks] - 10https://gerrit.wikimedia.org/r/1352559 (https://phabricator.wikimedia.org/T437615) [16:17:19] !log blake@deploy1003 Started scap sync-world: add restart policies for T417800 [16:17:23] T417800: Implement proper sidecar container support in mediawiki pods - https://phabricator.wikimedia.org/T417800 [16:18:07] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2001.codfw.wmnet with reason: host reimage [16:20:28] (03CR) 10Muehlenhoff: [C:03+1] cookbooks/idm: Move check user leftovers cookbook to under hadoop [cookbooks] - 10https://gerrit.wikimedia.org/r/1352559 (https://phabricator.wikimedia.org/T437615) (owner: 10Klausman) [16:21:14] !log blake@deploy1003 Finished scap sync-world: add restart policies for T417800 (duration: 05m 23s) [16:21:36] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-experimental: apply [16:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:22:27] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2130.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:22:30] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply [16:22:33] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2130.codfw.wmnet [16:23:56] !log cgoubert@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'. [16:24:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker2129.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:24:35] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2130.codfw.wmnet [16:24:49] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2131.codfw.wmnet [16:25:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2131.codfw.wmnet [16:25:44] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:26:02] (03CR) 10Clément Goubert: [C:03+2] services_proxy: Enable the linked-artifacts listener for MediaWiki [puppet] - 10https://gerrit.wikimedia.org/r/1351442 (https://phabricator.wikimedia.org/T432733) (owner: 10DLynch) [16:26:28] !log cgoubert@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [16:26:48] !log cgoubert@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [16:27:46] (03CR) 10Klausman: [V:03+2 C:03+2] cookbooks/idm: Move check user leftovers cookbook to under hadoop [cookbooks] - 10https://gerrit.wikimedia.org/r/1352559 (https://phabricator.wikimedia.org/T437615) (owner: 10Klausman) [16:28:33] !log cgoubert@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [16:29:13] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker2129.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:29:32] !log cgoubert@deploy1003 helmfile [codfw] START helmfile.d/admin 'apply'. [16:30:24] (03Merged) 10jenkins-bot: cookbooks/idm: Move check user leftovers cookbook to under hadoop [cookbooks] - 10https://gerrit.wikimedia.org/r/1352559 (https://phabricator.wikimedia.org/T437615) (owner: 10Klausman) [16:31:27] !log cgoubert@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'apply'. [16:31:41] !log cgoubert@deploy1003 helmfile [eqiad] START helmfile.d/admin 'apply'. [16:32:22] jouncebot: nowandnext [16:32:22] No deployments scheduled for the next 0 hour(s) and 27 minute(s) [16:32:23] In 0 hour(s) and 27 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T1700) [16:32:35] !log cgoubert@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'apply'. [16:32:49] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2001.codfw.wmnet with OS trixie [16:33:46] (03CR) 10Herron: [C:03+2] admin: add ealbizzati to group deployment [puppet] - 10https://gerrit.wikimedia.org/r/1352258 (https://phabricator.wikimedia.org/T437741) (owner: 10Herron) [16:35:40] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12407909 (10herron) 05In progress→03Resolved a:03herron The requested access has been provisioned and will be fully deployed within 30 minutes. I'll go a... [16:36:40] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2131.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:36:50] (03PS1) 10Blake: mcrouter: update to 1.3.7 (copy) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352571 (https://phabricator.wikimedia.org/T417800) [16:36:55] (03PS1) 10Blake: mcrouter: Add native sidecar capabilities to the mcrouter exporter [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352572 (https://phabricator.wikimedia.org/T417800) [16:36:58] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2131.codfw.wmnet [16:37:00] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2131.codfw.wmnet [16:37:13] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2132.codfw.wmnet [16:37:42] (03PS2) 10Herron: admin: add user greyolsen to deployment group [puppet] - 10https://gerrit.wikimedia.org/r/1352166 (https://phabricator.wikimedia.org/T439143) [16:37:48] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2132.codfw.wmnet [16:38:07] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:38:19] !log cgoubert@deploy1003 Started scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki | 1351442 | T432733 [16:38:22] T432733: Server Side Edit Suggestions - Linked Artifacts Cache precompute job - https://phabricator.wikimedia.org/T432733 [16:40:45] (03CR) 10Herron: [C:03+2] admin: add user greyolsen to deployment group [puppet] - 10https://gerrit.wikimedia.org/r/1352166 (https://phabricator.wikimedia.org/T439143) (owner: 10Herron) [16:44:28] (03CR) 10Eric Gardner: "Is the new usage documented anywhere? Previously we just added the experiment name, but now there is also a boolean for `bannerExperiment`" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344343 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [16:46:58] (03CR) 10Eric Gardner: "Also, in terms of deploying, I assumed that I would only be backporting a config change. I was assuming we'd wait until the CentralNotice " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344343 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [16:47:32] !log cgoubert@deploy1003 Finished scap sync-world: services_proxy: Enable the linked-artifacts listener for MediaWiki | 1351442 | T432733 (duration: 10m 40s) [16:47:36] T432733: Server Side Edit Suggestions - Linked Artifacts Cache precompute job - https://phabricator.wikimedia.org/T432733 [16:49:00] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2132.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:49:03] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to Deployment for GreyOlson/GOlson-WMF - https://phabricator.wikimedia.org/T439143#12407986 (10herron) 05Open→03Resolved a:03herron The requested access has been provisioned and will be fully deployed within 30 minutes. I'll go ahe... [16:49:18] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2132.codfw.wmnet [16:49:40] FIRING: [2x] KubernetesCalicoDown: wikikube-worker2131.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:50:58] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2132.codfw.wmnet [16:51:08] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for dhardy - https://phabricator.wikimedia.org/T440467#12408017 (10herron) Hi @thcipriani @dancy, could you please review/approve the groupadd to deployment? Thanks! [16:51:12] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2133.codfw.wmnet [16:51:18] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for dhardy - https://phabricator.wikimedia.org/T440467#12408020 (10herron) [16:51:43] (03CR) 10AKhatun: [C:03+1] revision-feature-counts: add deployment chart files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [16:51:46] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2133.codfw.wmnet [16:52:07] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:53:13] (03PS3) 10Jforrester: Abstractwiki: Drop temporary memcached/mainstash config, no longer needed [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1346644 [16:53:13] (03PS1) 10Jforrester: wikifunctions: Drop CodeMirror enablement, defailt on [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352573 (https://phabricator.wikimedia.org/T400015) [16:54:13] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker2131.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:54:18] (03PS5) 10Hashar: Change update to exactly match the given image name [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1166856 [16:54:43] (03CR) 10AKhatun: [C:03+1] revision-feature-counts: service catalog, listener [puppet] - 10https://gerrit.wikimedia.org/r/1347695 (https://phabricator.wikimedia.org/T439879) (owner: 10Clément Goubert) [16:55:02] (03CR) 10AKhatun: [C:03+1] Add revision-feature-counts ingress CNAME [dns] - 10https://gerrit.wikimedia.org/r/1347696 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [16:56:09] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users + Kerberos for OBenhmida - https://phabricator.wikimedia.org/T440464#12408050 (10herron) [16:57:30] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users + Kerberos for OBenhmida - https://phabricator.wikimedia.org/T440464#12408061 (10herron) Hi @OBenhmida could you please coordinate a comment of approval on this task from @SonjaPerry? Thanks! [16:57:48] !log pt1979@cumin1004 START - Cookbook sre.hosts.remove-downtime for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt [16:57:50] !log pt1979@cumin1004 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-codfw,cr2-codfw IPv6,re0.cr2-codfw.mgmt [16:58:38] (03CR) 10Scott French: [C:03+2] kubernetes.yaml: Revise pretrain logstash thresholds [puppet] - 10https://gerrit.wikimedia.org/r/1345206 (https://phabricator.wikimedia.org/T435419) (owner: 10Ahmon Dancy) [16:59:04] !log pt1979@cumin1004 START - Cookbook sre.hosts.remove-downtime for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt [16:59:06] !log pt1979@cumin1004 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr1-codfw,cr1-codfw IPv6,re0.cr1-codfw.mgmt [17:00:05] swfrench-wmf: How many deployers does it take to do MediaWiki infrastructure (UTC late) deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T1700). [17:00:12] o/ [17:00:21] (03CR) 10Scott French: "Thanks for the review!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335270 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [17:00:24] (03CR) 10Scott French: [C:03+2] api-gateway: Support PHP_ENGINE routing in the mw-api plugin [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335270 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [17:00:30] (^.^)/ [17:01:54] (03Merged) 10jenkins-bot: api-gateway: Support PHP_ENGINE routing in the mw-api plugin [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335270 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [17:02:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:02:38] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to dumps-roots for analytics_admins_members (people associated with analytics-admins) - https://phabricator.wikimedia.org/T440471#12408091 (10herron) Thanks for the patch! @Milimetric @Ahoelzl @Ottomata @BTullis could you please have a r... [17:03:00] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2133.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:03:18] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2133.codfw.wmnet [17:03:20] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2133.codfw.wmnet [17:03:33] (03CR) 10Jdlrobson: [C:04-1] "No CentralNotice doesn't get deployed by the train. https://wikitech.wikimedia.org/wiki/CentralNotice#Deployment" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344343 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [17:03:33] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2134.codfw.wmnet [17:04:08] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2134.codfw.wmnet [17:04:28] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:06:36] 10ops-codfw, 06SRE, 06DC-Ops: Enter CyrusOne's Rack Location in Netbox's "Facility ID" Field - https://phabricator.wikimedia.org/T440071#12408109 (10wiki_willy) Thanks Jenn, I'm good with either. As long as we're able to reference back which one goes to which. [17:07:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:07:59] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to dumps-roots for analytics_admins_members (people associated with analytics-admins) - https://phabricator.wikimedia.org/T440471#12408124 (10Ottomata) Approved, this makes sense to me. These are the folks that would have to administer dump... [17:08:27] !log swfrench@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [17:08:49] !log swfrench@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [17:12:02] !log swfrench@deploy1003 helmfile [codfw] START helmfile.d/services/rest-gateway: apply [17:12:37] !log swfrench@deploy1003 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [17:14:02] (03CR) 10Btullis: [C:04-1] "I'm not so sure that we want to go this way, so adding a -1 while we discuss on the ticket." [puppet] - 10https://gerrit.wikimedia.org/r/1352546 (https://phabricator.wikimedia.org/T440471) (owner: 10Dr0ptp4kt) [17:15:21] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2134.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:15:33] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2134.codfw.wmnet [17:15:35] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2134.codfw.wmnet [17:15:48] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2135.codfw.wmnet [17:16:21] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2135.codfw.wmnet [17:16:41] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:19:11] !log swfrench@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [17:19:28] !log dancy@deploy1003 Installing scap version "4.295.0" for 3 host(s) [17:19:35] !log swfrench@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [17:19:41] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to dumps-roots for analytics_admins_members (people associated with analytics-admins) - https://phabricator.wikimedia.org/T440471#12408156 (10BTullis) Thanks for the ping. I'm not so sure that this is the right way to go, at the moment. Goi... [17:21:23] !log dancy@deploy1003 Installation of scap version "4.295.0" completed for 3 hosts [17:24:31] !log swfrench@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [17:24:43] !log swfrench@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [17:25:23] !log swfrench@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [17:25:36] !log swfrench@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [17:26:03] !log swfrench@deploy1003 helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [17:26:24] !log swfrench@deploy1003 helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [17:27:03] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2135.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:27:23] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2135.codfw.wmnet [17:29:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker2134.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:29:14] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2135.codfw.wmnet [17:29:27] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2136.codfw.wmnet [17:29:57] (03PS1) 10DLynch: editcheck-headless: upgrade image to 2026-10-03-040829-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352575 (https://phabricator.wikimedia.org/T432286) [17:30:02] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2136.codfw.wmnet [17:30:21] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:30:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [17:33:56] alright, I believe I am done with the infra window :) [17:34:13] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker2134.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:35:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133216 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [17:37:31] (03PS1) 10Gmodena: wdqs: drop the internal releases [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352576 (https://phabricator.wikimedia.org/T431395) [17:39:28] (03CR) 10Ottomata: [C:03+1] editcheck-headless: upgrade image to 2026-10-03-040829-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352575 (https://phabricator.wikimedia.org/T432286) (owner: 10DLynch) [17:39:44] 10ops-ulsfo, 06SRE, 06DC-Ops: ulsfo: OOB migration from copper to fiber - https://phabricator.wikimedia.org/T436499#12408323 (10RobH) [17:41:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2136.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:41:34] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2136.codfw.wmnet [17:41:35] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2136.codfw.wmnet [17:41:49] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2137.codfw.wmnet [17:42:22] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2137.codfw.wmnet [17:42:42] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:48:23] (03PS1) 10CDanis: haproxy: record Backend-Timing in x-analytics [puppet] - 10https://gerrit.wikimedia.org/r/1352578 [17:48:32] (03PS1) 10Gmodena: wdqs: bump main-external qlever to 60 cores [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352579 (https://phabricator.wikimedia.org/T431395) [17:52:39] FIRING: [2x] TransitBGPDown: Transit BGP session down between cr2-esams and Hurricane Electric (193.239.116.14) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [17:53:08] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2137.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:53:26] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2137.codfw.wmnet [17:54:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker2136.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:55:40] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2137.codfw.wmnet [17:55:53] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2138.codfw.wmnet [17:55:55] (03PS1) 10Xcollazo: dumps: Remove the generator settings from the web cleanups [puppet] - 10https://gerrit.wikimedia.org/r/1352580 (https://phabricator.wikimedia.org/T437463) [17:56:27] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2138.codfw.wmnet [17:56:47] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:57:39] RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr2-esams and Hurricane Electric (193.239.116.14) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown [17:57:51] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 08 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352018 (https://phabricator.wikimedia.org/T437617) (owner: 10Dillon) [17:58:08] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, October 07 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352019 (https://phabricator.wikimedia.org/T437617) (owner: 10Dillon) [17:58:31] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 08 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352019 (https://phabricator.wikimedia.org/T437617) (owner: 10Dillon) [17:59:13] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker2136.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:59:42] (03CR) 10Xcollazo: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352580 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [17:59:44] (03CR) 10BCornwall: [C:03+1] "Looks good. Be sure to update https://wikitech.wikimedia.org/wiki/CDN/Headers#X-Analytics" [puppet] - 10https://gerrit.wikimedia.org/r/1352578 (owner: 10CDanis) [18:00:05] andre and brennen: It is that lovely time of the day again! You are hereby commanded to deploy MediaWiki train - Utc-0+Utc-7 Version (secondary timeslot). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T1800). [18:02:14] o/ nothing for this window. [18:02:26] (03CR) 10Xcollazo: "PPC run shows no changes to `clouddumps*` as expected. LGTM." [puppet] - 10https://gerrit.wikimedia.org/r/1352580 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [18:02:55] (03CR) 10Clare Ming: [C:03+2] Deploy GrowthBook to production based on a specific commit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352490 (https://phabricator.wikimedia.org/T439990) (owner: 10Santiago Faci) [18:04:13] RESOLVED: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:04:17] (03Merged) 10jenkins-bot: Deploy GrowthBook to production based on a specific commit [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352490 (https://phabricator.wikimedia.org/T439990) (owner: 10Santiago Faci) [18:06:11] 10ops-codfw, 06SRE, 06DC-Ops: Enter CyrusOne's Rack Location in Netbox's "Facility ID" Field - https://phabricator.wikimedia.org/T440071#12408469 (10RobH) If we're not listing the full info in the rack, perhaps we can in the row? The facility landing page doesn't really have a good spot for it: codfw facil... [18:07:41] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2138.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [18:07:53] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2138.codfw.wmnet [18:09:11] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2138.codfw.wmnet [18:09:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker2137.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:09:25] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2139.codfw.wmnet [18:09:59] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2139.codfw.wmnet [18:10:19] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [18:10:48] 10ops-codfw, 06SRE, 06DC-Ops: Enter CyrusOne's Rack Location in Netbox's "Facility ID" Field - https://phabricator.wikimedia.org/T440071#12408489 (10RobH) Please note I just appended the cage IDs into the 'notes' field for both codfw and eqiad (so thats new info and perhaps that is good enough for recording... [18:13:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:14:13] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker2137.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:18:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:21:18] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2139.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [18:21:36] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2139.codfw.wmnet [18:21:38] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2139.codfw.wmnet [18:21:52] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2140.codfw.wmnet [18:22:25] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2140.codfw.wmnet [18:22:35] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [18:26:03] (03PS2) 10CDanis: haproxy: record Backend-Timing in x-analytics [puppet] - 10https://gerrit.wikimedia.org/r/1352578 [18:26:25] (03CR) 10CDanis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352578 (owner: 10CDanis) [18:32:07] (03CR) 10CDanis: [C:03+2] "Will do, thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1352578 (owner: 10CDanis) [18:33:32] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2140.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [18:33:49] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2140.codfw.wmnet [18:33:50] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2140.codfw.wmnet [18:33:52] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2124-2140].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [18:34:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker2139.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:34:24] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker2139.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:36:03] FIRING: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:41:03] RESOLVED: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:57:12] (03PS3) 10Xcollazo: dumps: Keep pages-meta-history only in the 3 newest runs on replicas [puppet] - 10https://gerrit.wikimedia.org/r/1346629 (https://phabricator.wikimedia.org/T437463) [19:00:09] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to dumps-roots for analytics_admins_members (people associated with analytics-admins) - https://phabricator.wikimedia.org/T440471#12408619 (10dr0ptp4kt) Seems like a decent approach to me. Will circle back around with an amended CL. [19:02:21] (03PS3) 10Xcollazo: dumps: Write mirror rsync lists that mix v1 dumps and File Exports [puppet] - 10https://gerrit.wikimedia.org/r/1346764 (https://phabricator.wikimedia.org/T437463) [19:03:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:08:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:14:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:14:59] (03PS4) 10Xcollazo: dumps: Write mirror rsync lists that mix v1 dumps and File Exports [puppet] - 10https://gerrit.wikimedia.org/r/1346764 (https://phabricator.wikimedia.org/T437463) [19:16:02] (03CR) 10DLynch: [C:03+2] editcheck-headless: upgrade image to 2026-10-03-040829-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352575 (https://phabricator.wikimedia.org/T432286) (owner: 10DLynch) [19:17:26] (03Merged) 10jenkins-bot: editcheck-headless: upgrade image to 2026-10-03-040829-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352575 (https://phabricator.wikimedia.org/T432286) (owner: 10DLynch) [19:18:32] !log kemayo@deploy1003 helmfile [staging] START helmfile.d/services/editcheck-headless: apply [19:19:06] !log kemayo@deploy1003 helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply [19:25:04] (03CR) 10Trueg: [C:03+1] "Perfectly fine by me. Just keep in mind that for applying this change you will most likely have to delete some topolvm pvs, since you want" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352579 (https://phabricator.wikimedia.org/T431395) (owner: 10Gmodena) [19:26:45] (03CR) 10Gmodena: wdqs: bump main-external qlever to 60 cores (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352579 (https://phabricator.wikimedia.org/T431395) (owner: 10Gmodena) [19:29:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:34:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:35:11] (03PS1) 10Scott French: rest-gateway: Enable diversion of PHP_ENGINE-enrolled traffic to -next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352592 (https://phabricator.wikimedia.org/T433752) [19:42:06] (03CR) 10Scott French: "As described in the commit message, this is the last step to enable PHP_ENGINE diversion at rest-gateway using the new plugin. The feature" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352592 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [19:54:00] !log kemayo@deploy1003 helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply [19:54:34] !log kemayo@deploy1003 helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply [19:55:08] !log kemayo@deploy1003 helmfile [codfw] START helmfile.d/services/editcheck-headless: apply [19:55:45] !log kemayo@deploy1003 helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for UTC late backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T2000). [20:00:05] No Gerrit patches in the queue for this window AFAICS. [20:05:48] (03PS2) 10DLynch: EditSuggestionCounts: Wait for LAC, and retry on timeout [extensions/VisualEditor] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352600 (https://phabricator.wikimedia.org/T432733) [20:06:02] (03CR) 10DLynch: "The tests had moved since the branch was cut." [extensions/VisualEditor] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352600 (https://phabricator.wikimedia.org/T432733) (owner: 10DLynch) [20:06:26] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, October 07 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [extensions/VisualEditor] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352600 (https://phabricator.wikimedia.org/T432733) (owner: 10DLynch) [20:07:00] ^ I am now in this window with that patch. [20:10:45] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy1003 using scap backport" [extensions/VisualEditor] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352600 (https://phabricator.wikimedia.org/T432733) (owner: 10DLynch) [20:17:21] (03Merged) 10jenkins-bot: EditSuggestionCounts: Wait for LAC, and retry on timeout [extensions/VisualEditor] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1352600 (https://phabricator.wikimedia.org/T432733) (owner: 10DLynch) [20:17:42] !log kemayo@deploy1003 Started scap sync-world: Backport for [[gerrit:1352600|EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] [20:17:45] T432733: Server Side Edit Suggestions - Linked Artifacts Cache precompute job - https://phabricator.wikimedia.org/T432733 [20:18:30] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352606 [20:21:56] !log kemayo@deploy1003 kemayo: Backport for [[gerrit:1352600|EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:23:29] !log kemayo@deploy1003 kemayo: Continuing with deployment [20:27:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [20:28:53] !log kemayo@deploy1003 Finished scap sync-world: Backport for [[gerrit:1352600|EditSuggestionCounts: Wait for LAC, and retry on timeout (T432733)]] (duration: 11m 11s) [20:28:57] T432733: Server Side Edit Suggestions - Linked Artifacts Cache precompute job - https://phabricator.wikimedia.org/T432733 [20:29:28] (03CR) 10JHathaway: [C:03+1] Disable pki vhost on port 8443 [puppet] - 10https://gerrit.wikimedia.org/r/1346616 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [20:47:05] (03Abandoned) 10Eevans: cassandra: add linked_artifacts.edit_suggestions_count GRANTS [puppet] - 10https://gerrit.wikimedia.org/r/1327637 (https://phabricator.wikimedia.org/T431499) (owner: 10Eevans) [20:50:47] (03PS1) 10Bking: WIP: Blackbox autodiscovery job for dse-k8s prometheus [puppet] - 10https://gerrit.wikimedia.org/r/1352625 (https://phabricator.wikimedia.org/T436281) [20:51:48] (03PS2) 10Bking: WIP: Blackbox autodiscovery job for dse-k8s prometheus [puppet] - 10https://gerrit.wikimedia.org/r/1352625 (https://phabricator.wikimedia.org/T436281) [20:52:04] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352625 (https://phabricator.wikimedia.org/T436281) (owner: 10Bking) [20:59:47] (03PS2) 10Eevans: linked-artifacts: add edit_suggestions_counts config (production) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327633 (https://phabricator.wikimedia.org/T431499) [21:00:05] Deploy window Wikifunctions Services UTC Late (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T2100) [21:08:52] !log bking@cumin2003 START - Cookbook sre.ganeti.makevm for new host krb1005.eqiad.wmnet [21:08:54] !log bking@cumin2003 START - Cookbook sre.dns.netbox [21:09:24] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:10:50] (03CR) 10JHathaway: [C:03+1] linked-artifacts: add edit_suggestions_counts config (production) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327633 (https://phabricator.wikimedia.org/T431499) (owner: 10Eevans) [21:12:33] (03CR) 10Eevans: [C:03+2] linked-artifacts: add edit_suggestions_counts config (production) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327633 (https://phabricator.wikimedia.org/T431499) (owner: 10Eevans) [21:12:34] (03CR) 10DLynch: linked-artifacts: add edit_suggestions_counts config (production) (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327633 (https://phabricator.wikimedia.org/T431499) (owner: 10Eevans) [21:14:21] (03Merged) 10jenkins-bot: linked-artifacts: add edit_suggestions_counts config (production) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327633 (https://phabricator.wikimedia.org/T431499) (owner: 10Eevans) [21:14:25] bking@cumin2003 makevm (PID 4084733) is awaiting input [21:16:08] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003" [21:16:12] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM krb1005.eqiad.wmnet - bking@cumin2003" [21:16:13] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [21:16:13] !log bking@cumin2003 START - Cookbook sre.dns.wipe-cache krb1005.eqiad.wmnet on all recursors [21:16:17] !log bking@cumin2003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) krb1005.eqiad.wmnet on all recursors [21:16:34] (03PS1) 10Eevans: linked-artifacts: bump edit_suggestions_counts timeout to 30s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352634 (https://phabricator.wikimedia.org/T431499) [21:16:55] !log bking@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003" [21:16:59] !log bking@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM krb1005.eqiad.wmnet - bking@cumin2003" [21:17:06] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host krb1005.eqiad.wmnet with OS bookworm [21:17:13] 06SRE, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Site: EQIAD VM request for Kerberos - https://phabricator.wikimedia.org/T438229#12409114 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2003 for host krb1005.eqiad.wmnet with OS bookworm [21:17:25] (03PS1) 10Ahmon Dancy: profile::puppetserver::wmcs: Set `enc_path` in hiera [puppet] - 10https://gerrit.wikimedia.org/r/1352635 [21:17:40] (03CR) 10Eevans: [C:03+2] linked-artifacts: add edit_suggestions_counts config (production) (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1327633 (https://phabricator.wikimedia.org/T431499) (owner: 10Eevans) [21:18:32] 06SRE, 10Pywikibot, 06Traffic, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error on Wikidata - https://phabricator.wikimedia.org/T421642#12409116 (10ArthurPSmith) Note maxlag continues to be bad for much of the time, preventing bot edits. I see this thread is mostly about pywikibot and reads;... [21:18:51] (03CR) 10Eevans: [C:03+2] linked-artifacts: bump edit_suggestions_counts timeout to 30s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352634 (https://phabricator.wikimedia.org/T431499) (owner: 10Eevans) [21:20:17] (03Merged) 10jenkins-bot: linked-artifacts: bump edit_suggestions_counts timeout to 30s [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352634 (https://phabricator.wikimedia.org/T431499) (owner: 10Eevans) [21:21:26] !log eevans@deploy1003 helmfile [codfw] START helmfile.d/services/linked-artifacts: apply [21:22:00] !log eevans@deploy1003 helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply [21:22:08] !log eevans@deploy1003 helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply [21:22:28] !log eevans@deploy1003 helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply [21:27:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [21:32:55] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage [21:36:32] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on krb1005.eqiad.wmnet with reason: host reimage [21:38:22] (03PS2) 10Ahmon Dancy: profile::puppetserver::volatile: Allow use in Cloud VPS [puppet] - 10https://gerrit.wikimedia.org/r/1352635 [21:43:28] (03CR) 10Ahmon Dancy: "Please see https://gerrit.wikimedia.org/r/c/operations/puppet/+/1352635" [puppet] - 10https://gerrit.wikimedia.org/r/1137013 (owner: 10Krinkle) [21:44:39] (03CR) 10Ahmon Dancy: [V:03+1] "Cherry-picked on deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud" [puppet] - 10https://gerrit.wikimedia.org/r/1352635 (owner: 10Ahmon Dancy) [21:51:33] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-f6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T440479#12409204 (10Jclark-ctr) ps1-f6-eqiad.mgmt.eqiad.wmnet #1: Sensor: Phase, AA:L2-L3, Active Power Value: 1.685 kW (power) Thresholds: High: 1650 #2: Sensor: Phase,... [21:52:57] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host krb1005.eqiad.wmnet with OS bookworm [21:52:58] !log bking@cumin2003 END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host krb1005.eqiad.wmnet [21:53:10] 06SRE, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): Site: EQIAD VM request for Kerberos - https://phabricator.wikimedia.org/T438229#12409206 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2003 for host krb1005.eqiad.wmnet with OS bookworm com... [21:53:56] 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-f6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T440479#12409207 (10Jclark-ctr) 05Open→03Resolved a:03Jclark-ctr rebalanced [22:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261007T2200) [22:04:52] !log jclark@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-druid1003.eqiad.wmnet with OS bookworm [22:05:34] !log jclark@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [22:07:17] !log jclark@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1152.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [22:13:41] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 3 others: wikikube-worker1152.eqiad.wmnet networking issue - https://phabricator.wikimedia.org/T438819#12409234 (10Jclark-ctr) I think the board has other issues passed the nic. i am unable to use keyboard is inop it seems to boot fine i doubt its w... [22:31:19] (03PS1) 10Andrew Bogott: aptrepo: Add k8s v1.34, remove k8s v1.32 [puppet] - 10https://gerrit.wikimedia.org/r/1352642 (https://phabricator.wikimedia.org/T414595) [22:33:29] (03PS2) 10Andrew Bogott: aptrepo: Add k8s v1.34, remove k8s v1.32 [puppet] - 10https://gerrit.wikimedia.org/r/1352642 (https://phabricator.wikimedia.org/T414595) [22:33:33] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for dhardy - https://phabricator.wikimedia.org/T440467#12409259 (10thcipriani) Reason for access makes sense for `deployment` group membership. Approved! --- @Dillon I approved your SpiderPig access request as well. Please ensure you've joined the... [22:39:32] (03PS3) 10Andrew Bogott: aptrepo: Add k8s v1.34, remove k8s v1.32 [puppet] - 10https://gerrit.wikimedia.org/r/1352642 (https://phabricator.wikimedia.org/T414595) [22:40:20] (03PS4) 10Andrew Bogott: aptrepo: Add kubeadm v1.34, remove kubeadm v1.32 [puppet] - 10https://gerrit.wikimedia.org/r/1352642 (https://phabricator.wikimedia.org/T414595) [23:12:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [23:17:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [23:18:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [23:28:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [23:30:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [23:35:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [23:38:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [23:38:58] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1352646 [23:38:58] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1352646 (owner: 10TrainBranchBot) [23:43:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [23:45:45] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1352646 (owner: 10TrainBranchBot) [23:59:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures