[00:00:12] (03CR) 10Dzahn: [C:03+1] gerrit: cap index query results at 5000 [puppet] - 10https://gerrit.wikimedia.org/r/1308081 (https://phabricator.wikimedia.org/T431398) (owner: 10Arnaudb) [00:07:28] PROBLEM - Check unit status of kube-reload-certs on deploy1003 is CRITICAL: CRITICAL: Status of the systemd unit kube-reload-certs https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [00:08:35] (03CR) 10Dzahn: [C:03+2] gerrit: block Google fonts on the Documentation pages with CSP [puppet] - 10https://gerrit.wikimedia.org/r/1351534 (https://phabricator.wikimedia.org/T393763) (owner: 10Dzahn) [00:09:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [00:10:03] FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [00:10:04] (03Abandoned) 10Krinkle: puppetserver::volatile: Do not include profile::puppetserver [puppet] - 10https://gerrit.wikimedia.org/r/1137013 (owner: 10Krinkle) [00:12:46] PROBLEM - dump of s5 in codfw on backupmon1001 is CRITICAL: dump for s5 at codfw (db2201) taken more than a week ago: Most recent backup 2026-09-29 00:00:07 https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Rerun_a_failed_backup [00:15:03] RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures [00:17:42] PROBLEM - snapshot of s4 in eqiad on backupmon1001 is CRITICAL: Last snapshot for s4 at eqiad (db1265) taken on 2026-10-07 23:30:44 is 1137 GiB, but the previous one was 2102 GiB, a change of -45.9 % https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Rerun_a_failed_backup [00:18:00] !log gerrit - added content-security-policy headers on - style-src 'self' 'unsafe-inline'; font-src 'self' - blocking Google font (T393763) [00:18:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:18:03] T393763: Gerrit API docs loads fonts from Google - https://phabricator.wikimedia.org/T393763 [00:59:10] (03PS10) 10AKhatun: revision-feature-counts: add deployment chart files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [01:01:09] (03CR) 10AKhatun: "Made config name changes per the rename. Added tracing." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [01:07:48] (03PS11) 10AKhatun: revision-feature-counts: add deployment chart files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [01:08:14] (03CR) 10AKhatun: revision-feature-counts: add deployment chart files (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [01:09:04] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1352662 [01:09:05] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1352662 (owner: 10TrainBranchBot) [01:09:25] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:15:47] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1352662 (owner: 10TrainBranchBot) [01:33:27] (03PS1) 10Dzahn: miscweb: attribution: bump image to 2026-10-08-012832 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352674 (https://phabricator.wikimedia.org/T437635) [01:34:47] (03CR) 10Dzahn: [C:03+2] miscweb: attribution: bump image to 2026-10-08-012832 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352674 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn) [01:36:21] (03Merged) 10jenkins-bot: miscweb: attribution: bump image to 2026-10-08-012832 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352674 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn) [01:37:37] !log dzahn@deploy1003 helmfile [staging] START helmfile.d/services/miscweb: apply [01:38:02] !log dzahn@deploy1003 helmfile [staging] DONE helmfile.d/services/miscweb: apply [01:38:08] !log dzahn@deploy1003 helmfile [codfw] START helmfile.d/services/miscweb: apply [01:38:29] !log dzahn@deploy1003 helmfile [codfw] DONE helmfile.d/services/miscweb: apply [01:38:37] !log dzahn@deploy1003 helmfile [eqiad] START helmfile.d/services/miscweb: apply [01:38:51] !log dzahn@deploy1003 helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [01:54:30] (03CR) 10Lerickson: [C:03+1] wdqs: drop the internal releases [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352576 (https://phabricator.wikimedia.org/T431395) (owner: 10Gmodena) [01:55:17] (03CR) 10Lerickson: [C:03+1] "I homedir deployed this to main/eqiad/wdqs and didn't need to do anything special with PVCs. Seems like the pods were able to stay where t" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352579 (https://phabricator.wikimedia.org/T431395) (owner: 10Gmodena) [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T0200) [02:00:32] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:07:55] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 23s) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:13:39] (03CR) 10Mamouri: "Removing my -1: the approach here has been agreed on the Persian Wikipedia discussion." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352316 (https://phabricator.wikimedia.org/T124168) (owner: 10Mamouri) [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:36:06] (03CR) 10Ladsgroup: [C:03+2] thumbor-plugins: Rebuild to include latest librsvg backport [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1352505 (owner: 10Muehlenhoff) [02:43:09] (03Merged) 10jenkins-bot: thumbor-plugins: Rebuild to include latest librsvg backport [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1352505 (owner: 10Muehlenhoff) [03:08:04] (03PS1) 10KartikMistry: Update Recommendation API to 2026-10-07-145851-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352720 (https://phabricator.wikimedia.org/T437213) [03:16:48] (03PS1) 10KartikMistry: machinetranslation: Update to 2026-09-21-112314-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352722 (https://phabricator.wikimedia.org/T437213) [03:34:10] (03PS3) 10AKhatun: Add article-feature-counts alerts [alerts] - 10https://gerrit.wikimedia.org/r/1344530 (https://phabricator.wikimedia.org/T438384) [03:39:01] (03PS1) 10Seanleong-wmde: Setting stop sending LL notification as default for all groups [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352724 (https://phabricator.wikimedia.org/T437577) [03:47:43] (03PS2) 10KartikMistry: machinetranslation: Update to 2026-09-21-112314-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352722 (https://phabricator.wikimedia.org/T437213) [03:56:26] !log oblivian@cumin1004 START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [03:56:30] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1015.eqiad.wmnet [03:56:49] !log oblivian@cumin1004 START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[2141-2179].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [03:56:53] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2141.codfw.wmnet [03:57:02] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1015.eqiad.wmnet [03:57:22] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [03:57:26] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2141.codfw.wmnet [03:57:47] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:07:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1015.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:07:21] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1015.eqiad.wmnet [04:08:14] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2141.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:08:33] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2141.codfw.wmnet [04:08:57] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1015.eqiad.wmnet [04:09:11] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1016.eqiad.wmnet [04:09:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1015.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [04:09:25] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1015.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [04:09:56] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1016.eqiad.wmnet [04:10:16] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:10:34] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2141.codfw.wmnet [04:10:48] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2142.codfw.wmnet [04:11:21] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2142.codfw.wmnet [04:11:42] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:14:13] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker1015.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [04:20:07] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1016.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:20:26] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1016.eqiad.wmnet [04:21:37] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2142.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:21:56] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2142.codfw.wmnet [04:22:34] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1016.eqiad.wmnet [04:22:48] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1021.eqiad.wmnet [04:23:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1021.eqiad.wmnet [04:23:42] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:24:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1015.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [04:24:25] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1015.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [04:24:42] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2142.codfw.wmnet [04:24:56] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2143.codfw.wmnet [04:25:29] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2143.codfw.wmnet [04:25:49] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:29:13] RESOLVED: [4x] KubernetesCalicoDown: wikikube-worker1015.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [04:31:32] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1021.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:31:38] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1021.eqiad.wmnet [04:33:57] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1021.eqiad.wmnet [04:34:11] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1034.eqiad.wmnet [04:34:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1016.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [04:34:25] RESOLVED: [3x] KubernetesCalicoDown: wikikube-worker1016.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [04:34:45] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1034.eqiad.wmnet [04:34:54] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:38:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2143.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:38:51] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2143.codfw.wmnet [04:39:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1016.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [04:39:58] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2143.codfw.wmnet [04:40:12] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2144.codfw.wmnet [04:40:46] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2144.codfw.wmnet [04:40:55] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:44:13] RESOLVED: [3x] KubernetesCalicoDown: wikikube-worker1021.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [04:44:16] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1034.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:44:37] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1034.eqiad.wmnet [04:46:26] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1034.eqiad.wmnet [04:46:40] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1035.eqiad.wmnet [04:47:13] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1035.eqiad.wmnet [04:47:33] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:51:48] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2144.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:52:07] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2144.codfw.wmnet [04:53:36] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2144.codfw.wmnet [04:53:50] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2145.codfw.wmnet [04:54:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1021.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [04:54:24] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2145.codfw.wmnet [04:54:25] RESOLVED: [3x] KubernetesCalicoDown: wikikube-worker1034.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [04:54:44] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:56:54] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1035.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [04:57:13] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1035.eqiad.wmnet [04:58:39] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1035.eqiad.wmnet [04:58:52] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1036.eqiad.wmnet [04:59:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1034.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [04:59:29] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1036.eqiad.wmnet [04:59:48] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:04:13] RESOLVED: [3x] KubernetesCalicoDown: wikikube-worker1034.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:05:39] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2145.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:05:57] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2145.codfw.wmnet [05:07:11] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1036.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:07:22] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1036.eqiad.wmnet [05:07:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1036.eqiad.wmnet [05:07:37] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1037.eqiad.wmnet [05:07:37] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2145.codfw.wmnet [05:07:52] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2146.codfw.wmnet [05:08:13] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1037.eqiad.wmnet [05:08:25] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2146.codfw.wmnet [05:08:32] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:08:45] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:09:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1035.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:09:25] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:09:25] RESOLVED: [4x] KubernetesCalicoDown: wikikube-worker1035.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:15:25] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1037.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:15:43] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1037.eqiad.wmnet [05:15:44] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1037.eqiad.wmnet [05:15:58] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1038.eqiad.wmnet [05:16:33] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1038.eqiad.wmnet [05:16:53] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:19:10] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2146.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:19:23] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2146.codfw.wmnet [05:19:25] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1036.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:21:14] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2146.codfw.wmnet [05:21:27] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2147.codfw.wmnet [05:22:01] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2147.codfw.wmnet [05:22:22] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:24:13] RESOLVED: [4x] KubernetesCalicoDown: wikikube-worker1036.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:24:43] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1038.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:25:10] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1038.eqiad.wmnet [05:26:58] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1038.eqiad.wmnet [05:27:12] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1039.eqiad.wmnet [05:27:44] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1039.eqiad.wmnet [05:28:03] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:32:46] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2147.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:33:04] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2147.codfw.wmnet [05:34:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1037.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:35:06] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2147.codfw.wmnet [05:35:19] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2148.codfw.wmnet [05:35:53] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1039.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:35:53] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2148.codfw.wmnet [05:36:04] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1039.eqiad.wmnet [05:36:13] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:38:12] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1039.eqiad.wmnet [05:38:26] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1040.eqiad.wmnet [05:39:02] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1040.eqiad.wmnet [05:39:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1038.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:39:22] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:39:25] RESOLVED: [4x] KubernetesCalicoDown: wikikube-worker1038.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:46:36] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2148.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:46:54] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2148.codfw.wmnet [05:49:06] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2148.codfw.wmnet [05:49:12] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1040.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:49:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1039.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:49:20] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2149.codfw.wmnet [05:49:28] (03PS1) 10Ladsgroup: thumbor: Bump to the most recent image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352885 [05:49:40] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1040.eqiad.wmnet [05:49:55] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2149.codfw.wmnet [05:50:15] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:50:45] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1040.eqiad.wmnet [05:50:59] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1041.eqiad.wmnet [05:51:31] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1041.eqiad.wmnet [05:51:50] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [05:54:13] RESOLVED: [4x] KubernetesCalicoDown: wikikube-worker1039.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:55:20] (03CR) 10Ladsgroup: [C:03+2] thumbor: Bump to the most recent image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352885 (owner: 10Ladsgroup) [05:56:52] (03Merged) 10jenkins-bot: thumbor: Bump to the most recent image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352885 (owner: 10Ladsgroup) [05:57:11] (03CR) 10Ladsgroup: [C:04-1] "This needs sign off from reader growth/exp PMs (probably Olga should be aware at least)." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352316 (https://phabricator.wikimedia.org/T124168) (owner: 10Mamouri) [05:58:50] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [05:59:05] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [05:59:42] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1041.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T0600) [06:00:05] marostegui, cezmunsta, and federico3: #bothumor I � Unicode. All rise for Primary database switchover deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T0600). [06:00:10] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1041.eqiad.wmnet [06:00:22] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [06:01:08] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2149.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:01:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1041.eqiad.wmnet [06:01:28] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2149.codfw.wmnet [06:01:29] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1042.eqiad.wmnet [06:01:29] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2149.codfw.wmnet [06:01:43] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2150.codfw.wmnet [06:02:05] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1042.eqiad.wmnet [06:02:14] !log marostegui@cumin1004 START - Cookbook sre.mysql.depool depool db1262: preparing backups [06:02:18] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2150.codfw.wmnet [06:02:24] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:02:36] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1262: preparing backups [06:02:38] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:02:59] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1262.eqiad.wmnet with reason: preparing backups [06:06:33] (03PS1) 10Marostegui: db1265: Change buffer pool for s4 and s3 [puppet] - 10https://gerrit.wikimedia.org/r/1352895 (https://phabricator.wikimedia.org/T437279) [06:09:08] (03PS1) 10Ladsgroup: [Proposal] varnish: Tighten rate limit on upload cluster [puppet] - 10https://gerrit.wikimedia.org/r/1352896 (https://phabricator.wikimedia.org/T427465) [06:09:52] !log ladsgroup@deploy1003 helmfile [codfw] START helmfile.d/services/thumbor: apply [06:10:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1042.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:10:33] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1042.eqiad.wmnet [06:12:08] !log ladsgroup@deploy1003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [06:12:10] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1042.eqiad.wmnet [06:12:23] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1043.eqiad.wmnet [06:12:54] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1043.eqiad.wmnet [06:13:02] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2150.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:13:14] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:13:19] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2150.codfw.wmnet [06:14:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1041.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:14:48] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2150.codfw.wmnet [06:15:02] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2151.codfw.wmnet [06:15:36] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2151.codfw.wmnet [06:15:56] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:18:18] (03CR) 10Marostegui: [C:03+1] "[06:06:54] FIRING: ZarcilloDataIngestionMetricAbsent: Zarcillo data ingestion metric is missing - https://wikitech.wikimedia." [alerts] - 10https://gerrit.wikimedia.org/r/1351165 (https://phabricator.wikimedia.org/T438724) (owner: 10Federico Ceratto) [06:19:13] RESOLVED: [4x] KubernetesCalicoDown: wikikube-worker1041.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:19:15] (03CR) 10Marostegui: [C:03+2] db1265: Change buffer pool for s4 and s3 [puppet] - 10https://gerrit.wikimedia.org/r/1352895 (https://phabricator.wikimedia.org/T437279) (owner: 10Marostegui) [06:19:51] !log ladsgroup@deploy1003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [06:20:47] (03CR) 10Hashar: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343955 (https://phabricator.wikimedia.org/T384939) (owner: 10Hnowlan) [06:21:05] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1043.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:21:10] 10SRE-swift-storage: swift not found trying to revert image no Commons - https://phabricator.wikimedia.org/T439477#12409717 (10Pppery) [06:21:11] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1043.eqiad.wmnet [06:21:58] !log ladsgroup@deploy1003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [06:22:48] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1043.eqiad.wmnet [06:23:02] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1044.eqiad.wmnet [06:23:37] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1044.eqiad.wmnet [06:23:54] (03PS1) 10Marostegui: mariadb: Add x4 codfw backups [puppet] - 10https://gerrit.wikimedia.org/r/1352900 (https://phabricator.wikimedia.org/T437279) [06:23:57] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:26:49] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2151.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:27:08] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2151.codfw.wmnet [06:27:10] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2151.codfw.wmnet [06:27:23] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2152.codfw.wmnet [06:27:58] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2152.codfw.wmnet [06:28:17] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:28:19] (03CR) 10Muehlenhoff: [C:03+2] Remove unpriv_cumin_masters Hiera key and alertmanager config [puppet] - 10https://gerrit.wikimedia.org/r/1352510 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [06:31:39] (03PS2) 10Marostegui: mariadb: Add x4 codfw backups [puppet] - 10https://gerrit.wikimedia.org/r/1352900 (https://phabricator.wikimedia.org/T437279) [06:31:46] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1044.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:32:04] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1044.eqiad.wmnet [06:32:26] 10SRE-swift-storage: swift not found trying to revert image on Commons - https://phabricator.wikimedia.org/T439477#12409735 (10Ladsgroup) [06:33:53] (03PS1) 10Muehlenhoff: Remove role/profiles for rootless Cumin [puppet] - 10https://gerrit.wikimedia.org/r/1352905 (https://phabricator.wikimedia.org/T439522) [06:34:02] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1044.eqiad.wmnet [06:34:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1043.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:34:15] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1045.eqiad.wmnet [06:35:21] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1045.eqiad.wmnet [06:35:42] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:37:29] !log mwscript-k8s --follow --dblist=ores -- extensions/ORES/maintenance/PurgeScoreCache.php --model articlequality --old (T440358) [06:37:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:37:32] T440358: Lots of old data in ores_classification table - https://phabricator.wikimedia.org/T440358 [06:39:10] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2152.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:39:13] RESOLVED: [3x] KubernetesCalicoDown: wikikube-worker1043.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:39:29] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2152.codfw.wmnet [06:40:47] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2152.codfw.wmnet [06:41:01] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2153.codfw.wmnet [06:41:35] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2153.codfw.wmnet [06:41:54] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:44:05] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1045.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:44:11] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1045.eqiad.wmnet [06:44:12] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1045.eqiad.wmnet [06:44:25] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1046.eqiad.wmnet [06:45:03] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1046.eqiad.wmnet [06:45:23] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:47:13] (03CR) 10Marostegui: "For now just adding codfw, I will do a manual run." [puppet] - 10https://gerrit.wikimedia.org/r/1352900 (https://phabricator.wikimedia.org/T437279) (owner: 10Marostegui) [06:47:15] (03CR) 10Marostegui: [C:03+2] mariadb: Add x4 codfw backups [puppet] - 10https://gerrit.wikimedia.org/r/1352900 (https://phabricator.wikimedia.org/T437279) (owner: 10Marostegui) [06:49:36] (03PS1) 10Anzx: nrwiki, xhwiki: update logo and tagline [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352889 (https://phabricator.wikimedia.org/T440138) [06:50:31] (03PS1) 10Filippo Giunchedi: install_server: generic reuse-parts recipe for standard configs [puppet] - 10https://gerrit.wikimedia.org/r/1352912 (https://phabricator.wikimedia.org/T439831) [06:51:07] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 08 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352889 (https://phabricator.wikimedia.org/T440138) (owner: 10Anzx) [06:52:47] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2153.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:53:05] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2153.codfw.wmnet [06:54:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1044.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:54:14] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1046.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:54:33] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1046.eqiad.wmnet [06:54:34] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1046.eqiad.wmnet [06:54:48] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1047.eqiad.wmnet [06:55:07] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2153.codfw.wmnet [06:55:21] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2154.codfw.wmnet [06:55:21] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1047.eqiad.wmnet [06:55:41] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:55:55] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2154.codfw.wmnet [06:56:05] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [06:59:13] RESOLVED: [4x] KubernetesCalicoDown: wikikube-worker1045.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:00:05] Amir1, urbanecm, and awight: I, the Bot under the Fountain, call upon thee, The Deployer, to do UTC morning backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T0700). [07:00:05] anzx: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:00:25] o/ [07:02:41] (03PS1) 10Muehlenhoff: Remove the labtest-roots group [puppet] - 10https://gerrit.wikimedia.org/r/1352915 [07:02:53] (03PS2) 10Muehlenhoff: Remove the labtest-roots group [puppet] - 10https://gerrit.wikimedia.org/r/1352915 [07:04:33] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1047.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:05:02] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1047.eqiad.wmnet [07:06:49] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1047.eqiad.wmnet [07:06:59] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2154.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:07:02] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1048.eqiad.wmnet [07:07:17] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2154.codfw.wmnet [07:07:35] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1048.eqiad.wmnet [07:07:42] (03PS1) 10Marostegui: db1265: Add x4 backup [puppet] - 10https://gerrit.wikimedia.org/r/1352920 (https://phabricator.wikimedia.org/T431115) [07:07:55] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:08:35] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2154.codfw.wmnet [07:08:49] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2155.codfw.wmnet [07:09:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1046.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:09:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2155.codfw.wmnet [07:09:25] RESOLVED: [4x] KubernetesCalicoDown: wikikube-worker1046.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:09:43] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:11:24] (03CR) 10Marostegui: [C:03+2] db1265: Add x4 backup [puppet] - 10https://gerrit.wikimedia.org/r/1352920 (https://phabricator.wikimedia.org/T431115) (owner: 10Marostegui) [07:12:10] (03Abandoned) 10Brouberol: deployment_server: define the postgresql-growthbook-fundraising kubeconfigs [puppet] - 10https://gerrit.wikimedia.org/r/1345540 (https://phabricator.wikimedia.org/T438350) (owner: 10Brouberol) [07:12:13] (03Abandoned) 10Brouberol: dse-k8s-eqiad: define the postgresql-growthbook-fundraising ns [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345537 (https://phabricator.wikimedia.org/T438350) (owner: 10Brouberol) [07:12:17] (03Abandoned) 10Brouberol: dse-k8s-eqiad: define the postgresql-growthbook-fundraising release [deployment-charts] - 10https://gerrit.wikimedia.org/r/1345538 (https://phabricator.wikimedia.org/T438350) (owner: 10Brouberol) [07:12:18] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1265.eqiad.wmnet with reason: preparing backups [07:12:20] (03Abandoned) 10Brouberol: Define the postgresql-growthbook-fundraising discovery record [dns] - 10https://gerrit.wikimedia.org/r/1345543 (https://phabricator.wikimedia.org/T438350) (owner: 10Brouberol) [07:12:46] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [07:13:15] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [07:13:50] !log brouberol@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [07:14:29] !log brouberol@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [07:15:25] (03CR) 10Brouberol: [C:03+1] "LG!" [puppet] - 10https://gerrit.wikimedia.org/r/1352580 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [07:15:42] (03PS3) 10Anzx: knwikisource: lift IP cap for workshop on 09-October-2026 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352919 (https://phabricator.wikimedia.org/T440546) [07:16:29] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 08 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352919 (https://phabricator.wikimedia.org/T440546) (owner: 10Anzx) [07:16:45] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1048.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:17:03] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1048.eqiad.wmnet [07:17:04] (03CR) 10Slyngshede: [C:03+1] idp-test: Configure to use the new Trixie/MDB LDAP servers for tests [puppet] - 10https://gerrit.wikimedia.org/r/1352525 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [07:17:14] anzx: I'm around now [07:17:37] Amir1: ok [07:17:43] (03CR) 10Brouberol: dumps: Keep pages-meta-history only in the 3 newest runs on replicas (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1346629 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [07:18:39] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1048.eqiad.wmnet [07:18:53] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1049.eqiad.wmnet [07:19:02] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352889 (https://phabricator.wikimedia.org/T440138) (owner: 10Anzx) [07:19:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1047.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:19:29] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1049.eqiad.wmnet [07:19:48] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:20:04] (03Merged) 10jenkins-bot: nrwiki, xhwiki: update logo and tagline [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352889 (https://phabricator.wikimedia.org/T440138) (owner: 10Anzx) [07:20:38] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2155.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:20:51] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1352889|nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] [07:20:56] T440138: Add wordmark for Ndebele Wikipedia - https://phabricator.wikimedia.org/T440138 [07:20:56] T440139: Add wordmark for Xhosa Wikipedia - https://phabricator.wikimedia.org/T440139 [07:20:56] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2155.codfw.wmnet [07:20:58] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2155.codfw.wmnet [07:21:12] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2156.codfw.wmnet [07:21:46] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2156.codfw.wmnet [07:22:06] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:24:13] RESOLVED: [4x] KubernetesCalicoDown: wikikube-worker1047.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:25:19] !log ladsgroup@deploy1003 anzx, ladsgroup: Backport for [[gerrit:1352889|nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:25:24] Looking [07:25:27] (03CR) 10Brouberol: "Could you explain why we're doing this? I'm gathering that we're replacing some dumpsv1 artifacts by their v2 counterpart, but is that onl" [puppet] - 10https://gerrit.wikimedia.org/r/1346764 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [07:26:26] Amir1: both logos looks good, ok to continue [07:26:52] (03CR) 10Brouberol: [C:03+1] "Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1348874 (owner: 10Hashar) [07:26:55] (03CR) 10Brouberol: [C:03+2] install_server: fix spec to show all examples [puppet] - 10https://gerrit.wikimedia.org/r/1348874 (owner: 10Hashar) [07:28:39] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1049.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:28:50] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1049.eqiad.wmnet [07:29:10] on it [07:29:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1048.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:29:16] !log ladsgroup@deploy1003 anzx, ladsgroup: Continuing with deployment [07:31:09] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1049.eqiad.wmnet [07:31:23] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1050.eqiad.wmnet [07:32:31] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1050.eqiad.wmnet [07:32:51] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:33:04] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2156.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:33:22] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2156.codfw.wmnet [07:33:24] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2156.codfw.wmnet [07:33:38] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2157.codfw.wmnet [07:34:12] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2157.codfw.wmnet [07:34:13] RESOLVED: [4x] KubernetesCalicoDown: wikikube-worker1048.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:34:22] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:34:36] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1352889|nrwiki, xhwiki: update logo and tagline (T440138 T440139)]] (duration: 13m 44s) [07:34:40] T440138: Add wordmark for Ndebele Wikipedia - https://phabricator.wikimedia.org/T440138 [07:34:40] T440139: Add wordmark for Xhosa Wikipedia - https://phabricator.wikimedia.org/T440139 [07:35:24] (03CR) 10Muehlenhoff: [C:03+2] idp-test: Configure to use the new Trixie/MDB LDAP servers for tests [puppet] - 10https://gerrit.wikimedia.org/r/1352525 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [07:37:07] (03CR) 10Gmodena: [C:03+2] wdqs: drop the internal releases [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352576 (https://phabricator.wikimedia.org/T431395) (owner: 10Gmodena) [07:37:13] (03CR) 10Gmodena: [C:03+2] wdqs: bump main-external qlever to 60 cores [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352579 (https://phabricator.wikimedia.org/T431395) (owner: 10Gmodena) [07:38:37] (03Merged) 10jenkins-bot: wdqs: drop the internal releases [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352576 (https://phabricator.wikimedia.org/T431395) (owner: 10Gmodena) [07:38:58] (03Merged) 10jenkins-bot: wdqs: bump main-external qlever to 60 cores [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352579 (https://phabricator.wikimedia.org/T431395) (owner: 10Gmodena) [07:42:09] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1050.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:42:37] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1050.eqiad.wmnet [07:44:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1049.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:44:28] ACKNOWLEDGEMENT - snapshot of s4 in eqiad on backupmon1001 is CRITICAL: Last snapshot for s4 at eqiad (db1265) taken on 2026-10-07 23:30:44 is 1137 GiB, but the previous one was 2102 GiB, a change of -45.9 % Marostegui Expected https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Rerun_a_failed_backup [07:44:56] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1050.eqiad.wmnet [07:45:10] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1051.eqiad.wmnet [07:45:18] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2157.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:45:36] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2157.codfw.wmnet [07:45:38] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2157.codfw.wmnet [07:45:43] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1051.eqiad.wmnet [07:45:52] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2158.codfw.wmnet [07:46:02] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:46:26] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2158.codfw.wmnet [07:46:46] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:47:45] (03PS1) 10Muehlenhoff: acmechief: Add additional SNIs for ldap-ro-test.[eqiad|codfw] [puppet] - 10https://gerrit.wikimedia.org/r/1352988 (https://phabricator.wikimedia.org/T439730) [07:47:55] (03PS2) 10Muehlenhoff: acmechief: Add additional SNIs for ldap-ro-test.[eqiad|codfw] [puppet] - 10https://gerrit.wikimedia.org/r/1352988 (https://phabricator.wikimedia.org/T439730) [07:49:13] RESOLVED: [4x] KubernetesCalicoDown: wikikube-worker1049.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:49:53] (03Abandoned) 10Cathal Mooney: Add user account for Abi Bendall to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1309136 (https://phabricator.wikimedia.org/T430938) (owner: 10Cathal Mooney) [07:50:26] oblivian@cumin1004 enable_virtualization (PID 3622331) is awaiting input [07:50:51] (03PS1) 10Dpogorzelski: team-data-platform: add Hummingbird alerts [alerts] - 10https://gerrit.wikimedia.org/r/1352991 (https://phabricator.wikimedia.org/T440547) [07:51:48] (03PS2) 10Dpogorzelski: dpe: add Hummingbird alerts [alerts] - 10https://gerrit.wikimedia.org/r/1352991 (https://phabricator.wikimedia.org/T440547) [07:51:57] PROBLEM - dump of x4 in codfw on backupmon1001 is CRITICAL: We could not find any completed dump for x4 at codfw https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Rerun_a_failed_backup [07:53:51] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1051.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:54:19] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1051.eqiad.wmnet [07:54:20] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1051.eqiad.wmnet [07:54:25] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1050.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:54:33] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1052.eqiad.wmnet [07:55:05] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1052.eqiad.wmnet [07:55:24] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [07:56:57] PROBLEM - snapshot of x4 in codfw on backupmon1001 is CRITICAL: We could not find any completed snapshot for x4 at codfw https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Rerun_a_failed_backup [07:59:13] RESOLVED: [3x] KubernetesCalicoDown: wikikube-worker1050.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:00:05] andre and brennen: gettimeofday() says it's time for MediaWiki train - Utc-0+Utc-7 Version. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T0800) [08:00:06] o/ [08:01:29] !log installing mbedtls security updates [08:01:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:03:13] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:03:32] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1052.eqiad.wmnet [08:03:33] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1052.eqiad.wmnet [08:03:47] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1053.eqiad.wmnet [08:04:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1050.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:04:22] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1053.eqiad.wmnet [08:04:25] RESOLVED: [3x] KubernetesCalicoDown: wikikube-worker1051.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:04:32] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:05:57] (03CR) 10Filippo Giunchedi: "See inline, LGTM otherwise" [puppet] - 10https://gerrit.wikimedia.org/r/1352915 (owner: 10Muehlenhoff) [08:09:04] alright, time to deploy the train [08:10:12] (03PS1) 10TrainBranchBot: group2 to 1.47.0-wmf.23 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353009 (https://phabricator.wikimedia.org/T438219) [08:10:14] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by aklapper@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353009 (https://phabricator.wikimedia.org/T438219) (owner: 10TrainBranchBot) [08:10:59] (03Merged) 10jenkins-bot: group2 to 1.47.0-wmf.23 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353009 (https://phabricator.wikimedia.org/T438219) (owner: 10TrainBranchBot) [08:12:51] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1053.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:13:12] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1053.eqiad.wmnet [08:13:13] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1053.eqiad.wmnet [08:13:26] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1054.eqiad.wmnet [08:13:28] (03PS3) 10Muehlenhoff: Remove the labtest-roots group [puppet] - 10https://gerrit.wikimedia.org/r/1352915 [08:13:41] (03CR) 10Hashar: "The `mask`? I am not sure I follow but it does not really matter. The systemd unit does spin up zuul-server (which forks a second process)" [puppet] - 10https://gerrit.wikimedia.org/r/1343955 (https://phabricator.wikimedia.org/T384939) (owner: 10Hnowlan) [08:13:49] (03CR) 10Hashar: [C:03+1] zuul: migrate icinga checks to prometheus [puppet] - 10https://gerrit.wikimedia.org/r/1343955 (https://phabricator.wikimedia.org/T384939) (owner: 10Hnowlan) [08:14:00] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1054.eqiad.wmnet [08:14:08] (03CR) 10Muehlenhoff: Remove the labtest-roots group (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1352915 (owner: 10Muehlenhoff) [08:14:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1051.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:14:19] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:14:25] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker1052.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:14:35] (03CR) 10Vgutierrez: acmechief: Add additional SNIs for ldap-ro-test.[eqiad|codfw] (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1352988 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [08:14:46] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.7 point update - https://phabricator.wikimedia.org/T437715#12410020 (10MoritzMuehlenhoff) [08:20:28] !log aklapper@deploy1003 rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.23 refs T438219 [08:20:31] T438219: 1.47.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T438219 [08:21:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [08:22:38] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1054.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:22:51] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1054.eqiad.wmnet [08:23:02] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1054.eqiad.wmnet [08:23:16] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1055.eqiad.wmnet [08:23:52] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1055.eqiad.wmnet [08:24:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1053.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:24:14] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:24:25] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker1053.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:26:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [08:26:14] (03CR) 10Muehlenhoff: acmechief: Add additional SNIs for ldap-ro-test.[eqiad|codfw] (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1352988 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [08:32:02] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1055.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:32:21] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1055.eqiad.wmnet [08:32:22] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1055.eqiad.wmnet [08:32:28] 10ops-codfw, 06SRE, 06DC-Ops: Enter CyrusOne's Rack Location in Netbox's "Facility ID" Field - https://phabricator.wikimedia.org/T440071#12410112 (10ayounsi) @RobH Please use netbox-next for tests, to avoid the risk of having invalid/stale date in the prod instance until we come to a full agreement and prope... [08:32:36] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1056.eqiad.wmnet [08:32:42] (03PS1) 10Btullis: dumps: Enable brownout for dumps v1 history from 2026-10-26 [puppet] - 10https://gerrit.wikimedia.org/r/1353019 (https://phabricator.wikimedia.org/T437461) [08:33:07] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1056.eqiad.wmnet [08:33:11] (03CR) 10Atsuko: [C:03+1] dpe: add Hummingbird alerts [alerts] - 10https://gerrit.wikimedia.org/r/1352991 (https://phabricator.wikimedia.org/T440547) (owner: 10Dpogorzelski) [08:33:26] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:33:49] (03PS5) 10Elukey: Disable pki vhost on port 8443 [puppet] - 10https://gerrit.wikimedia.org/r/1346616 (https://phabricator.wikimedia.org/T436809) [08:35:29] PROBLEM - Host wikikube-worker2158 is DOWN: PING CRITICAL - Packet loss = 100% [08:37:07] (03CR) 10Elukey: [C:03+2] Disable pki vhost on port 8443 [puppet] - 10https://gerrit.wikimedia.org/r/1346616 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [08:37:55] (03PS7) 10Elukey: profile::pki::multirootca: remove support for port 8443 [puppet] - 10https://gerrit.wikimedia.org/r/1346617 (https://phabricator.wikimedia.org/T436809) [08:39:51] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1353019 (https://phabricator.wikimedia.org/T437461) (owner: 10Btullis) [08:42:17] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1056.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:42:30] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1056.eqiad.wmnet [08:42:39] !log installing python3.13 security updates [08:42:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:42:48] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2158.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:42:53] RECOVERY - Host wikikube-worker2158 is UP: PING OK - Packet loss = 0%, RTA = 30.40 ms [08:43:00] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2158.codfw.wmnet [08:44:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1055.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:44:59] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1056.eqiad.wmnet [08:45:13] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1057.eqiad.wmnet [08:45:13] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2158.codfw.wmnet [08:45:27] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2159.codfw.wmnet [08:46:02] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2159.codfw.wmnet [08:46:21] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:46:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1057.eqiad.wmnet [08:46:25] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox: sync [08:46:28] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [08:46:31] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:46:32] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [08:46:39] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [08:46:42] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 17 hosts with reason: Primary switchover x1 T436922 [08:46:45] T436922: Switchover x1 master (db2191 -> db2249) - https://phabricator.wikimedia.org/T436922 [08:46:54] !log marostegui@cumin1004 dbctl commit (dc=all): 'Set db2249 with weight 0 T436922', diff saved to https://phabricator.wikimedia.org/P96625 and previous config saved to /var/cache/conftool/dbconfig/20261008-084653-marostegui.json [08:47:27] RECOVERY - Check unit status of kube-reload-certs on deploy1003 is OK: OK: Status of the systemd unit kube-reload-certs https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [08:48:04] (03PS2) 10Gerrit maintenance bot: mariadb: Promote db2249 to x1 master [puppet] - 10https://gerrit.wikimedia.org/r/1334811 (https://phabricator.wikimedia.org/T436922) [08:48:32] (03CR) 10Vgutierrez: [C:03+1] acmechief: Add additional SNIs for ldap-ro-test.[eqiad|codfw] (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1352988 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [08:49:13] RESOLVED: [3x] KubernetesCalicoDown: wikikube-worker1055.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:49:23] (03CR) 10Marostegui: [C:03+2] mariadb: Promote db2249 to x1 master [puppet] - 10https://gerrit.wikimedia.org/r/1334811 (https://phabricator.wikimedia.org/T436922) (owner: 10Gerrit maintenance bot) [08:50:03] !log Starting x1 codfw failover from db2191 to db2249 - T436922 [08:50:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:50:20] (03PS1) 10Ladsgroup-claude: swift: Load originals in 64MB chunks [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1353028 (https://phabricator.wikimedia.org/T420223) [08:50:50] !log marostegui@cumin1004 dbctl commit (dc=all): 'Promote db2249 to x1 primary T436922', diff saved to https://phabricator.wikimedia.org/P96626 and previous config saved to /var/cache/conftool/dbconfig/20261008-085050-marostegui.json [08:51:32] 06SRE, 10Pywikibot, 06Traffic, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error on Wikidata - https://phabricator.wikimedia.org/T421642#12410199 (10Xqt) @ArthurPSmith: With Pywikibot 11.8, I've disabled maxlag for meta queries, paraminfo and help, and enabled a separate maxlag threshold of 30... [08:52:39] !log marostegui@cumin1004 dbctl commit (dc=all): 'Depool db2191 T436922', diff saved to https://phabricator.wikimedia.org/P96627 and previous config saved to /var/cache/conftool/dbconfig/20261008-085238-marostegui.json [08:52:42] T436922: Switchover x1 master (db2191 -> db2249) - https://phabricator.wikimedia.org/T436922 [08:52:43] (03PS1) 10Elukey: kube-reload-certs: add "helm repo update" [puppet] - 10https://gerrit.wikimedia.org/r/1353030 [08:53:16] !log marostegui@cumin1004 START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover [08:53:43] !log marostegui@cumin1004 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2191: Repooling after switchover [08:53:52] !log marostegui@cumin1004 START - Cookbook sre.mysql.pool pool db2191: Repooling after switchover [08:54:21] PROBLEM - pt-heartbeat-wikimedia process on db2191 is CRITICAL: PROCS CRITICAL: 0 processes with args pt-heartbeat-wikimedia https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23pt-heartbeat [08:54:43] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 2 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [08:56:09] (03CR) 10Atsuko: [C:03+1] dumps: Enable brownout for dumps v1 history from 2026-10-26 [puppet] - 10https://gerrit.wikimedia.org/r/1353019 (https://phabricator.wikimedia.org/T437461) (owner: 10Btullis) [08:56:28] (03CR) 10Btullis: [C:03+2] dumps: Enable brownout for dumps v1 history from 2026-10-26 [puppet] - 10https://gerrit.wikimedia.org/r/1353019 (https://phabricator.wikimedia.org/T437461) (owner: 10Btullis) [08:56:54] (03CR) 10Arnaudb: "I forgot to ask for a review on this change, once it is merged I'll try again to do a `git ls-remote` on Envoy's port. Once this is confir" [puppet] - 10https://gerrit.wikimedia.org/r/1328152 (https://phabricator.wikimedia.org/T420184) (owner: 10Arnaudb) [08:57:18] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2159.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [08:57:31] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2159.codfw.wmnet [08:57:43] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [08:59:00] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2159.codfw.wmnet [08:59:13] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2160.codfw.wmnet [08:59:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1056.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:59:17] !log installing Wireshark security updates [08:59:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:59:48] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2160.codfw.wmnet [09:00:08] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:04:13] RESOLVED: [3x] KubernetesCalicoDown: wikikube-worker1056.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [09:04:14] (03CR) 10Jgiannelos: [C:03+1] mobile-apps: add more mathjax csp rules [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352074 (https://phabricator.wikimedia.org/T431693) (owner: 10Isabelle Hurbain-Palatin) [09:04:17] (03CR) 10Muehlenhoff: [C:03+2] acmechief: Add additional SNIs for ldap-ro-test.[eqiad|codfw] [puppet] - 10https://gerrit.wikimedia.org/r/1352988 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [09:07:03] PROBLEM - Host wikikube-worker1057 is DOWN: PING CRITICAL - Packet loss = 100% [09:09:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [09:09:25] FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:10:34] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2160.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:10:53] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2160.codfw.wmnet [09:10:55] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2160.codfw.wmnet [09:10:59] (03PS2) 10Elukey: kube-reload-certs: add "helm repo update" [puppet] - 10https://gerrit.wikimedia.org/r/1353030 [09:11:09] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2161.codfw.wmnet [09:11:42] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2161.codfw.wmnet [09:12:02] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:12:56] (03PS3) 10Elukey: kube-reload-certs: add "helm repo update" [puppet] - 10https://gerrit.wikimedia.org/r/1353030 [09:14:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [09:15:49] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 2 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [09:16:09] (03CR) 10Elukey: [C:03+1] Remove role/profiles for rootless Cumin [puppet] - 10https://gerrit.wikimedia.org/r/1352905 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [09:16:52] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1057.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:18:50] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12410260 (10cmooney) [09:21:07] (03CR) 10Dpogorzelski: [C:03+2] dpe: add Hummingbird alerts [alerts] - 10https://gerrit.wikimedia.org/r/1352991 (https://phabricator.wikimedia.org/T440547) (owner: 10Dpogorzelski) [09:22:56] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2161.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:23:08] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2161.codfw.wmnet [09:23:09] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2161.codfw.wmnet [09:23:23] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2162.codfw.wmnet [09:23:42] (03Merged) 10jenkins-bot: dpe: add Hummingbird alerts [alerts] - 10https://gerrit.wikimedia.org/r/1352991 (https://phabricator.wikimedia.org/T440547) (owner: 10Dpogorzelski) [09:23:58] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2162.codfw.wmnet [09:24:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [09:24:18] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:25:45] 10SRE-swift-storage, 10MediaWiki-File-management, 06MediaWiki-Media-Platform-Team: Unknown error while undeleting a file on Wikimedia Commons - https://phabricator.wikimedia.org/T437114#12410296 (10JTweed-WMF) p:05Triage→03Medium [09:26:11] (03PS1) 10Fabfur: cace:haproxy: deprecating mathoid as a service endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1353052 (https://phabricator.wikimedia.org/T431372) [09:26:27] (03PS2) 10Fabfur: cache:haproxy: deprecating mathoid as a service endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1353052 (https://phabricator.wikimedia.org/T431372) [09:27:02] (03PS1) 10JMeybohm: k8s: Label nodes by link speed [puppet] - 10https://gerrit.wikimedia.org/r/1353053 (https://phabricator.wikimedia.org/T420223) [09:27:51] (03CR) 10JMeybohm: [C:03+1] kube-reload-certs: add "helm repo update" [puppet] - 10https://gerrit.wikimedia.org/r/1353030 (owner: 10Elukey) [09:28:19] (03PS3) 10Fabfur: cache:haproxy: deprecating mathoid as a service endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1353052 (https://phabricator.wikimedia.org/T431372) [09:28:19] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [09:28:43] (03PS1) 10Santiago Faci: Test Kitchen UI: Deploy v2.1.0 release to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353054 (https://phabricator.wikimedia.org/T438519) [09:29:07] (03PS1) 10Gkyziridis: ml-services: Copy paste the isvc from the llm ns into experimental [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353055 (https://phabricator.wikimedia.org/T440307) [09:29:30] !log begin draining traffic from cr1-codfw in advance of JunOS upgrade T417871 [09:29:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:29:33] T417871: codfw: upgrade routers (2026) - https://phabricator.wikimedia.org/T417871 [09:29:55] (03PS2) 10JMeybohm: k8s: Label nodes by link speed [puppet] - 10https://gerrit.wikimedia.org/r/1353053 (https://phabricator.wikimedia.org/T420223) [09:30:50] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1353053 (https://phabricator.wikimedia.org/T420223) (owner: 10JMeybohm) [09:34:39] !log marostegui@cumin1004 START - Cookbook sre.mysql.pool pool db1262: Pooling back after cloning a host [09:34:41] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2162.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:35:00] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2162.codfw.wmnet [09:35:19] oblivian@cumin1004 enable_virtualization (PID 3622236) is awaiting input [09:37:02] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2162.codfw.wmnet [09:37:16] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2163.codfw.wmnet [09:37:50] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2163.codfw.wmnet [09:37:59] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:38:33] (03CR) 10Blake: [C:03+1] k8s: Label nodes by link speed [puppet] - 10https://gerrit.wikimedia.org/r/1353053 (https://phabricator.wikimedia.org/T420223) (owner: 10JMeybohm) [09:38:59] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2191: Repooling after switchover [09:39:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [09:41:24] (03CR) 10Clément Goubert: [C:03+2] git::clone: add optional sparse-checkout support [puppet] - 10https://gerrit.wikimedia.org/r/1343101 (owner: 10Aghirelli) [09:41:34] (03CR) 10Clément Goubert: [C:03+2] eventschemas: add api-platform-schemas repository [puppet] - 10https://gerrit.wikimedia.org/r/1343092 (https://phabricator.wikimedia.org/T438208) (owner: 10Aghirelli) [09:42:42] (03PS2) 10Aghirelli: eventschemas: sparse-checkout the mediawiki repo to its mediawiki/ dir [puppet] - 10https://gerrit.wikimedia.org/r/1343109 [09:42:50] (03CR) 10Clément Goubert: [C:03+2] eventschemas: sparse-checkout the mediawiki repo to its mediawiki/ dir [puppet] - 10https://gerrit.wikimedia.org/r/1343109 (owner: 10Aghirelli) [09:42:51] (03CR) 10Elukey: [C:03+2] kube-reload-certs: add "helm repo update" [puppet] - 10https://gerrit.wikimedia.org/r/1353030 (owner: 10Elukey) [09:43:13] elukey: merging your change yeah? [09:43:19] yeahhhh [09:44:15] (03PS3) 10JMeybohm: k8s: Label nodes by link speed [puppet] - 10https://gerrit.wikimedia.org/r/1353053 (https://phabricator.wikimedia.org/T420223) [09:44:36] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1353053 (https://phabricator.wikimedia.org/T420223) (owner: 10JMeybohm) [09:44:47] elukey: done [09:46:28] (03CR) 10Blake: [C:03+1] k8s: Label nodes by link speed [puppet] - 10https://gerrit.wikimedia.org/r/1353053 (https://phabricator.wikimedia.org/T420223) (owner: 10JMeybohm) [09:48:52] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2163.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:49:11] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2163.codfw.wmnet [09:49:25] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [09:49:35] (03CR) 10Elukey: [C:03+1] "Looks really nice! Maybe link-speed-mbps=kvm could end up being confusing at first, but I don't have any other suggestion." [puppet] - 10https://gerrit.wikimedia.org/r/1353053 (https://phabricator.wikimedia.org/T420223) (owner: 10JMeybohm) [09:51:13] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2163.codfw.wmnet [09:51:14] (03CR) 10JMeybohm: [C:03+2] k8s: Label nodes by link speed [puppet] - 10https://gerrit.wikimedia.org/r/1353053 (https://phabricator.wikimedia.org/T420223) (owner: 10JMeybohm) [09:51:26] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2164.codfw.wmnet [09:52:01] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2164.codfw.wmnet [09:52:06] !log cmooney@cumin1004 DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade [09:52:10] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [09:52:53] (03CR) 10Gkyziridis: [C:03+2] ml-services: Copy paste the isvc from the llm ns into experimental [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353055 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [09:52:58] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 16 hosts with reason: cr1-codfw upgrade [09:54:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [09:54:21] (03Merged) 10jenkins-bot: ml-services: Copy paste the isvc from the llm ns into experimental [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353055 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [09:59:17] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T1000) [10:00:22] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:02:33] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2164.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:02:52] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2164.codfw.wmnet [10:02:54] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2164.codfw.wmnet [10:03:07] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2165.codfw.wmnet [10:03:41] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2165.codfw.wmnet [10:04:01] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:04:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:05:22] RESOLVED: CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:06:18] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug) [10:08:21] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug) [10:14:25] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2165.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:14:43] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2165.codfw.wmnet [10:14:45] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2165.codfw.wmnet [10:14:58] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2166.codfw.wmnet [10:15:33] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2166.codfw.wmnet [10:15:53] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:19:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:19:43] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Pooling back after cloning a host [10:21:35] (03CR) 10Hnowlan: [C:03+1] eventgate-logging-external: replace kafka-logging200[1-3] with 200[6-8] [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351204 (https://phabricator.wikimedia.org/T432444) (owner: 10Tiziano Fogli) [10:26:47] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2166.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:26:58] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2166.codfw.wmnet [10:28:16] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2166.codfw.wmnet [10:28:19] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [10:28:30] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2167.codfw.wmnet [10:29:04] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2167.codfw.wmnet [10:29:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:29:23] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:29:25] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:30:14] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply [10:31:04] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply [10:36:31] !log cr1-codfw switch to backup routing engine T426197 [10:36:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:36:35] T426197: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197 [10:39:47] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2167.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:40:16] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2167.codfw.wmnet [10:40:18] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2167.codfw.wmnet [10:40:31] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2168.codfw.wmnet [10:41:06] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2168.codfw.wmnet [10:41:25] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:44:00] (03PS1) 10Jdrewniak: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353100 (https://phabricator.wikimedia.org/T128546) [10:44:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:46:07] (03PS1) 10Hnowlan: sre.dns.admin: (de)pool metamonitoring along with DNS [cookbooks] - 10https://gerrit.wikimedia.org/r/1353101 (https://phabricator.wikimedia.org/T436076) [10:51:34] (03PS1) 10Muehlenhoff: acmechief: Remove static LDAP replica names [puppet] - 10https://gerrit.wikimedia.org/r/1353104 (https://phabricator.wikimedia.org/T439730) [10:51:38] (03CR) 10Vgutierrez: "as suggested on IRC by @dkertesz@wikimedia.org 410 could be a better match for this use case than a 403" [puppet] - 10https://gerrit.wikimedia.org/r/1353052 (https://phabricator.wikimedia.org/T431372) (owner: 10Fabfur) [10:51:49] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2168.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:52:08] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2168.codfw.wmnet [10:52:13] (03PS1) 10AikoChou: ml-services: temporarily move GPU services off ml-serve1015 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353105 (https://phabricator.wikimedia.org/T431553) [10:54:10] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2168.codfw.wmnet [10:54:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:54:24] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2169.codfw.wmnet [10:54:27] (03CR) 10Ozge: [C:03+1] ml-services: temporarily move GPU services off ml-serve1015 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353105 (https://phabricator.wikimedia.org/T431553) (owner: 10AikoChou) [10:54:40] (03CR) 10Vgutierrez: cache:haproxy: deprecating mathoid as a service endpoint (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1353052 (https://phabricator.wikimedia.org/T431372) (owner: 10Fabfur) [10:54:58] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2169.codfw.wmnet [10:55:18] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [10:55:39] (03CR) 10Marostegui: [C:03+1] team-data-persistence: Try setting alert only in eqiad [alerts] - 10https://gerrit.wikimedia.org/r/1352562 (owner: 10Federico Ceratto) [10:57:52] (03PS4) 10Fabfur: cace:haproxy: deprecating mathoid as a service endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1353052 (https://phabricator.wikimedia.org/T431372) [10:57:52] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D{wikikube-worker[1006-1007,1015-1016,1021,1034-1057].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [10:58:04] (03CR) 10Fabfur: cace:haproxy: deprecating mathoid as a service endpoint (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1353052 (https://phabricator.wikimedia.org/T431372) (owner: 10Fabfur) [10:59:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:03:25] (03CR) 10Clément Goubert: [C:03+1] mcrouter: update to 1.3.7 (copy) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352571 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [11:03:35] (03PS5) 10Fabfur: cache:haproxy: deprecating mathoid as a service endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1353052 (https://phabricator.wikimedia.org/T431372) [11:03:35] (03CR) 10Clément Goubert: [C:03+1] mcrouter: Add native sidecar capabilities to the mcrouter exporter [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352572 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [11:04:20] (03CR) 10Blake: [C:03+2] mcrouter: update to 1.3.7 (copy) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352571 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [11:04:24] (03PS1) 10Marostegui: cumin1004.cnf: Add x4 backup [puppet] - 10https://gerrit.wikimedia.org/r/1353106 (https://phabricator.wikimedia.org/T437279) [11:05:45] (03Merged) 10jenkins-bot: mcrouter: update to 1.3.7 (copy) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352571 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [11:06:05] (03PS2) 10Blake: mcrouter: Add native sidecar capabilities to the mcrouter exporter [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352572 (https://phabricator.wikimedia.org/T417800) [11:06:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2169.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:06:33] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2169.codfw.wmnet [11:06:35] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2169.codfw.wmnet [11:06:48] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2170.codfw.wmnet [11:06:53] (03CR) 10AikoChou: [C:03+2] ml-services: temporarily move GPU services off ml-serve1015 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353105 (https://phabricator.wikimedia.org/T431553) (owner: 10AikoChou) [11:07:02] (03CR) 10Slyngshede: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1353104 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [11:07:22] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2170.codfw.wmnet [11:07:42] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:07:58] (03CR) 10Marostegui: [C:03+2] cumin1004.cnf: Add x4 backup [puppet] - 10https://gerrit.wikimedia.org/r/1353106 (https://phabricator.wikimedia.org/T437279) (owner: 10Marostegui) [11:08:06] (03CR) 10Blake: [C:03+2] mcrouter: Add native sidecar capabilities to the mcrouter exporter [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352572 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [11:08:17] (03Merged) 10jenkins-bot: ml-services: temporarily move GPU services off ml-serve1015 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353105 (https://phabricator.wikimedia.org/T431553) (owner: 10AikoChou) [11:09:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:09:27] (03Merged) 10jenkins-bot: mcrouter: Add native sidecar capabilities to the mcrouter exporter [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352572 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [11:10:13] !log cr1-codfw switch back to master routing engine T426197 [11:10:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:10:16] T426197: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197 [11:10:47] (03PS6) 10Fabfur: cache:haproxy: deprecating mathoid as a service endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1353052 (https://phabricator.wikimedia.org/T431372) [11:11:34] !log aikochou@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [11:14:44] (03PS1) 10Blake: mediawiki: update mcrouter chart to 1.3.7 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353111 [11:15:08] (03PS2) 10Blake: mediawiki: update mcrouter template to 1.3.7 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353111 [11:17:23] (03PS1) 10AikoChou: ml-services: deploy llm-qwen3-14b-24gb on ml-serve1015 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353115 (https://phabricator.wikimedia.org/T431553) [11:18:06] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2170.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:18:18] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2170.codfw.wmnet [11:18:20] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2170.codfw.wmnet [11:18:34] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2171.codfw.wmnet [11:19:07] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2171.codfw.wmnet [11:19:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:19:25] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:19:27] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:22:49] (03PS1) 10Cathal Mooney: cr1-codfw: enable rib-sharding [homer/public] - 10https://gerrit.wikimedia.org/r/1353117 (https://phabricator.wikimedia.org/T320264) [11:24:51] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2171.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:26:21] (03CR) 10Cathal Mooney: [C:03+2] cr1-codfw: enable rib-sharding [homer/public] - 10https://gerrit.wikimedia.org/r/1353117 (https://phabricator.wikimedia.org/T320264) (owner: 10Cathal Mooney) [11:27:17] (03Merged) 10jenkins-bot: cr1-codfw: enable rib-sharding [homer/public] - 10https://gerrit.wikimedia.org/r/1353117 (https://phabricator.wikimedia.org/T320264) (owner: 10Cathal Mooney) [11:30:57] (03CR) 10Clément Goubert: [C:03+1] mediawiki: update mcrouter template to 1.3.7 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353111 (owner: 10Blake) [11:32:03] (03PS1) 10Aghirelli: eventschemas: fix sparse-checkout cone for mediawiki repo [puppet] - 10https://gerrit.wikimedia.org/r/1353121 (https://phabricator.wikimedia.org/T438208) [11:32:35] (03CR) 10Blake: [C:03+2] mediawiki: update mcrouter template to 1.3.7 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353111 (owner: 10Blake) [11:32:46] 10ops-eqiad, 06DC-Ops: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1057.eqiad.wmnet - https://phabricator.wikimedia.org/T440573 (10Joe) 03NEW [11:34:39] (03Merged) 10jenkins-bot: mediawiki: update mcrouter template to 1.3.7 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353111 (owner: 10Blake) [11:35:18] (03CR) 10Clément Goubert: [C:03+2] eventschemas: fix sparse-checkout cone for mediawiki repo [puppet] - 10https://gerrit.wikimedia.org/r/1353121 (https://phabricator.wikimedia.org/T438208) (owner: 10Aghirelli) [11:36:10] !log blake@deploy1003 Started scap sync-world: update mcrouter exporter template for T417800 [11:36:13] T417800: Implement proper sidecar container support in mediawiki pods - https://phabricator.wikimedia.org/T417800 [11:36:41] ACKNOWLEDGEMENT - SSH on wikikube-worker1057 is CRITICAL: CRITICAL - Socket timeout after 10 seconds Giuseppe Lavagetto see T440573 https://wikitech.wikimedia.org/wiki/SSH/monitoring [11:36:41] ACKNOWLEDGEMENT - Host wikikube-worker1057 is DOWN: PING CRITICAL - Packet loss = 100% Giuseppe Lavagetto see T440573 [11:36:52] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2171.codfw.wmnet [11:36:53] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2171.codfw.wmnet [11:37:07] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2172.codfw.wmnet [11:37:19] (03CR) 10Federico Ceratto: [C:03+2] team-data-persistence: Try setting alert only in eqiad [alerts] - 10https://gerrit.wikimedia.org/r/1352562 (owner: 10Federico Ceratto) [11:37:43] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2172.codfw.wmnet [11:38:03] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:39:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1057.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:39:15] (03Merged) 10jenkins-bot: team-data-persistence: Try setting alert only in eqiad [alerts] - 10https://gerrit.wikimedia.org/r/1352562 (owner: 10Federico Ceratto) [11:39:46] !log blake@deploy1003 Finished scap sync-world: update mcrouter exporter template for T417800 (duration: 04m 15s) [11:40:26] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-experimental: apply [11:41:27] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply [11:41:28] !log blake@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-experimental: apply [11:42:21] !log blake@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply [11:43:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2172.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:44:19] (03PS1) 10WMDE-Fisch: Switch to CommunityConfiguration to enable better autonames in Cite VE [extensions/Cite] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353131 (https://phabricator.wikimedia.org/T436255) [11:44:52] !log oblivian@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wikikube-worker1057.eqiad.wmnet with reason: HW failure [11:45:01] 10ops-eqiad, 06DC-Ops: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1057.eqiad.wmnet - https://phabricator.wikimedia.org/T440573#12410929 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=0415e15e-2f75-4151-a3ac-d87e40a6400c) set by oblivian@cumin1004 for 14 days, 0:00:00... [11:45:03] oblivian@cumin1004 enable_virtualization (PID 3684262) is awaiting input [11:45:16] (03CR) 10Gkyziridis: [C:03+1] ml-services: deploy llm-qwen3-14b-24gb on ml-serve1015 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353115 (https://phabricator.wikimedia.org/T431553) (owner: 10AikoChou) [11:46:12] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, October 12 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc" [extensions/Cite] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353131 (https://phabricator.wikimedia.org/T436255) (owner: 10WMDE-Fisch) [11:49:13] RESOLVED: KubernetesCalicoDown: wikikube-worker2171.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2171.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:53:49] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [11:54:12] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2172.codfw.wmnet [11:54:13] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2172.codfw.wmnet [11:54:27] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2173.codfw.wmnet [11:55:01] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2173.codfw.wmnet [11:55:21] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [11:58:23] (03Abandoned) 10Anzx: knwikisource: lift IP cap for workshop on 09-October-2026 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352919 (https://phabricator.wikimedia.org/T440546) (owner: 10Anzx) [11:58:49] !log oblivian@cumin1004 START - Cookbook sre.k8s.enable_virtualization rolling provision on D{wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [11:58:52] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1064.eqiad.wmnet [11:59:24] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1064.eqiad.wmnet [11:59:45] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:00:04] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T1200) [12:00:09] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:00:18] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:00:33] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2173.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:02:12] (03PS1) 10Btullis: ceph: Point the eqiad cluster hosts at thirdparty/ceph-squid [puppet] - 10https://gerrit.wikimedia.org/r/1353141 (https://phabricator.wikimedia.org/T428445) [12:02:14] (03PS1) 10Btullis: ceph: Match the eqiad admin host client to the cluster version [puppet] - 10https://gerrit.wikimedia.org/r/1353142 (https://phabricator.wikimedia.org/T428445) [12:04:38] i'm going to deploy mobileapps [12:04:56] (03CR) 10Isabelle Hurbain-Palatin: [C:03+2] mobile-apps: add more mathjax csp rules [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352074 (https://phabricator.wikimedia.org/T431693) (owner: 10Isabelle Hurbain-Palatin) [12:06:09] Hi folks, I'd need to run a query in production (x1.wikishared) to restore 3 accidentally-deleted records. The query is in https://phabricator.wikimedia.org/T440557#12411029. May I go ahead now? [12:06:36] (03Merged) 10jenkins-bot: mobile-apps: add more mathjax csp rules [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352074 (https://phabricator.wikimedia.org/T431693) (owner: 10Isabelle Hurbain-Palatin) [12:07:07] 10SRE-SLO, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): page_change SLO windows - https://phabricator.wikimedia.org/T438054#12411038 (10APizzata-WMF) Hey @RLazarus, thank you for the message and have fun on your PTO! The monthly data points are unfortunately a limitation of our systems. The sou... [12:07:18] (03CR) 10Michael Große: [C:03+1] "Thanks!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352515 (https://phabricator.wikimedia.org/T439738) (owner: 10EMcFarland) [12:08:35] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1064.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:08:53] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1064.eqiad.wmnet [12:08:57] !log ihurbain@deploy1003 helmfile [staging] START helmfile.d/services/mobileapps: apply [12:09:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1064.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:09:29] !log ihurbain@deploy1003 helmfile [staging] DONE helmfile.d/services/mobileapps: apply [12:09:52] !log ihurbain@deploy1003 helmfile [eqiad] START helmfile.d/services/mobileapps: apply [12:10:34] !log ihurbain@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [12:10:35] (03PS1) 10Zabe: Use Hadoop for UnusedTemplates on commonswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353155 (https://phabricator.wikimedia.org/T422443) [12:10:46] !log ihurbain@deploy1003 helmfile [codfw] START helmfile.d/services/mobileapps: apply [12:11:33] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1064.eqiad.wmnet [12:11:46] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1065.eqiad.wmnet [12:11:56] !log ihurbain@deploy1003 helmfile [codfw] DONE helmfile.d/services/mobileapps: apply [12:12:19] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1065.eqiad.wmnet [12:12:34] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2173.codfw.wmnet [12:12:36] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2173.codfw.wmnet [12:12:37] (03PS2) 10Zabe: Use Hadoop for UnusedTemplates on commonswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353155 (https://phabricator.wikimedia.org/T422443) [12:12:38] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:12:49] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2174.codfw.wmnet [12:13:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2174.codfw.wmnet [12:13:43] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:14:13] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker1064.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:15:50] jouncebot: nowandnext [12:15:50] For the next 0 hour(s) and 44 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T1200) [12:15:50] In 0 hour(s) and 44 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T1300) [12:16:32] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12411083 (10Jclark-ctr) Finished cabling for the A/B upgrade. Everything is complete except for **1 cross-connect between the cages** Cables are pre ran thou... [12:18:30] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12411092 (10Jclark-ctr) Additionally cabled lsw1-C1 Adjusting rack positions so scs-c1 is above lsw and mgmt | Network | Device | Port | Con... [12:18:32] (03PS1) 10Btullis: ceph: Make thirdparty/ceph-squid the default for the DPE Ceph hosts [puppet] - 10https://gerrit.wikimedia.org/r/1353163 (https://phabricator.wikimedia.org/T428445) [12:18:56] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2174.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:19:00] (03PS2) 10Abijeet Patro: ULS: Define LanguageSelectorV2SupportedSkins config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353160 (https://phabricator.wikimedia.org/T439707) [12:19:06] (03CR) 10Zabe: [C:03+2] Use Hadoop for UnusedTemplates on commonswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353155 (https://phabricator.wikimedia.org/T422443) (owner: 10Zabe) [12:19:26] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, October 12 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353160 (https://phabricator.wikimedia.org/T439707) (owner: 10Abijeet Patro) [12:19:56] (03Merged) 10jenkins-bot: Use Hadoop for UnusedTemplates on commonswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353155 (https://phabricator.wikimedia.org/T422443) (owner: 10Zabe) [12:20:47] !log zabe@deploy1003 Started scap sync-world: Backport for [[gerrit:1353155|Use Hadoop for UnusedTemplates on commonswiki (T422443)]] [12:20:50] T422443: Move SpecialUnusedTemplates computation to Hadoop - https://phabricator.wikimedia.org/T422443 [12:21:28] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1065.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:21:52] (03CR) 10Muehlenhoff: [C:03+2] acmechief: Remove static LDAP replica names [puppet] - 10https://gerrit.wikimedia.org/r/1353104 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [12:21:55] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1065.eqiad.wmnet [12:21:57] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1065.eqiad.wmnet [12:22:10] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1066.eqiad.wmnet [12:22:42] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1066.eqiad.wmnet [12:23:01] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:23:50] (03PS2) 10Btullis: ceph: Make thirdparty/ceph-squid the default for the DPE Ceph hosts [puppet] - 10https://gerrit.wikimedia.org/r/1353163 (https://phabricator.wikimedia.org/T428445) [12:23:59] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1353163 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [12:24:43] !log zabe@deploy1003 zabe: Backport for [[gerrit:1353155|Use Hadoop for UnusedTemplates on commonswiki (T422443)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:25:12] !log zabe@deploy1003 zabe: Continuing with deployment [12:27:41] (03PS1) 10Gkyziridis: ml-services: Remove qwen3-14b from experimental ns. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353167 (https://phabricator.wikimedia.org/T440307) [12:29:54] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2174.codfw.wmnet [12:29:55] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2174.codfw.wmnet [12:30:09] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2175.codfw.wmnet [12:30:34] !log zabe@deploy1003 Finished scap sync-world: Backport for [[gerrit:1353155|Use Hadoop for UnusedTemplates on commonswiki (T422443)]] (duration: 09m 47s) [12:30:37] T422443: Move SpecialUnusedTemplates computation to Hadoop - https://phabricator.wikimedia.org/T422443 [12:30:43] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2175.codfw.wmnet [12:31:03] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:32:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1066.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:32:42] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1066.eqiad.wmnet [12:34:13] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1065.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:34:29] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1066.eqiad.wmnet [12:34:34] (03PS2) 10Gkyziridis: ml-services: Remove qwen3-14b from experimental ns and add llm-qwen36-27b. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353167 (https://phabricator.wikimedia.org/T440307) [12:34:42] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1067.eqiad.wmnet [12:35:17] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1067.eqiad.wmnet [12:35:36] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:36:29] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2175.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:36:40] (03CR) 10Gkyziridis: [C:03+2] ml-services: Remove qwen3-14b from experimental ns and add llm-qwen36-27b. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353167 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [12:37:21] (03CR) 10Muehlenhoff: [C:03+2] New Cumin alias memcached-all [puppet] - 10https://gerrit.wikimedia.org/r/1352155 (owner: 10Muehlenhoff) [12:38:06] (03Merged) 10jenkins-bot: ml-services: Remove qwen3-14b from experimental ns and add llm-qwen36-27b. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353167 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [12:38:16] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.7 point update - https://phabricator.wikimedia.org/T437715#12411158 (10MoritzMuehlenhoff) [12:38:40] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 08 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352515 (https://phabricator.wikimedia.org/T439738) (owner: 10EMcFarland) [12:39:13] RESOLVED: [3x] KubernetesCalicoDown: wikikube-worker1065.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:39:44] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [12:41:21] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad A/B switch cabling documentation - https://phabricator.wikimedia.org/T418018#12411175 (10Jclark-ctr) Finished cabling for the A/B upgrade. Everything is complete except for **1 cross-connect between the cages** Cables are pre ran... [12:42:33] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad A/B switch cabling documentation - https://phabricator.wikimedia.org/T418018#12411177 (10Jclark-ctr) A1 was not cabled I have updated netbox for cableid's for all the uplinks except the spine links since they are not setup. [12:43:29] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1067.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:43:50] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1067.eqiad.wmnet [12:43:51] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1067.eqiad.wmnet [12:44:05] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1068.eqiad.wmnet [12:44:06] (03PS1) 10Zabe: Simplify Hadoop configuration for query pages [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353172 (https://phabricator.wikimedia.org/T309738) [12:44:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1066.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:44:25] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker1066.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:44:37] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1068.eqiad.wmnet [12:44:55] (03PS1) 10WMDE-Fisch: Enable better autonames CommunityConfiguration on group0 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353173 (https://phabricator.wikimedia.org/T440357) [12:44:56] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:46:05] (03CR) 10Dbrant: [C:03+2] mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352606 (owner: 10PipelineBot) [12:46:23] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, October 12 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353173 (https://phabricator.wikimedia.org/T440357) (owner: 10WMDE-Fisch) [12:46:57] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, October 12 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [extensions/Cite] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353131 (https://phabricator.wikimedia.org/T436255) (owner: 10WMDE-Fisch) [12:47:34] (03Merged) 10jenkins-bot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352606 (owner: 10PipelineBot) [12:47:48] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db1901: test [12:48:29] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2175.codfw.wmnet [12:48:31] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2175.codfw.wmnet [12:48:45] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2176.codfw.wmnet [12:49:12] !log zabe@deploy1003:~$ mwscript recountCategories.php commonswiki --mode all # T440023 T439902 [12:49:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:49:17] T440023: Category file counts display negative page counts - https://phabricator.wikimedia.org/T440023 [12:49:17] T439902: Wikimedia\Rdbms\DBQueryError: Error 1205: Lock wait timeout exceeded; try restarting transactionFunction: MediaWiki\JobQueue\Jobs\CategoryCountUpdateJob::updateCategoryCountsQuery: UPDATE `category` SET cat_pages = cat_pages - https://phabricator.wikimedia.org/T439902 [12:49:56] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2176.codfw.wmnet [12:50:16] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:50:49] !log dbrant@deploy1003 helmfile [staging] START helmfile.d/services/mobileapps: apply [12:51:09] !log dbrant@deploy1003 helmfile [staging] DONE helmfile.d/services/mobileapps: apply [12:52:28] !log dbrant@deploy1003 helmfile [eqiad] START helmfile.d/services/mobileapps: apply [12:52:45] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1068.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:53:04] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1068.eqiad.wmnet [12:53:05] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1068.eqiad.wmnet [12:53:12] !log dbrant@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [12:53:19] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1069.eqiad.wmnet [12:53:24] !log dbrant@deploy1003 helmfile [codfw] START helmfile.d/services/mobileapps: apply [12:53:50] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1069.eqiad.wmnet [12:53:58] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:54:10] !log dbrant@deploy1003 helmfile [codfw] DONE helmfile.d/services/mobileapps: apply [12:54:13] !log fceratto@cumin1004 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test [12:54:22] !log fceratto@cumin1004 START - Cookbook sre.mysql.depool depool db1901: test [12:54:25] !log fceratto@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1901: test [12:54:30] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db1901: test [12:54:39] !log fceratto@cumin1004 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test [12:54:50] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db1901: test [12:54:58] !log fceratto@cumin1004 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test [12:55:05] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db1901: test [12:58:40] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2176.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [12:58:59] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2176.codfw.wmnet [12:59:01] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2176.codfw.wmnet [12:59:13] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1067.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:59:15] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2177.codfw.wmnet [12:59:25] RESOLVED: [3x] KubernetesCalicoDown: wikikube-worker1067.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:59:49] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2177.codfw.wmnet [13:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: Your horoscope predicts another UTC afternoon backport window deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T1300). [13:00:05] dhardy, jan_drewniak, and eileen-m: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:09] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:00:18] I’m in a meeting but can maybe deploy later [13:00:21] o/ [13:00:59] * eileen-m__16 waves [13:01:04] jan_drewniak: are you able to self-deploy? I'm working with dhardy getting them set up to do their deployment [13:01:22] eileen-m__16: will you also be able to self-deploy or do you need a deployer? :) [13:01:44] (03PS2) 10WMDE-Fisch: Enable sub-references on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353180 (https://phabricator.wikimedia.org/T440243) [13:02:19] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1069.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:02:24] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, October 14 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353180 (https://phabricator.wikimedia.org/T440243) (owner: 10WMDE-Fisch) [13:02:25] I would appreciate a deployer, please. [13:02:47] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1069.eqiad.wmnet [13:02:47] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1069.eqiad.wmnet [13:03:01] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1070.eqiad.wmnet [13:03:34] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1070.eqiad.wmnet [13:03:53] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:04:25] RESOLVED: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:04:31] !log fceratto@cumin1004 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1901: test [13:04:35] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db1901: test [13:04:39] !log fceratto@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1901: test [13:04:42] (03CR) 10Elukey: [C:03+2] profile::pki::multirootca: remove support for port 8443 [puppet] - 10https://gerrit.wikimedia.org/r/1346617 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [13:05:22] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2177.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:05:25] (03PS5) 10Elukey: Remove mTLS support for pki clients [puppet] - 10https://gerrit.wikimedia.org/r/1346630 (https://phabricator.wikimedia.org/T436809) [13:05:37] eileen-m__16: ack, will just see if jan_drewniak wants to go first else I'll do yours in a sec :) [13:07:42] TheresNoTime thank you! [13:08:05] Going to do yours now [13:08:29] (03CR) 10TrainBranchBot: [C:03+2] "Approved by samtar@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352515 (https://phabricator.wikimedia.org/T439738) (owner: 10EMcFarland) [13:09:23] (03Merged) 10jenkins-bot: Home Experience: Feature flag to enable Home [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352515 (https://phabricator.wikimedia.org/T439738) (owner: 10EMcFarland) [13:09:33] !log samtar@deploy1003 Started scap sync-world: Backport for [[gerrit:1352515|Home Experience: Feature flag to enable Home (T439738)]] [13:09:39] T439738: Create config option that enables the new Home experience and set it to true on test2wiki - https://phabricator.wikimedia.org/T439738 [13:10:50] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, October 14 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347373 (https://phabricator.wikimedia.org/T437054) (owner: 10Sadiya.mohammed13) [13:11:42] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1070.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:12:00] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1070.eqiad.wmnet [13:12:01] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1070.eqiad.wmnet [13:12:15] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1071.eqiad.wmnet [13:12:29] Hey folks sorry I'm late to the deployment window, I'll do mine last. [13:12:41] (ack) [13:12:48] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1071.eqiad.wmnet [13:13:07] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:13:43] !log samtar@deploy1003 emc-wmf, samtar: Backport for [[gerrit:1352515|Home Experience: Feature flag to enable Home (T439738)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:15:31] I see "FAIL: 190 requests sent to mwdebug-next.discovery.wmnet. 1 request with failed assertions.", retrying.. [13:16:10] Retry worked - eileen-m__16, ready for testing once you're able to (fingers crossed the SSH works) [13:16:45] I ran ` curl "https://config-master.wikimedia.org/known_hosts" -o ~/.ssh/known_hosts` but still can't ssh in.  Not sure what I am doing wrong.  I'm investigating. [13:17:33] eileen-m__16: worth asking the question here ref "No ED25519 host key is known for deployment.eqiad.wmnet and you have requested strict checking." [13:17:50] Lucas_WMDE: I know you're sorta around but in a meeting, any idea? ^ [13:17:58] oh I’m here now [13:18:00] * Lucas_WMDE looks [13:18:18] * TheresNoTime thought that was just a case of pulling the fingerprints.. [13:18:43] Question below: [13:18:44] I am currently trying to ssh into a deployment server in order to run mwscript shell.php --wiki test2wiki and confirm that the backport deployment for the config option that enables home experience worked. When I run ssh deployment.eqiad.wmnet I get the errorNo ED25519 host key is known for deployment.eqiad.wmnet and you have requested strict [13:18:44] checking. [13:18:45] Host key verification failed. [13:19:45] eileen-m__16: what access mode does `ls -l ~/.ssh/known_hosts` show? [13:19:50] I also tried to run `ssh bast3007.wikimedia.org ` and got eileenmcfarland@bast3007.wikimedia.org: Permission denied (publickey). [13:19:53] for me it’s user rw, group/other nothing [13:20:05] I think that curl command might have resulted in a default world-readable file, and IIRC ssh can be very picky about that [13:20:21] that’s the best idea I have right now an yway [13:20:36] It shows -rw-r--r--@ [13:20:57] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1071.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:21:03] I would try `chmod 600`, shouldn’t hurt anyway [13:21:08] will do, thanks [13:21:09] though the permission error sounds like something else :P [13:21:10] * :/ [13:21:25] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1071.eqiad.wmnet [13:21:26] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1071.eqiad.wmnet [13:21:40] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1072.eqiad.wmnet [13:21:55] I did chmod 600  ~/.ssh/known_hosts and still get the same error [13:22:02] eileen-m__16: if that doesn't work, I can run any commands you need [13:22:07] ssh deployment.eqiad.wmnet [13:22:08] No ED25519 host key is known for deployment.eqiad.wmnet and you have requested strict checking. [13:22:08] Host key verification failed. [13:22:12] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1072.eqiad.wmnet [13:22:13] (but still worth tracking down why this is happening.) [13:22:32] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:22:34] eileen-m__16: “eileenmcfarland@bast3007.wikimedia.org: Permission denied (publickey” – shouldn’t that user name be emc-wmf, judging by puppet? [13:22:34] TheresNoTime thanks, I can send you the commands I need to run to confirm the config option is in place [13:22:53] Lucas_WMDE ohhhhhhh [13:23:35] (I can run the commands if the username change doesn't solve it :P) [13:23:49] oblivian@cumin1004 enable_virtualization (PID 3622331) is awaiting input [13:23:50] Lucas_WMDE I checked my config and it does say `User emc-wmf`..... [13:24:18] hmm [13:24:29] maybe it’s in the wrong Host/Match block or something? [13:25:31] (03PS6) 10Elukey: sre.hosts.reimage: refactor DHCP configs as cookbook calls [cookbooks] - 10https://gerrit.wikimedia.org/r/1348725 [13:26:02] though at this point I would say, let’s let TheresNoTime run the commands to unblock the rest of the window, and debug SSH afterwards ^^ [13:26:27] yes, i am getting the commands now and will post them shortly [13:26:37] (DM would be best) [13:26:40] (03PS1) 10Cathal Mooney: Add group for new PKI host VIPs and allow on cloud-hosts ACL [homer/public] - 10https://gerrit.wikimedia.org/r/1353197 (https://phabricator.wikimedia.org/T436809) [13:26:48] ok [13:27:01] * TheresNoTime dreams of the day when ssh_config is Easy(tm) [13:27:03] FIRING: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:27:06] (03PS7) 10Elukey: sre.hosts.reimage: refactor DHCP configs as cookbook calls [cookbooks] - 10https://gerrit.wikimedia.org/r/1348725 [13:27:15] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (live debug) [13:27:33] (03PS1) 10JMeybohm: thumbor: Add priorityClassName value [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353198 (https://phabricator.wikimedia.org/T420223) [13:27:35] (03PS1) 10JMeybohm: wikikube: Add high-priority priority class [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353199 (https://phabricator.wikimedia.org/T420223) [13:27:37] (03PS1) 10JMeybohm: thumbor: Schedule on nodes with 10G NIC only [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353200 (https://phabricator.wikimedia.org/T420223) [13:28:13] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (live debug) [13:28:32] TheresNoTime: our ssh_config would be way easier™ if we exposed all hosts directly to the internet without a bastion and also allowed anyone to log in as root *ducks* [13:29:06] lemme just create a phab task for that /j [13:29:18] +1 let's do it :P [13:29:24] (03CR) 10Filippo Giunchedi: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1352915 (owner: 10Muehlenhoff) [13:29:31] Hypothesis: if we [paste], deployers will be able to access the servers more easily [13:30:51] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1072.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:31:11] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1072.eqiad.wmnet [13:31:12] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1072.eqiad.wmnet [13:31:25] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1073.eqiad.wmnet [13:31:33] (03CR) 10Filippo Giunchedi: "Discussed with Luca, FWIW LGTM" [cookbooks] - 10https://gerrit.wikimedia.org/r/1348725 (owner: 10Elukey) [13:31:45] (03CR) 10Filippo Giunchedi: "Discussed with Luca, FWIW LGTM" [cookbooks] - 10https://gerrit.wikimedia.org/r/1347466 (owner: 10Elukey) [13:31:58] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1073.eqiad.wmnet [13:32:03] RESOLVED: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:32:18] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:32:23] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1057.eqiad.wmnet - https://phabricator.wikimedia.org/T440573#12411427 (10Jclark-ctr) Warranty expired Ended on August 11, 2026 Upgrading firmware on Bios, Idrac , Backplane , HBA355i , SSD SKhynix firmware [13:33:38] !log fceratto@cumin1004 START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet [13:33:38] !log fceratto@cumin1004 END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet [13:33:51] (03CR) 10Blake: [C:03+1] thumbor: Add priorityClassName value [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353198 (https://phabricator.wikimedia.org/T420223) (owner: 10JMeybohm) [13:34:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:34:44] * TheresNoTime is still testing the current deploy [13:35:11] (03CR) 10Blake: [C:03+1] wikikube: Add high-priority priority class (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353199 (https://phabricator.wikimedia.org/T420223) (owner: 10JMeybohm) [13:35:22] (03PS1) 10Jelto: deployment_server::deploy_service::service_catalog: add aux-k8s [puppet] - 10https://gerrit.wikimedia.org/r/1352555 (https://phabricator.wikimedia.org/T412941) [13:36:15] (03CR) 10Blake: thumbor: Schedule on nodes with 10G NIC only (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353200 (https://phabricator.wikimedia.org/T420223) (owner: 10JMeybohm) [13:38:18] !log fceratto@cumin1004 START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet [13:38:18] !log fceratto@cumin1004 END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2901.codfw.wmnet onto db2902.codfw.wmnet [13:39:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:41:08] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1073.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:41:36] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1073.eqiad.wmnet [13:41:47] (03CR) 10Atsuko: [C:03+1] ceph: Make thirdparty/ceph-squid the default for the DPE Ceph hosts [puppet] - 10https://gerrit.wikimedia.org/r/1353163 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [13:42:13] (03PS1) 10Marostegui: dbprov1005.cnf.erb: Add eqiad x4 [puppet] - 10https://gerrit.wikimedia.org/r/1353209 (https://phabricator.wikimedia.org/T437279) [13:42:26] !log samtar@deploy1003 emc-wmf, samtar: Continuing with deployment [13:43:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1073.eqiad.wmnet [13:43:30] (03PS8) 10Elukey: sre.hosts.reimage: refactor DHCP configs as cookbook calls [cookbooks] - 10https://gerrit.wikimedia.org/r/1348725 [13:43:37] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1074.eqiad.wmnet [13:43:39] oblivian@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [13:44:12] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1074.eqiad.wmnet [13:44:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1072.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:44:25] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker1072.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:44:32] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:45:28] (03CR) 10Marostegui: [C:03+2] dbprov1005.cnf.erb: Add eqiad x4 [puppet] - 10https://gerrit.wikimedia.org/r/1353209 (https://phabricator.wikimedia.org/T437279) (owner: 10Marostegui) [13:46:55] RECOVERY - Host wikikube-worker1057 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [13:47:16] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [13:47:49] !log samtar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1352515|Home Experience: Feature flag to enable Home (T439738)]] (duration: 38m 16s) [13:47:52] T439738: Create config option that enables the new Home experience and set it to true on test2wiki - https://phabricator.wikimedia.org/T439738 [13:48:05] eileen-m__16: live on prod now :) [13:48:10] jan_drewniak: all yours [13:48:21] TheresNoTime Yay! [13:48:23] ok thanks! [13:49:08] 07sre-alert-triage, 06Infrastructure-Foundations, 10Kafka-Infrastructure, 06ServiceOps, and 2 others: Alert in need of triage: PKICertificateExpiry (instance pki2002:9100) - https://phabricator.wikimedia.org/T438703#12411522 (10MLechvien-WMF) @LSobanski @elukey is anyone from I/F taking a look at this aler... [13:52:06] (03CR) 10Xcollazo: "Could you please merge @brouberol@wikimedia.org?" [puppet] - 10https://gerrit.wikimedia.org/r/1352580 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [13:52:28] (03PS9) 10Elukey: sre.hosts.reimage: refactor DHCP configs as cookbook calls [cookbooks] - 10https://gerrit.wikimedia.org/r/1348725 [13:53:23] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1074.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:53:42] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1074.eqiad.wmnet [13:54:06] (03CR) 10Xcollazo: dumps: Keep pages-meta-history only in the 3 newest runs on replicas (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1346629 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [13:54:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1073.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:56:12] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1074.eqiad.wmnet [13:56:25] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1075.eqiad.wmnet [13:56:57] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1075.eqiad.wmnet [13:57:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:57:16] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:57:51] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdrewniak@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353100 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [13:58:37] (03Merged) 10jenkins-bot: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353100 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [13:58:51] !log jdrewniak@deploy1003 Started scap sync-world: Backport for [[gerrit:1353100|Bumping portals to master (T128546)]] [13:58:54] T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546 [13:58:56] (03CR) 10Elukey: "The last version of the patch is still to be tested, I am removing all parameters like --mac --opt82 --uuid, it is the dhcp cookbook that " [cookbooks] - 10https://gerrit.wikimedia.org/r/1348725 (owner: 10Elukey) [13:59:13] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker1073.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:00:39] (03CR) 10Brouberol: [C:03+2] dumps: Remove the generator settings from the web cleanups [puppet] - 10https://gerrit.wikimedia.org/r/1352580 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [14:02:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:03:02] !log jdrewniak@deploy1003 jdrewniak: Backport for [[gerrit:1353100|Bumping portals to master (T128546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:03:03] !log fceratto@cumin1004 START - Cookbook sre.mysql.clone of db2901.codfw.wmnet onto db2902.codfw.wmnet [14:03:05] !log fceratto@cumin1004 START - Cookbook sre.mysql.depool depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004 [14:03:09] !log fceratto@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2901: Depool db2901.codfw.wmnet to then clone it to db2902.codfw.wmnet - fceratto@cumin1004 [14:03:10] !log fceratto@cumin1004 START - Cookbook sre.mysql.depool depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004 [14:03:15] !log fceratto@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2902: Depool db2902.codfw.wmnet - fceratto@cumin1004 [14:03:15] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: "Comm Error: backplane 0" for wikikube-worker1057.eqiad.wmnet - https://phabricator.wikimedia.org/T440573#12411609 (10Jclark-ctr) 05Open→03Resolved a:03Jclark-ctr @Joe Server is back up right now [14:05:35] (03PS1) 10Ozge: ml-services: Increase llm namespace cpu quota from 200 to 250 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353230 (https://phabricator.wikimedia.org/T440575) [14:06:06] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1075.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:07:36] (03CR) 10CDanis: [C:03+1] wmnet: move codfw, eqsin, ulsfo etcd (RO) client SRV records to codfw [dns] - 10https://gerrit.wikimedia.org/r/1344391 (owner: 10Scott French) [14:08:38] (03CR) 10JMeybohm: [C:03+1] swift: Load originals in 64MB chunks [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1353028 (https://phabricator.wikimedia.org/T420223) (owner: 10Ladsgroup-claude) [14:09:29] (03PS4) 10Xcollazo: dumps: Keep pages-meta-history only in the 3 newest runs on replicas [puppet] - 10https://gerrit.wikimedia.org/r/1346629 (https://phabricator.wikimedia.org/T437463) [14:09:30] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db2901: Pool db2901.codfw.wmnet in after cloning [14:09:53] (03CR) 10CI reject: [V:04-1] dumps: Keep pages-meta-history only in the 3 newest runs on replicas [puppet] - 10https://gerrit.wikimedia.org/r/1346629 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [14:13:04] (03CR) 10Gkyziridis: [C:03+1] ml-services: Increase llm namespace cpu quota from 200 to 250 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353230 (https://phabricator.wikimedia.org/T440575) (owner: 10Ozge) [14:13:23] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.7 point update - https://phabricator.wikimedia.org/T437715#12411684 (10MoritzMuehlenhoff) [14:14:35] (03PS5) 10Xcollazo: dumps: Keep pages-meta-history only in the 3 newest runs on replicas [puppet] - 10https://gerrit.wikimedia.org/r/1346629 (https://phabricator.wikimedia.org/T437463) [14:14:59] (03CR) 10Brouberol: [C:03+1] ml-services: Increase llm namespace cpu quota from 200 to 250 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353230 (https://phabricator.wikimedia.org/T440575) (owner: 10Ozge) [14:15:01] (03CR) 10Brouberol: [C:03+2] ml-services: Increase llm namespace cpu quota from 200 to 250 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353230 (https://phabricator.wikimedia.org/T440575) (owner: 10Ozge) [14:17:05] !log fceratto@cumin1004 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db2901: Pool db2901.codfw.wmnet in after cloning [14:17:06] (03CR) 10Xcollazo: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1346629 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [14:17:58] !log brouberol@deploy1003 helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [14:18:36] !log brouberol@deploy1003 helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [14:18:52] (03CR) 10Muehlenhoff: [C:03+2] Remove the labtest-roots group [puppet] - 10https://gerrit.wikimedia.org/r/1352915 (owner: 10Muehlenhoff) [14:19:13] !log brouberol@deploy1003 helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [14:19:51] (03PS2) 10Muehlenhoff: Add Cumin alias for wdqs::alternatives [puppet] - 10https://gerrit.wikimedia.org/r/1346444 [14:20:12] !log brouberol@deploy1003 helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [14:20:49] (03Abandoned) 10Muehlenhoff: Revert "Add explicit replica hostnames to LDAP cert SNIs" [puppet] - 10https://gerrit.wikimedia.org/r/1352478 (owner: 10Muehlenhoff) [14:21:01] (03CR) 10Muehlenhoff: [C:03+2] Remove role/profiles for rootless Cumin [puppet] - 10https://gerrit.wikimedia.org/r/1352905 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [14:21:20] (03PS1) 10Cathal Mooney: lsw1-c1-eqiad: add definitions needed for new switch [puppet] - 10https://gerrit.wikimedia.org/r/1353239 (https://phabricator.wikimedia.org/T403031) [14:21:36] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1075.eqiad.wmnet [14:21:38] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1075.eqiad.wmnet [14:21:51] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1076.eqiad.wmnet [14:22:24] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1076.eqiad.wmnet [14:22:30] (03CR) 10Xcollazo: "Ran PPC with latest changes. LGTM." [puppet] - 10https://gerrit.wikimedia.org/r/1346629 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [14:22:44] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:23:12] (03PS1) 10Muehlenhoff: pontoon: Remove onsolete spec map [puppet] - 10https://gerrit.wikimedia.org/r/1353240 (https://phabricator.wikimedia.org/T439522) [14:23:56] !log installing perl security updates [14:23:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:25:06] 10SRE-Access-Requests: Setting up Yubikey for foks - https://phabricator.wikimedia.org/T440598 (10jrbs) 03NEW [14:26:03] (03CR) 10JHathaway: [C:03+1] profile::puppetserver::volatile: Allow use in Cloud VPS [puppet] - 10https://gerrit.wikimedia.org/r/1352635 (owner: 10Ahmon Dancy) [14:27:23] (03CR) 10Pmiazga: [C:03+1] Add URLs for MobileAppRedirect Special Page [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339561 (https://phabricator.wikimedia.org/T434930) (owner: 10Milazg) [14:29:10] !log jdrewniak@deploy1003 jdrewniak: Continuing with deployment [14:29:29] (03CR) 10Herron: [C:03+2] sre.opensearch.roll-restart-reboot: include checklist items (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1334048 (https://phabricator.wikimedia.org/T435265) (owner: 10Herron) [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T1430) [14:32:00] (03CR) 10Filippo Giunchedi: [C:03+1] pontoon: Remove onsolete spec map [puppet] - 10https://gerrit.wikimedia.org/r/1353240 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [14:32:05] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1076.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:32:05] (03Merged) 10jenkins-bot: sre.opensearch.roll-restart-reboot: include checklist items [cookbooks] - 10https://gerrit.wikimedia.org/r/1334048 (https://phabricator.wikimedia.org/T435265) (owner: 10Herron) [14:32:10] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1076.eqiad.wmnet [14:32:12] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1076.eqiad.wmnet [14:32:25] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1077.eqiad.wmnet [14:32:29] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [14:32:58] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1077.eqiad.wmnet [14:33:05] (03CR) 10Muehlenhoff: [C:03+2] pontoon: Remove onsolete spec map [puppet] - 10https://gerrit.wikimedia.org/r/1353240 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [14:33:17] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:35:00] !log jdrewniak@deploy1003 Finished scap sync-world: Backport for [[gerrit:1353100|Bumping portals to master (T128546)]] (duration: 36m 09s) [14:35:03] T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546 [14:35:20] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" [14:38:23] cmooney@cumin1004 netbox (PID 3801623) is awaiting input [14:38:32] (03PS1) 10Cathal Mooney: Add INCLUDE statements for IPv6 reverse DNS new eqiad c1 vlans [dns] - 10https://gerrit.wikimedia.org/r/1353248 (https://phabricator.wikimedia.org/T403031) [14:38:56] (03CR) 10CI reject: [V:04-1] Add INCLUDE statements for IPv6 reverse DNS new eqiad c1 vlans [dns] - 10https://gerrit.wikimedia.org/r/1353248 (https://phabricator.wikimedia.org/T403031) (owner: 10Cathal Mooney) [14:39:12] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" [14:39:12] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:39:14] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [14:41:02] (03PS1) 10Blake: bgp: add port number to var-device. [alerts] - 10https://gerrit.wikimedia.org/r/1353249 (https://phabricator.wikimedia.org/T439944) [14:41:37] (03CR) 10Elukey: [C:03+1] "<3" [homer/public] - 10https://gerrit.wikimedia.org/r/1353197 (https://phabricator.wikimedia.org/T436809) (owner: 10Cathal Mooney) [14:42:05] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 08 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339561 (https://phabricator.wikimedia.org/T434930) (owner: 10Milazg) [14:42:09] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1077.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:42:29] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1077.eqiad.wmnet [14:43:07] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" [14:43:10] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" [14:43:11] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:43:20] (03CR) 10Cathal Mooney: [C:03+2] Add group for new PKI host VIPs and allow on cloud-hosts ACL [homer/public] - 10https://gerrit.wikimedia.org/r/1353197 (https://phabricator.wikimedia.org/T436809) (owner: 10Cathal Mooney) [14:44:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1076.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:44:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1077.eqiad.wmnet [14:44:29] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1078.eqiad.wmnet [14:44:35] (03Merged) 10jenkins-bot: Add group for new PKI host VIPs and allow on cloud-hosts ACL [homer/public] - 10https://gerrit.wikimedia.org/r/1353197 (https://phabricator.wikimedia.org/T436809) (owner: 10Cathal Mooney) [14:44:44] (03PS2) 10Cathal Mooney: Add INCLUDE statements for IPv6 reverse DNS new eqiad c1 vlans [dns] - 10https://gerrit.wikimedia.org/r/1353248 (https://phabricator.wikimedia.org/T403031) [14:45:01] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1078.eqiad.wmnet [14:45:09] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:45:40] (03CR) 10CI reject: [V:04-1] Add INCLUDE statements for IPv6 reverse DNS new eqiad c1 vlans [dns] - 10https://gerrit.wikimedia.org/r/1353248 (https://phabricator.wikimedia.org/T403031) (owner: 10Cathal Mooney) [14:48:38] !log installing pyasn1 security updates [14:48:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:48:42] !log cgoubert@dns2004 START - running authdns-update [14:49:13] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker1076.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:50:57] !log cgoubert@dns2004 END - running authdns-update [14:51:05] !log cmooney@dns3003 START - running authdns-update [14:52:53] (03PS2) 10AikoChou: ml-services: deploy llm-qwen3-14b-24gb on ml-serve1015 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353115 (https://phabricator.wikimedia.org/T431553) [14:53:03] (03CR) 10AikoChou: [C:03+2] ml-services: deploy llm-qwen3-14b-24gb on ml-serve1015 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353115 (https://phabricator.wikimedia.org/T431553) (owner: 10AikoChou) [14:53:26] (03CR) 10Clément Goubert: [C:03+1] revision-feature-counts: add deployment chart files (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [14:53:50] (03Merged) 10jenkins-bot: ml-services: deploy llm-qwen3-14b-24gb on ml-serve1015 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353115 (https://phabricator.wikimedia.org/T431553) (owner: 10AikoChou) [14:54:10] !log cmooney@dns3003 END - running authdns-update [14:54:28] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:54:57] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1078.eqiad.wmnet [14:56:12] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1078.eqiad.wmnet [14:56:26] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1079.eqiad.wmnet [14:57:02] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1079.eqiad.wmnet [14:57:21] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [14:58:06] !log aikochou@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [14:59:17] (03PS1) 10Matthias Mullie: Temporary: don't catch exception [extensions/ReaderExperiments] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353257 [14:59:37] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 08 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/ReaderExperiments] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353257 (owner: 10Matthias Mullie) [14:59:48] !log btullis@cumin1004 START - Cookbook sre.druid.reboot-workers for Druid public cluster: Reboot Druid nodes [15:00:56] (03CR) 10Clare Ming: [C:03+2] Test Kitchen UI: Deploy v2.1.0 release to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353054 (https://phabricator.wikimedia.org/T438519) (owner: 10Santiago Faci) [15:01:39] (03CR) 10Btullis: [C:03+2] ceph: Make thirdparty/ceph-squid the default for the DPE Ceph hosts [puppet] - 10https://gerrit.wikimedia.org/r/1353163 (https://phabricator.wikimedia.org/T428445) (owner: 10Btullis) [15:02:24] (03Merged) 10jenkins-bot: Test Kitchen UI: Deploy v2.1.0 release to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353054 (https://phabricator.wikimedia.org/T438519) (owner: 10Santiago Faci) [15:06:13] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1079.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:06:31] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1079.eqiad.wmnet [15:07:57] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1079.eqiad.wmnet [15:08:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [15:08:11] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1080.eqiad.wmnet [15:08:44] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1080.eqiad.wmnet [15:09:03] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:11:19] (03CR) 10Muehlenhoff: [C:03+2] ferm: Absent the NRPE check when migrating from ferm to nftables [puppet] - 10https://gerrit.wikimedia.org/r/1283620 (owner: 10Muehlenhoff) [15:12:23] jouncebot: nowandnext [15:12:23] No deployments scheduled for the next 0 hour(s) and 47 minute(s) [15:12:24] In 0 hour(s) and 47 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T1600) [15:13:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [15:15:08] (03CR) 10BCornwall: [C:03+1] Remove the url-downloader legacy CNAMEs [dns] - 10https://gerrit.wikimedia.org/r/1351840 (https://phabricator.wikimedia.org/T439713) (owner: 10Muehlenhoff) [15:17:35] !log fceratto@cumin1004 START - Cookbook sre.mysql.pool pool db2902: Pool db2902.codfw.wmnet in after cloning [15:17:55] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1080.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:17:58] 10ops-codfw, 06SRE, 06DC-Ops: Enter CyrusOne's Rack Location in Netbox's "Facility ID" Field - https://phabricator.wikimedia.org/T440071#12412226 (10Jhancock.wm) I didn't think about it that way. So we should use long form. The numbers we received in the incident report indicate that the racks are identified... [15:18:14] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1080.eqiad.wmnet [15:19:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1079.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:20:53] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1080.eqiad.wmnet [15:21:07] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1081.eqiad.wmnet [15:21:41] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1081.eqiad.wmnet [15:22:01] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:22:33] (03PS12) 10Clément Goubert: revision-feature-counts: add deployment chart files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) [15:22:33] (03PS1) 10Clément Goubert: lac-lambdas: Add base generic files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353264 (https://phabricator.wikimedia.org/T439879) [15:23:58] (03CR) 10CI reject: [V:04-1] lac-lambdas: Add base generic files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353264 (https://phabricator.wikimedia.org/T439879) (owner: 10Clément Goubert) [15:24:13] RESOLVED: [2x] KubernetesCalicoDown: wikikube-worker1079.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:24:26] (03CR) 10Clément Goubert: "Split the original CR into this one that implements the generic boilerplate for the namespace, and I64a00b9ca7afd that has only the additi" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353264 (https://phabricator.wikimedia.org/T439879) (owner: 10Clément Goubert) [15:25:23] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [15:25:44] (03CR) 10Clément Goubert: "CI is kind of supposed to fail since there's no actual releases." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353264 (https://phabricator.wikimedia.org/T439879) (owner: 10Clément Goubert) [15:27:22] (03PS4) 10Muehlenhoff: Rename puppetmaster::puppetdb to puppetdb [puppet] - 10https://gerrit.wikimedia.org/r/1331470 [15:28:19] (03PS1) 10Ejegg: Update CentralNotice to 658422ff5fed886 [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353269 [15:28:32] (03PS2) 10Clément Goubert: lac-lambdas: Add base generic files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353264 (https://phabricator.wikimedia.org/T439879) [15:28:32] (03PS13) 10Clément Goubert: revision-feature-counts: add deployment chart files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) [15:29:46] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" [15:29:53] (03CR) 10CI reject: [V:04-1] lac-lambdas: Add base generic files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353264 (https://phabricator.wikimedia.org/T439879) (owner: 10Clément Goubert) [15:29:54] (03PS2) 10Cathal Mooney: lsw1-c1-eqiad: add definitions needed for new switch [puppet] - 10https://gerrit.wikimedia.org/r/1353239 (https://phabricator.wikimedia.org/T403031) [15:32:25] (03CR) 10AKhatun: "My patch was un-done it seems." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [15:32:49] cmooney@cumin1004 netbox (PID 3865718) is awaiting input [15:32:56] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 08 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353269 (owner: 10Ejegg) [15:33:51] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1331470 (owner: 10Muehlenhoff) [15:34:37] (03PS14) 10Clément Goubert: revision-feature-counts: add deployment chart files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) [15:35:56] (03CR) 10Clément Goubert: "*silently pushes a new patch*" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [15:36:02] !log elukey@cumin1004 START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS trixie [15:37:23] (03CR) 10Muehlenhoff: [C:03+2] Rename puppetmaster::puppetdb to puppetdb [puppet] - 10https://gerrit.wikimedia.org/r/1331470 (owner: 10Muehlenhoff) [15:38:01] (03PS15) 10Clément Goubert: revision-feature-counts: add deployment chart files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) [15:44:13] FIRING: KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1081.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:44:52] 10ops-eqiad, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be110[1-4] - https://phabricator.wikimedia.org/T438368#12412484 (10Jhancock.wm) [15:45:50] 10ops-codfw, 06SRE, 10SRE-swift-storage, 06Data-Persistence, 06DC-Ops: Q1:rack/setup/install ms-be2[099-104] - https://phabricator.wikimedia.org/T438370#12412491 (10Jhancock.wm) [15:46:08] PROBLEM - Host wikikube-worker1081 is DOWN: PING CRITICAL - Packet loss = 100% [15:47:51] 10ops-codfw, 06SRE, 06DC-Ops: codfw: move public baremetal servers to per rack vlan - https://phabricator.wikimedia.org/T428060#12412525 (10Jhancock.wm) putting this on the back burner for a moment. still plan on doing it, just a little busy. [15:47:53] !log elukey@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage [15:49:22] 06SRE-OnFire, 14SRE-Sprint-Week-Sustainability-March2023, 10Scap, 06serviceops-deprecated, 07Sustainability (Incident Followup): Add etcdmirror status check to scap - https://phabricator.wikimedia.org/T317403#12412545 (10dancy) Please provide information on how to determine if etcdmirror is down, or conf... [15:49:58] (03CR) 10Elukey: thumbor: Schedule on nodes with 10G NIC only (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353200 (https://phabricator.wikimedia.org/T420223) (owner: 10JMeybohm) [15:52:22] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1081.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [15:53:01] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2009.codfw.wmnet with reason: host reimage [15:55:00] (03PS1) 10Jforrester: abstractwiki-rust: Re-pin the Rust toolchain to 1.95.0 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1353276 (https://phabricator.wikimedia.org/T440334) [15:55:50] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest2009.codfw.wmnet (live debug) [15:56:45] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2009.codfw.wmnet (live debug) [16:00:05] jhathaway and rzl: How many deployers does it take to do Puppet request window deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T1600). [16:00:05] dancy: A patch you scheduled for Puppet request window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [16:00:10] o/ [16:00:41] o/ [16:00:47] merging both... [16:00:51] (03CR) 10JHathaway: [C:03+2] scap: Remove unused php7_admin_port option [puppet] - 10https://gerrit.wikimedia.org/r/1219164 (https://phabricator.wikimedia.org/T224491) (owner: 10Krinkle) [16:00:51] Thansk! [16:00:55] Thanks! [16:00:56] (03CR) 10JHathaway: [C:03+2] profile::puppetserver::volatile: Allow use in Cloud VPS [puppet] - 10https://gerrit.wikimedia.org/r/1352635 (owner: 10Ahmon Dancy) [16:01:20] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.7 point update - https://phabricator.wikimedia.org/T437715#12412593 (10MoritzMuehlenhoff) [16:01:22] merged [16:02:57] (03CR) 10Elukey: "To keep archives happy - after some thoughts I think this change is safe to be merged. I was concerned by puppet Execs being fired on all " [puppet] - 10https://gerrit.wikimedia.org/r/1346630 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [16:04:49] (03CR) 10Scott French: [C:03+2] wmnet: move codfw, eqsin, ulsfo etcd (RO) client SRV records to codfw [dns] - 10https://gerrit.wikimedia.org/r/1344391 (owner: 10Scott French) [16:04:58] !log swfrench@dns1004 START - running authdns-update [16:07:16] (03PS10) 10Elukey: sre.hosts.reimage: refactor DHCP configs as cookbook calls [cookbooks] - 10https://gerrit.wikimedia.org/r/1348725 [16:07:35] !log swfrench@dns1004 END - running authdns-update [16:10:41] !log elukey@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host sretest2009.codfw.wmnet with OS trixie [16:10:47] (03PS1) 10Ahmon Dancy: profile::openstack::base::puppetserver::enc_client: Read `enc_path` from hiera [puppet] - 10https://gerrit.wikimedia.org/r/1353290 [16:10:49] oblivian@cumin1004 enable_virtualization (PID 3684262) is awaiting input [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:13:58] !log begin rolling restarts of confd in ulsfo, eqsin, codfw to pick up etcd client SRV change [16:13:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:14:53] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new dns entries for IPs on lsw1-c1-eqiad - cmooney@cumin1004" [16:14:53] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [16:15:09] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-c1-eqiad [16:16:21] (03Abandoned) 10Ahmon Dancy: profile::openstack::base::puppetserver::enc_client: Read `enc_path` from hiera [puppet] - 10https://gerrit.wikimedia.org/r/1353290 (owner: 10Ahmon Dancy) [16:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:55] !log cmooney@cumin1004 END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-c1-eqiad [16:18:39] (03CR) 10Clément Goubert: [C:03+1] "Confirmed we're only returning the headers for now, blocking traffic will come later." [puppet] - 10https://gerrit.wikimedia.org/r/1353052 (https://phabricator.wikimedia.org/T431372) (owner: 10Fabfur) [16:19:31] !log dancy@deploy1003 Installing scap version "4.296.0" for 3 host(s) [16:20:10] (03PS5) 10Xcollazo: dumps: Write mirror rsync lists that mix v1 dumps and File Exports [puppet] - 10https://gerrit.wikimedia.org/r/1346764 (https://phabricator.wikimedia.org/T437463) [16:21:25] !log dancy@deploy1003 Installation of scap version "4.296.0" completed for 3 hosts [16:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:25:12] !log ssastry@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [16:25:40] !log ssastry@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [16:25:41] !log ssastry@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [16:26:12] !log ssastry@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [16:27:27] (03CR) 10Ottomata: revision-feature-counts: add deployment chart files (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [16:27:31] (03PS1) 10Cathal Mooney: Add lsw1-c1-eqiad to devices [homer/public] - 10https://gerrit.wikimedia.org/r/1353292 (https://phabricator.wikimedia.org/T403031) [16:28:21] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.enable_virtualization (exit_code=1) rolling provision on D{wikikube-worker[1064-1081,1084-1087,1093-1095].eqiad.wmnet} and (A:wikikube-master-eqiad or A:wikikube-worker-eqiad) [16:33:50] !log oblivian@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet [16:34:13] (03CR) 10Xcollazo: "Correct." [puppet] - 10https://gerrit.wikimedia.org/r/1346764 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [16:35:35] (03CR) 10Ayounsi: [C:03+1] Add lsw1-c1-eqiad to devices [homer/public] - 10https://gerrit.wikimedia.org/r/1353292 (https://phabricator.wikimedia.org/T403031) (owner: 10Cathal Mooney) [16:35:58] !log btullis@cumin1004 END (FAIL) - Cookbook sre.druid.reboot-workers (exit_code=99) for Druid public cluster: Reboot Druid nodes [16:37:51] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1057.eqiad.wmnet [16:38:41] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host wikikube-worker1057.eqiad.wmnet [16:39:18] (03PS2) 10MusikAnimal: CommonSettings: remove redundant CodeMirror config settings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352303 (https://phabricator.wikimedia.org/T440077) [16:40:52] (03CR) 10MusikAnimal: "We need another config change to flip the config settings while the default values in each extension ride the train." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352303 (https://phabricator.wikimedia.org/T440077) (owner: 10MusikAnimal) [16:43:26] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2177.codfw.wmnet [16:43:28] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2177.codfw.wmnet [16:43:34] (03CR) 10MusikAnimal: "> We need another config change to flip the config settings while the default values in each extension ride the train." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352303 (https://phabricator.wikimedia.org/T440077) (owner: 10MusikAnimal) [16:43:41] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2178.codfw.wmnet [16:44:15] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2178.codfw.wmnet [16:44:35] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:45:27] (03PS1) 10Hnowlan: Alert on node_textfile scrape error [alerts] - 10https://gerrit.wikimedia.org/r/1353299 (https://phabricator.wikimedia.org/T312516) [16:49:51] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2178.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [16:50:25] (03CR) 10Hnowlan: [C:03+1] docker-reporter: send alerts to infrastructure foundations [puppet] - 10https://gerrit.wikimedia.org/r/1353251 (https://phabricator.wikimedia.org/T440455) (owner: 10Blake) [16:52:10] (03PS1) 10Atsuko: airflow3: tuning dag-processor [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353296 (https://phabricator.wikimedia.org/T435480) [16:56:43] (03CR) 10Btullis: airflow3: tuning dag-processor (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353296 (https://phabricator.wikimedia.org/T435480) (owner: 10Atsuko) [17:00:04] bd808: I, the Bot under the Fountain, call upon thee, The Deployer, to do Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T1700). [17:00:04] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T1700) [17:00:11] o/ [17:00:37] I'll be deploying a patch to rest-gateway staging during this window, but otherwise have nothing else planned [17:02:58] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2178.codfw.wmnet [17:03:00] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2178.codfw.wmnet [17:03:13] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2179.codfw.wmnet [17:03:31] (03CR) 10Scott French: "Thanks for the review!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346837 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [17:03:32] (03CR) 10Scott French: [C:03+2] rest-gateway: Adopt cluster specifier and XWD routing in staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346837 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [17:03:46] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker2179.codfw.wmnet [17:03:47] (03CR) 10Atsuko: airflow3: tuning dag-processor (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353296 (https://phabricator.wikimedia.org/T435480) (owner: 10Atsuko) [17:04:06] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:04:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:05:12] (03Merged) 10jenkins-bot: rest-gateway: Adopt cluster specifier and XWD routing in staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346837 (https://phabricator.wikimedia.org/T433752) (owner: 10Scott French) [17:06:38] (03CR) 10Btullis: [C:03+1] airflow3: tuning dag-processor (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353296 (https://phabricator.wikimedia.org/T435480) (owner: 10Atsuko) [17:07:17] (03PS1) 10Ahmon Dancy: profile::puppetserver::volatile: Remove GeoIP fetch without private access [puppet] - 10https://gerrit.wikimedia.org/r/1353306 [17:07:34] (03PS1) 10Cathal Mooney: wmf-config: Add new subnet for eqiad rack C1 to reverse-proxy.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1353307 (https://phabricator.wikimedia.org/T403031) [17:09:18] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2179.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [17:09:23] !log swfrench@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [17:10:24] !log swfrench@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [17:12:47] (03PS1) 10Eevans: cassandra: renaming article_feature_counts to revision_feature_counts [puppet] - 10https://gerrit.wikimedia.org/r/1353313 (https://phabricator.wikimedia.org/T438203) [17:14:33] PROBLEM - Host druid1010 is DOWN: PING CRITICAL - Packet loss = 100% [17:20:15] !log oblivian@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2179.codfw.wmnet [17:20:17] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2179.codfw.wmnet [17:20:18] !log oblivian@cumin1004 END (PASS) - Cookbook sre.k8s.enable_virtualization (exit_code=0) rolling provision on D{wikikube-worker[2141-2179].codfw.wmnet} and (A:wikikube-master-codfw or A:wikikube-worker-codfw) [17:23:46] (03PS1) 10KineticPelagic: PP3.4: Update main_app.version to desired tag of container [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353317 (https://phabricator.wikimedia.org/T434137) [17:24:13] (03CR) 10Ottomata: [C:03+1] cassandra: renaming article_feature_counts to revision_feature_counts [puppet] - 10https://gerrit.wikimedia.org/r/1353313 (https://phabricator.wikimedia.org/T438203) (owner: 10Eevans) [17:24:13] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:24:38] (03CR) 10Eevans: [C:03+2] cassandra: renaming article_feature_counts to revision_feature_counts [puppet] - 10https://gerrit.wikimedia.org/r/1353313 (https://phabricator.wikimedia.org/T438203) (owner: 10Eevans) [17:26:58] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631 (10cmooney) 03NEW p:05Triage→03Medium [17:27:20] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12412990 (10cmooney) [17:27:20] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 13Patch-For-Review: Eqiad: Replacement top-of-rack switch for rack C1 - https://phabricator.wikimedia.org/T403031#12412989 (10cmooney) [17:28:55] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: lsw1-c1-eqiad: mgmt0 traffic not working after upgrade to SR-Linux 26.7.3 - https://phabricator.wikimedia.org/T440631#12412998 (10cmooney) [17:31:38] !log fceratto@deploy1003 helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [17:32:34] (03CR) 10Andrew Bogott: [C:03+2] PP3.4: Update main_app.version to desired tag of container [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353317 (https://phabricator.wikimedia.org/T434137) (owner: 10KineticPelagic) [17:34:07] (03Merged) 10jenkins-bot: PP3.4: Update main_app.version to desired tag of container [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353317 (https://phabricator.wikimedia.org/T434137) (owner: 10KineticPelagic) [17:36:30] !log andrew@deploy1003 helmfile [staging] START helmfile.d/services/developer-portal: apply [17:36:46] !log andrew@deploy1003 helmfile [staging] DONE helmfile.d/services/developer-portal: apply [17:41:03] RECOVERY - snapshot of s4 in eqiad on backupmon1001 is OK: Last snapshot for s4 at eqiad (db1265) taken on 2026-10-08 16:46:05 (1138 GiB, +0.0 %) https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Rerun_a_failed_backup [17:45:47] !log andrew@deploy1003 helmfile [codfw] START helmfile.d/services/developer-portal: apply [17:46:07] !log andrew@deploy1003 helmfile [codfw] DONE helmfile.d/services/developer-portal: apply [17:46:55] !log andrew@deploy1003 helmfile [eqiad] START helmfile.d/services/developer-portal: apply [17:47:10] !log andrew@deploy1003 helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply [17:54:23] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 08 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352261 (https://phabricator.wikimedia.org/T437720) (owner: 10Clare Ming) [18:00:04] andre and brennen: It is that lovely time of the day again! You are hereby commanded to deploy MediaWiki train - Utc-0+Utc-7 Version (secondary timeslot). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T1800). [18:00:13] nah, thanks [18:05:03] (03PS5) 10Jdlrobson: CentralNotice: Exclude the preferred sources experiment from banners [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344343 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [18:05:17] (03CR) 10Jdlrobson: [C:03+1] "Can be backported when Depends-On has merged later today." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344343 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [18:07:10] (03PS1) 10Kamila Součková: shellbox: Pick up new images [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353336 (https://phabricator.wikimedia.org/T439100) [18:22:13] (03PS1) 10KineticPelagic: PP3.4: Update main_app.version to desired tag of container [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353354 (https://phabricator.wikimedia.org/T434137) [18:24:46] (03CR) 10Dzahn: [C:03+2] zuul: migrate icinga checks to prometheus [puppet] - 10https://gerrit.wikimedia.org/r/1343955 (https://phabricator.wikimedia.org/T384939) (owner: 10Hnowlan) [18:26:33] (03CR) 10Dzahn: "the blackbox check runs from an external host. but monitoring hosts are generally allowed everywhere from some rules in the base class." [puppet] - 10https://gerrit.wikimedia.org/r/1343955 (https://phabricator.wikimedia.org/T384939) (owner: 10Hnowlan) [18:34:06] (03CR) 10AKhatun: [C:03+1] "TY!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353264 (https://phabricator.wikimedia.org/T439879) (owner: 10Clément Goubert) [18:54:21] (03CR) 10Ayounsi: "Post merge -1 but the SVC records are handled by the capirca script. They're just not grouped." [homer/public] - 10https://gerrit.wikimedia.org/r/1353197 (https://phabricator.wikimedia.org/T436809) (owner: 10Cathal Mooney) [18:56:29] (03CR) 10Andrew Bogott: [C:03+2] PP3.4: Update main_app.version to desired tag of container [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353354 (https://phabricator.wikimedia.org/T434137) (owner: 10KineticPelagic) [18:56:45] !log oblivian@cumin1004 END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts wikikube-worker1081.eqiad.wmnet [18:58:16] (03Merged) 10jenkins-bot: PP3.4: Update main_app.version to desired tag of container [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353354 (https://phabricator.wikimedia.org/T434137) (owner: 10KineticPelagic) [18:58:27] (03PS7) 10Krinkle: Profiler: Fold excimer line parsing into one method [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1329640 [19:00:48] !log andrew@deploy1003 helmfile [staging] START helmfile.d/services/developer-portal: apply [19:00:59] !log andrew@deploy1003 helmfile [staging] DONE helmfile.d/services/developer-portal: apply [19:01:27] !log andrew@deploy1003 helmfile [codfw] START helmfile.d/services/developer-portal: apply [19:01:46] !log andrew@deploy1003 helmfile [codfw] DONE helmfile.d/services/developer-portal: apply [19:02:02] !log andrew@deploy1003 helmfile [eqiad] START helmfile.d/services/developer-portal: apply [19:02:11] (03CR) 10Blake: [C:03+1] shellbox: Pick up new images [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353336 (https://phabricator.wikimedia.org/T439100) (owner: 10Kamila Součková) [19:02:19] !log andrew@deploy1003 helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply [19:03:23] (03PS1) 10Dzahn: miscweb: attribution: bump image to 2026-10-08-185901 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353370 (https://phabricator.wikimedia.org/T437635) [19:04:18] (03CR) 10Dzahn: [C:03+2] miscweb: attribution: bump image to 2026-10-08-185901 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353370 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn) [19:05:58] (03Merged) 10jenkins-bot: miscweb: attribution: bump image to 2026-10-08-185901 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353370 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn) [19:06:26] !log dzahn@deploy1003 helmfile [staging] START helmfile.d/services/miscweb: apply [19:06:46] !log dzahn@deploy1003 helmfile [staging] DONE helmfile.d/services/miscweb: apply [19:07:00] !log dzahn@deploy1003 helmfile [eqiad] START helmfile.d/services/miscweb: apply [19:07:16] !log dzahn@deploy1003 helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [19:07:28] !log dzahn@deploy1003 helmfile [codfw] START helmfile.d/services/miscweb: apply [19:07:49] !log dzahn@deploy1003 helmfile [codfw] DONE helmfile.d/services/miscweb: apply [19:13:03] (03CR) 10Scott French: [C:03+1] shellbox: Pick up new images [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353336 (https://phabricator.wikimedia.org/T439100) (owner: 10Kamila Součková) [19:17:21] (03PS2) 10Ahmon Dancy: profile::puppetserver::volatile: Remove GeoIP fetch without private access [puppet] - 10https://gerrit.wikimedia.org/r/1353306 [19:17:42] (03CR) 10Kamila Součková: [C:03+2] shellbox: Pick up new images [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353336 (https://phabricator.wikimedia.org/T439100) (owner: 10Kamila Součková) [19:19:36] (03Merged) 10jenkins-bot: shellbox: Pick up new images [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353336 (https://phabricator.wikimedia.org/T439100) (owner: 10Kamila Součková) [19:22:39] !log kamila@deploy1003 Started scap deploy-service shellbox: PHP update [19:22:39] !log kamila@deploy1003 scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video [19:25:31] yay! [19:26:08] !log kamila@deploy1003 scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video [19:26:50] Looking at these announcements from the outside, it is very unclear what "DEFAULT" means. [19:42:10] (03PS3) 10Ahmon Dancy: profile::puppetserver::volatile: Remove GeoIP fetch without private access [puppet] - 10https://gerrit.wikimedia.org/r/1353306 [19:42:37] (03CR) 10Ahmon Dancy: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1353306 (owner: 10Ahmon Dancy) [19:45:19] (03CR) 10Ahmon Dancy: "Jesse, for your consideration. This is currently beta cherry picked." [puppet] - 10https://gerrit.wikimedia.org/r/1353306 (owner: 10Ahmon Dancy) [19:57:03] (03CR) 10DLynch: VisualEditor: Precompute edit suggestion counts on testwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351444 (https://phabricator.wikimedia.org/T432733) (owner: 10DLynch) [19:57:20] !log kamila@deploy1003 scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video [19:57:35] (03PS3) 10DLynch: VisualEditor: Precompute edit suggestion counts on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351444 (https://phabricator.wikimedia.org/T432733) [19:58:07] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 08 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351444 (https://phabricator.wikimedia.org/T432733) (owner: 10DLynch) [19:58:45] !log kamila@deploy1003 Finished scap deploy-service shellbox: PHP update (duration: 36m 05s) [19:58:59] (03PS1) 10Clare Ming: Override experiment enrollments via X-Wikimedia-Debug header [extensions/TestKitchen] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353392 (https://phabricator.wikimedia.org/T437240) [19:59:11] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 08 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/TestKitchen] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353392 (https://phabricator.wikimedia.org/T437240) (owner: 10Clare Ming) [19:59:16] amazing :D [19:59:32] dancy: true, what does it mean? [20:00:04] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T2000). Please do the needful. [20:00:04] obvious-dust, matthiasmullie, ejegg, cjming, and kemayo: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:22] o/ [20:00:27] o/ [20:01:32] I need a deployer [20:02:00] hello [20:02:01] o/ [20:04:42] If others want to go first, I think mine has the highest odds of needing a rollback. [20:05:14] hehe, I'm slightly skittish about my CentralNotice change too, so i'll go second-to-last [20:05:35] raine: It's the stage name, defined in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/role/common/deployment_server/kubernetes.yaml#533 [20:05:38] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cjming@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339561 (https://phabricator.wikimedia.org/T434930) (owner: 10Milazg) [20:05:38] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cjming@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352261 (https://phabricator.wikimedia.org/T437720) (owner: 10Clare Ming) [20:05:59] Raine: in this context, it's the deployment stage that contains all non-canary releases (and indeed shellbox doesn't have canaries, so you only ever see default). as dancy points out, the name is arbitrary, and we can perhaps make it clearer :) [20:06:25] (03Merged) 10jenkins-bot: Add URLs for MobileAppRedirect Special Page [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1339561 (https://phabricator.wikimedia.org/T434930) (owner: 10Milazg) [20:06:28] (03Merged) 10jenkins-bot: Update Test Kitchen instruments intake URL on beta cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352261 (https://phabricator.wikimedia.org/T437720) (owner: 10Clare Ming) [20:06:40] !log cjming@deploy1003 Started scap sync-world: Backport for [[gerrit:1339561|Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261|Update Test Kitchen instruments intake URL on beta cluster (T437720)]] [20:06:40] oh shoot - sorry i was in the wrong channel [20:06:46] T434930: Create MobileAppRedirect special page to redirect users to the correct app store to download the Wikipedia App - https://phabricator.wikimedia.org/T434930 [20:06:46] T437720: Enable instruments to send events when running on Beta Cluster - https://phabricator.wikimedia.org/T437720 [20:07:33] i started with obvious-dust's and my config patch - after it's done, i'll pass over to matthiasmullie [20:07:39] Sounds good. [20:07:51] ejegg: do you need a deployer? [20:08:10] cjming: I have spiderpig access, so I think I can do this one. Thank you! [20:08:23] cjming: perfect, I’ll wait for your ping [20:08:56] cool - so i think we can self-service from here on out - just ping the next person in the queue after you're done [20:09:04] cool cool [20:10:43] !log cjming@deploy1003 milazg, cjming: Backport for [[gerrit:1339561|Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261|Update Test Kitchen instruments intake URL on beta cluster (T437720)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:11:01] obvious-dust: good to sync? [20:11:15] I will check [20:12:23] (03PS4) 10DLynch: VisualEditor: Precompute edit suggestion counts on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351444 (https://phabricator.wikimedia.org/T432733) [20:14:03] !log cjming@deploy1003 milazg, cjming: Continuing with deployment [20:14:18] ^^ per greenlight from obvious-dust [20:14:59] 👍 [20:17:32] (03CR) 10Ottomata: [C:03+1] VisualEditor: Precompute edit suggestion counts on testwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351444 (https://phabricator.wikimedia.org/T432733) (owner: 10DLynch) [20:18:05] (03CR) 10DLynch: VisualEditor: Precompute edit suggestion counts on testwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351444 (https://phabricator.wikimedia.org/T432733) (owner: 10DLynch) [20:20:45] ejegg: do you want me to go before you? otherwise I will ask you to ping me after you're done [20:20:54] (per backscroll) [20:21:14] sure, please do cjming [20:21:21] I mean please do go before me [20:21:29] cool - thanks! [20:21:36] !log cjming@deploy1003 Finished scap sync-world: Backport for [[gerrit:1339561|Add URLs for MobileAppRedirect Special Page (T434930)]], [[gerrit:1352261|Update Test Kitchen instruments intake URL on beta cluster (T437720)]] (duration: 14m 56s) [20:21:39] then I will ping Kemayo when I finish [20:21:42] T434930: Create MobileAppRedirect special page to redirect users to the correct app store to download the Wikipedia App - https://phabricator.wikimedia.org/T434930 [20:21:42] T437720: Enable instruments to send events when running on Beta Cluster - https://phabricator.wikimedia.org/T437720 [20:21:49] matthiasmullie: over to you - will you ping me when you're done? [20:22:02] Will do! [20:22:15] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/ReaderExperiments] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353257 (owner: 10Matthias Mullie) [20:23:13] (03Merged) 10jenkins-bot: Temporary: don't catch exception [extensions/ReaderExperiments] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353257 (owner: 10Matthias Mullie) [20:23:26] !log mlitn@deploy1003 Started scap sync-world: Backport for [[gerrit:1353257|Temporary: don't catch exception]] [20:26:35] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users + Kerberos for OBenhmida - https://phabricator.wikimedia.org/T440464#12413829 (10MMiller_WMF) Sonja is on PTO, and I am Sonja's manager (@OBenhmida's skip level) -- I approve of this access. Thank you! [20:27:02] (03PS2) 10Dr0ptp4kt: Create dumps-admins for dumpsgen access for analytics_admins_members [puppet] - 10https://gerrit.wikimedia.org/r/1352546 (https://phabricator.wikimedia.org/T440471) [20:27:26] !log mlitn@deploy1003 mlitn: Backport for [[gerrit:1353257|Temporary: don't catch exception]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:27:54] !log mlitn@deploy1003 mlitn: Continuing with deployment [20:33:08] !log mlitn@deploy1003 Finished scap sync-world: Backport for [[gerrit:1353257|Temporary: don't catch exception]] (duration: 09m 41s) [20:33:38] (03CR) 10Clare Ming: [C:03+2] Override experiment enrollments via X-Wikimedia-Debug header [extensions/TestKitchen] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353392 (https://phabricator.wikimedia.org/T437240) (owner: 10Clare Ming) [20:33:41] cjming: done! Back to you! [20:33:54] ty! [20:34:05] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cjming@deploy1003 using scap backport" [extensions/TestKitchen] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353392 (https://phabricator.wikimedia.org/T437240) (owner: 10Clare Ming) [20:37:17] (03CR) 10Dr0ptp4kt: "Updated per conversation." [puppet] - 10https://gerrit.wikimedia.org/r/1352546 (https://phabricator.wikimedia.org/T440471) (owner: 10Dr0ptp4kt) [20:38:22] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to dumps-roots for analytics_admins_members (people associated with analytics-admins) - https://phabricator.wikimedia.org/T440471#12413885 (10dr0ptp4kt) I've updated the CL, please do advise in case of any further changes needed. Thanks! [20:38:23] 10ops-codfw, 06SRE, 06Collaboration-Services, 06DC-Ops: lists2001 has multiple bus errors - https://phabricator.wikimedia.org/T423159#12413886 (10Jhancock.wm) yeah errors are gone. sorry i missed these [20:38:28] (03Merged) 10jenkins-bot: Override experiment enrollments via X-Wikimedia-Debug header [extensions/TestKitchen] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353392 (https://phabricator.wikimedia.org/T437240) (owner: 10Clare Ming) [20:38:42] !log cjming@deploy1003 Started scap sync-world: Backport for [[gerrit:1353392|Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] [20:38:45] T437240: Add Test Kitchen functionality to WikimediaDebug - https://phabricator.wikimedia.org/T437240 [20:39:03] (03CR) 10Dr0ptp4kt: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1352546 (https://phabricator.wikimedia.org/T440471) (owner: 10Dr0ptp4kt) [20:42:20] 06SRE, 10MoveComms-Support, 10Wikimedia-Mailing-lists, 07Epic: rename VRT mailing lists - https://phabricator.wikimedia.org/T289065#12413898 (10Keegan) 05Open→03Declined [20:42:34] !log cjming@deploy1003 cjming: Backport for [[gerrit:1353392|Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:43:06] !log cjming@deploy1003 cjming: Continuing with deployment [20:47:59] Could someone ping me when the entire backport window is done? (cc cjming) [20:48:03] I need to deploy a security patch after [20:48:21] !log cjming@deploy1003 Finished scap sync-world: Backport for [[gerrit:1353392|Override experiment enrollments via X-Wikimedia-Debug header (T437240)]] (duration: 09m 40s) [20:48:25] ejegg: all yours [20:48:26] T437240: Add Test Kitchen functionality to WikimediaDebug - https://phabricator.wikimedia.org/T437240 [20:48:33] RoanKattouw: sure [20:48:57] Kemayo: mind pinging Roan after you're done? [20:49:05] cjming: Can do. [20:49:46] ejegg: still around? [20:49:54] thanks cjming [20:49:56] starting [20:50:28] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ejegg@deploy1003 using scap backport" [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353269 (owner: 10Ejegg) [20:57:37] (03Merged) 10jenkins-bot: Update CentralNotice to 658422ff5fed886 [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353269 (owner: 10Ejegg) [20:58:08] ah dang, finished with errors? [20:59:32] ah, probably the now-public security patches conflicting with the private ones [21:00:04] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261008T2100) [21:00:15] let me see if I can figure out how to fix that [21:02:41] ok, I've confirmed that the two CN security patches in /srv/patches are included in the public merge I'm trying to deploy now [21:03:14] cjming: so do I just delete them from /srv/patches/.../CentralNotice? [21:03:23] and then redo the scap backport? [21:03:35] or do I need another step in between? [21:05:13] looks like I should be able to re-scap with just the deletion [21:06:09] ah, i need to commit in /srv/patches [21:07:09] !log ejegg@deploy1003 Started scap sync-world: Backport for [[gerrit:1353269|Update CentralNotice to 658422ff5fed886]] [21:13:27] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply [21:14:06] !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply [21:14:36] Sorry about that ejegg, yes we published a big chunk of private security mitigations last week [21:15:06] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-c1-eqiad [21:15:24] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-c1-eqiad [21:16:05] (03CR) 10Cathal Mooney: [C:03+2] Add lsw1-c1-eqiad to devices [homer/public] - 10https://gerrit.wikimedia.org/r/1353292 (https://phabricator.wikimedia.org/T403031) (owner: 10Cathal Mooney) [21:16:20] heh, yep, nice that we're learning about the vulns early though. i wrote those two patches so I should have anticipated the conflict, just been a while since I deployed [21:16:25] ejegg: sorry i walked away - are you good? [21:16:41] cjming: yep, just waiting for k8s images build/push [21:16:46] seems to be taking its sweet time [21:16:56] thanks for checking! [21:17:08] (03Merged) 10jenkins-bot: Add lsw1-c1-eqiad to devices [homer/public] - 10https://gerrit.wikimedia.org/r/1353292 (https://phabricator.wikimedia.org/T403031) (owner: 10Cathal Mooney) [21:21:30] k, builds completed, finally syncing [21:23:32] (03CR) 10Ladsgroup: [C:03+2] "Yeah, I'm planning to do that. We have done this before." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1353028 (https://phabricator.wikimedia.org/T420223) (owner: 10Ladsgroup-claude) [21:24:25] FIRING: KubernetesCalicoDown: wikikube-worker1081.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1081.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [21:25:19] * ejegg wonders if ^^^ is related to this backport taking forever [21:25:39] FIRING: CoreBGPDown: Core BGP session down between ssw1-d8-eqiad and lsw1-c1-eqiad (10.64.128.51) - group ibgp_evpn - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=ssw1-d8-eqiad:9804&var-bgp_group=ibgp_evpn&var-bgp_neighbor=lsw1-c1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [21:26:03] errr, core BGP down? [21:26:04] FIRING: OspfAdjDown: OSPF Adjacency down on ssw1-d8-eqiad interface ethernet-1/1.0 - https://wikitech.wikimedia.org/wiki/Network_monitoring#OSPF_status - https://grafana.wikimedia.org/d/b77db156-d852-4601-acc5-4065b888e5fe/ospf-status-nokia?orgId=1&var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DOspfAdjDown [21:27:08] !log ejegg@deploy1003 ejegg: Backport for [[gerrit:1353269|Update CentralNotice to 658422ff5fed886]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:27:44] ejegg: so, I've never been able to track this down, but *sometimes* changing things in /srv/patches can trigger a full MediaWiki image rebuild ... similar to the effect of an l10n update. that means both a slow build and a slow push (using ~ 30-35m end-to-end). [21:27:49] ops folks, should I abort the backport till those BGP and OSPF things are worked out? [21:28:03] ahhh, that's probably it swfrench-wmf [21:28:29] now that it's on the test servers i will verify [21:28:44] ejegg: I think you're good to continue - I think t.opranks is doing work on that ToR swtich [21:29:02] ok, thanks for the confirmation [21:29:19] ref. https://gerrit.wikimedia.org/r/1353292 merged just above in the backscroll [21:30:16] (03Merged) 10jenkins-bot: swift: Load originals in 64MB chunks [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1353028 (https://phabricator.wikimedia.org/T420223) (owner: 10Ladsgroup-claude) [21:32:04] ok, only change to ChoiceData is a re-ordering of property names [21:32:48] oh darn, Error: Cannot require undefined file config.json [21:33:00] 10ops-codfw, 06SRE, 06Collaboration-Services, 06DC-Ops: lists2001 has multiple bus errors - https://phabricator.wikimedia.org/T423159#12414014 (10Dzahn) ok, great! thank you [21:34:59] Is this just a resourceloader server / client sync issue? [21:35:15] clearing cache locally does no good [21:37:26] i need to revert [21:37:41] ah which means I need to restore those private security patches [21:37:49] !log ejegg@deploy1003 ejegg: Rolling back deployment [21:39:46] ejegg: my apologies you can ignore those alerts [21:40:13] I enabled new ports for a new switch, but it turns out the cables were wired the wrong way around, so ospf/bgp didn't come up [21:40:23] dc-ops will reverse tomorrow, I've ack'd the alerts for now [21:40:38] thanks for the explanation! [21:40:58] np - and thanks Scott as always for being on top of things :) [21:41:36] And sorry to everyone waiting for deploy, I just figured out the issue with my deploy [21:42:04] but I'll go to the back of the line while I do more local testing with the fixed merge [21:42:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:42:31] !log ejegg@deploy1003 Finished scap sync-world: Backport for [[gerrit:1353269|Update CentralNotice to 658422ff5fed886]] (duration: 35m 22s) [21:42:53] ejegg: I'll get my config one in then, if you're done with any cleanup needed? [21:43:04] Kemayo: go right ahead [21:43:13] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351444 (https://phabricator.wikimedia.org/T432733) (owner: 10DLynch) [21:44:10] (03Merged) 10jenkins-bot: VisualEditor: Precompute edit suggestion counts on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1351444 (https://phabricator.wikimedia.org/T432733) (owner: 10DLynch) [21:45:01] Hm, scap errored out after the merge. [21:45:16] `prep failed: Command '['/usr/bin/scap', 'apply-patches', '--abort-git-am-on-fail', '--train', '1.47.0-wmf.23']' returned non-zero exit status 1. (scap version: 4.296.0) (duration: 00m 07s)` [21:45:57] hmm maybe try rerunning? [21:46:20] Worth a shot. [21:46:26] Oh no wait [21:46:34] It's because of the CentralNotice security patch that no longer applies [21:46:37] ejegg has to fix that first [21:47:33] oh sorry, I need to revert the wmf-23 submodule update too? [21:48:02] (03PS1) 10Ejegg: Revert "Update CentralNotice to 658422ff5fed886" [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353451 [21:48:29] RoanKattouw: I should just C+2 that myself, right? ^^^ [21:48:48] (03CR) 10Ejegg: [C:03+2] Revert "Update CentralNotice to 658422ff5fed886" [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353451 (owner: 10Ejegg) [21:48:54] * ejegg hopes so [21:49:06] Ah yes I believe so [21:49:16] Or [21:49:30] Maybe not, let me check in a second [21:49:57] it never actually synced out, so I would assume it needs to be reverted to let anything else go ahead [21:50:22] I actually thought spiderpig would revert it for me but it appears not [21:50:40] CN is also weird because I think it's the only extension where you have to manually do submodule updates [21:50:45] yeah [21:50:54] TIL [21:51:19] (We should consider changing that IMO and making CN work more similarly to everything else) [21:51:45] (But that also implicates other special snowflake aspects of CN) [21:52:08] yeah, I guess there are potentially things that need to sync between metawiki and target wikis but we haven't made that kind of breaking change in a decade [21:52:48] the other worry was always changes going out during big english, but it's just as likely to get a change from elsewhere in the ecosystem breaking banner display [21:54:50] (03PS1) 10Ladsgroup: thumbor: Bump to the most recent image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353453 (https://phabricator.wikimedia.org/T420223) [21:55:01] (03CR) 10Dzahn: gerrit: allow puppetservers to reach the envoy tlsproxy (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1328152 (https://phabricator.wikimedia.org/T420184) (owner: 10Arnaudb) [21:55:48] (03Merged) 10jenkins-bot: Revert "Update CentralNotice to 658422ff5fed886" [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353451 (owner: 10Ejegg) [21:55:58] ok Kemayo want to try again? [21:56:36] Hmm. "There were unexpected commits pulled from origin for /srv/mediawiki-staging/php-1.47.0-wmf.23. Would you like to see the diff?" [21:57:10] (03CR) 10Ladsgroup: [C:03+2] thumbor: Bump to the most recent image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353453 (https://phabricator.wikimedia.org/T420223) (owner: 10Ladsgroup) [21:57:10] The diff does look very centralnotice-y. Should I just go ahead with it? [21:57:20] Visible in the log here: https://spiderpig.wikimedia.org/jobs/2964 [21:57:31] (03CR) 10Dzahn: gerrit: allow puppetservers to reach the envoy tlsproxy (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1328152 (https://phabricator.wikimedia.org/T420184) (owner: 10Arnaudb) [21:58:04] (03CR) 10Dzahn: [C:03+1] gerrit: allow puppetservers to reach the envoy tlsproxy [puppet] - 10https://gerrit.wikimedia.org/r/1328152 (https://phabricator.wikimedia.org/T420184) (owner: 10Arnaudb) [21:58:05] (03CR) 10Herron: [C:03+2] Test new rsyslog image on mw-experimental [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352121 (https://phabricator.wikimedia.org/T438375) (owner: 10Herron) [21:58:47] (03Merged) 10jenkins-bot: thumbor: Bump to the most recent image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353453 (https://phabricator.wikimedia.org/T420223) (owner: 10Ladsgroup) [21:59:05] Kemayo: I think that's expected, it's ejegg's revert [21:59:42] (03Merged) 10jenkins-bot: Test new rsyslog image on mw-experimental [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352121 (https://phabricator.wikimedia.org/T438375) (owner: 10Herron) [21:59:55] I'm sort of surprised the diff is showing more than just the submodule commit-change, but I don't mind going ahead and seeing if it'll cause problems. [21:59:58] (03PS1) 10Ejegg: Update CentralNotice to cd3f46a4053cb74ff50b003e [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353458 [22:00:12] !log ladsgroup@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [22:00:19] !log kemayo@deploy1003 Started scap sync-world: Backport for [[gerrit:1351444|VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] [22:00:22] !log ladsgroup@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [22:00:22] T432733: Server Side Edit Suggestions - Linked Artifacts Cache precompute job - https://phabricator.wikimedia.org/T432733 [22:01:51] Kemayo: once yours is up on test servers I'll check again to make sure the bug I saw in my deploy isn't there [22:02:06] > 22:00:54 549 languages rebuilt out of 549 [22:02:11] Oh great this is gonna take forever [22:02:23] (Because the CN revert included i18n changes) [22:02:28] oof, sorry again [22:02:31] !log herron@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-experimental: apply [22:02:37] That's OK, stuff happens [22:03:01] upside: we're heading into a long weekend? [22:03:04] But if you all put CN on the train then this kind of thing will be much rarer ;) [22:03:12] heh, yep [22:03:13] !log herron@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply [22:03:33] !log herron@deploy1003 helmfile [codfw] START helmfile.d/services/mw-experimental: apply [22:04:24] !log herron@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-experimental: apply [22:11:23] !log ladsgroup@deploy1003 helmfile [codfw] START helmfile.d/services/thumbor: apply [22:13:35] !log ladsgroup@deploy1003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [22:16:37] (03CR) 10Dzahn: "from my understanding I would suggest this order:" [dns] - 10https://gerrit.wikimedia.org/r/1351524 (https://phabricator.wikimedia.org/T436917) (owner: 10Jasmine) [22:20:23] !log ladsgroup@deploy1003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [22:20:45] !log kemayo@deploy1003 kemayo: Backport for [[gerrit:1351444|VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:20:48] T432733: Server Side Edit Suggestions - Linked Artifacts Cache precompute job - https://phabricator.wikimedia.org/T432733 [22:20:52] ejegg: It's on the testservers if you want to check for your bug. [22:21:27] looking [22:22:10] looks good Kemayo, no errors [22:22:35] !log ladsgroup@deploy1003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [22:25:10] !log kemayo@deploy1003 kemayo: Continuing with deployment [22:33:29] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1353471 [22:34:27] 06SRE, 06Collaboration-Services, 06Traffic: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12414147 (10Dzahn) notes from debugging: - Apache isn't the problem. It sent the whole 99 MB file in 0.23 seconds (about 435 MB/s), so the disk and Apache are fine. (from c... [22:36:58] !log kemayo@deploy1003 Finished scap sync-world: Backport for [[gerrit:1351444|VisualEditor: Precompute edit suggestion counts on testwiki (T432733)]] (duration: 36m 40s) [22:37:02] T432733: Server Side Edit Suggestions - Linked Artifacts Cache precompute job - https://phabricator.wikimedia.org/T432733 [22:37:26] ejegg / RoanKattouw: okay, I'm done, you're free to carry one. [22:37:28] *on [22:37:40] thanks, RoanKattouw please go ahead [22:45:13] (03PS1) 10Dzahn: service: mark releases as active/passive [puppet] - 10https://gerrit.wikimedia.org/r/1353476 (https://phabricator.wikimedia.org/T440068) [22:45:49] I'm getting an auto-away from RoanKattouw so I'll just deploy the fixed version of my patch [22:46:45] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ejegg@deploy1003 using scap backport" [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353458 (owner: 10Ejegg) [22:47:46] ejegg: sorry yes please go ahead, I'm eating dinner [22:49:21] cool cool, sure to take a while again this time but at least I have the patches sorted in advance to avoid conflicts [22:51:02] (03Merged) 10jenkins-bot: Update CentralNotice to cd3f46a4053cb74ff50b003e [core] (wmf/1.47.0-wmf.23) - 10https://gerrit.wikimedia.org/r/1353458 (owner: 10Ejegg) [22:51:19] !log ejegg@deploy1003 Started scap sync-world: Backport for [[gerrit:1353458|Update CentralNotice to cd3f46a4053cb74ff50b003e]] [23:11:41] !log ejegg@deploy1003 ejegg: Backport for [[gerrit:1353458|Update CentralNotice to cd3f46a4053cb74ff50b003e]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [23:12:13] ok, ChoiceData looks good, just sorted now [23:15:21] No errors anyway [23:15:34] I'd like to get myself targeted for a banner [23:16:30] yep, country=FR gets a banner. will proceed [23:16:42] !log ejegg@deploy1003 ejegg: Continuing with deployment [23:27:43] FIRING: RipeAtlasAnchorUnreachable: ipv4 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140314 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [23:28:45] !log ejegg@deploy1003 Finished scap sync-world: Backport for [[gerrit:1353458|Update CentralNotice to cd3f46a4053cb74ff50b003e]] (duration: 37m 26s) [23:32:43] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv4 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140314 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [23:35:21] looks all quiet in the client error logs [23:39:00] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1353500 [23:39:00] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1353500 (owner: 10TrainBranchBot) [23:39:12] !log fceratto@cumin1004 END (FAIL) - Cookbook sre.mysql.pool (exit_code=99) pool db2902: Pool db2902.codfw.wmnet in after cloning [23:42:12] fceratto@cumin1004 clone (PID 3770943) is awaiting input [23:45:08] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1353500 (owner: 10TrainBranchBot) [23:47:14] 06SRE, 06Collaboration-Services, 06Traffic, 13Patch-For-Review: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12414306 (10Bewfip) Tried downloading mediawiki tarball from my home ISP after noticing slow transfer speed with the API when receiving large responses...