[00:00:07] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [00:00:17] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [00:00:21] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [00:00:31] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [00:00:34] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [00:00:45] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [00:01:00] FIRING: PingLossPercent: Blackbox probe packet loss 6.667% from codfw to magru - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=magru - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [00:03:35] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97323 and previous config saved to /var/cache/conftool/dbconfig/20261105-000334-ladsgroup.json [00:04:18] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet [00:04:20] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet [00:04:25] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet [00:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:05:01] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet [00:11:28] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet [00:11:30] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet [00:11:35] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet [00:13:37] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2221 (T441633)', diff saved to https://phabricator.wikimedia.org/P97324 and previous config saved to /var/cache/conftool/dbconfig/20261105-001336-ladsgroup.json [00:13:56] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance [00:13:58] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97325 and previous config saved to /var/cache/conftool/dbconfig/20261105-001358-ladsgroup.json [00:17:22] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [00:18:46] (03PS1) 10Dwisehaupt: Add FIDO SSH key for dwisehaupt [puppet] - 10https://gerrit.wikimedia.org/r/1363504 (https://phabricator.wikimedia.org/T442759) [00:21:02] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97326 and previous config saved to /var/cache/conftool/dbconfig/20261105-002101-ladsgroup.json [00:21:53] (03PS1) 10Dwisehaupt: Add FIDO SSH key for eileen [puppet] - 10https://gerrit.wikimedia.org/r/1363505 [00:22:20] (03CR) 10CI reject: [V:04-1] Add FIDO SSH key for eileen [puppet] - 10https://gerrit.wikimedia.org/r/1363505 (owner: 10Dwisehaupt) [00:30:05] (03Abandoned) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1362259 (owner: 10TrainBranchBot) [00:31:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.542% from codfw to magru - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=magru - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [00:31:04] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97327 and previous config saved to /var/cache/conftool/dbconfig/20261105-003103-ladsgroup.json [00:39:02] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1363507 [00:39:02] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1363507 (owner: 10TrainBranchBot) [00:41:06] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97328 and previous config saved to /var/cache/conftool/dbconfig/20261105-004105-ladsgroup.json [00:41:38] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet [00:45:31] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1363507 (owner: 10TrainBranchBot) [00:45:52] (03PS1) 10Dduvall: ci: Configure docker to use the containerd snapshotter [puppet] - 10https://gerrit.wikimedia.org/r/1363509 (https://phabricator.wikimedia.org/T440668) [00:51:08] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97329 and previous config saved to /var/cache/conftool/dbconfig/20261105-005107-ladsgroup.json [00:51:28] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance [00:51:30] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2228 (T441633)', diff saved to https://phabricator.wikimedia.org/P97330 and previous config saved to /var/cache/conftool/dbconfig/20261105-005129-ladsgroup.json [00:52:26] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet [00:52:27] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet [00:52:33] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet [01:09:05] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1363523 [01:09:05] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1363523 (owner: 10TrainBranchBot) [01:15:02] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1363523 (owner: 10TrainBranchBot) [01:22:36] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet [01:31:16] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet [01:31:17] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet [01:31:23] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet [01:35:33] 06SRE, 10LDAP-Access-Requests: Grant Access to Logstash for SomeRandomDeveloper - https://phabricator.wikimedia.org/T442855#12520479 (10andrea.denisse) 05Stalled→03In progress [01:51:02] (03PS1) 10Andrea Denisse: admin: Add somerandomdeveloper to ldap_only_users [puppet] - 10https://gerrit.wikimedia.org/r/1363539 (https://phabricator.wikimedia.org/T442855) [01:51:54] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2228 (T441633)', diff saved to https://phabricator.wikimedia.org/P97331 and previous config saved to /var/cache/conftool/dbconfig/20261105-015153-ladsgroup.json [01:52:04] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 92.84% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [02:00:04] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T0200) [02:01:20] 06SRE, 10LDAP-Access-Requests, 13Patch-For-Review: Grant Access to Logstash for SomeRandomDeveloper - https://phabricator.wikimedia.org/T442855#12520597 (10andrea.denisse) >>! In T442855#12505727, @KFrancis wrote: > Hi all, I also answered this on another ticket, but yes, the user was accidentally omitted fr... [02:01:24] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet [02:01:48] !log mwpresync@deploy2003 Started scap build-images: Publishing wmf/next image [02:01:56] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97332 and previous config saved to /var/cache/conftool/dbconfig/20261105-020155-ladsgroup.json [02:03:29] 06SRE, 10LDAP-Access-Requests, 13Patch-For-Review: Grant Access to Logstash for SomeRandomDeveloper - https://phabricator.wikimedia.org/T442855#12520599 (10andrea.denisse) 05In progress→03Resolved @SomeRandomDeveloper I added you to the nda LDAP group and your entry is in data.yaml as "(known to Lega... [02:09:18] 06SRE, 10LDAP-Access-Requests, 13Patch-For-Review: Grant Access to Logstash for SomeRandomDeveloper - https://phabricator.wikimedia.org/T442855#12520616 (10SomeRandomDeveloper) Thank you! I've submitted the access request through IDM. [02:09:19] !log mwpresync@deploy2003 Finished scap build-images: Publishing wmf/next image (duration: 07m 31s) [02:09:34] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet [02:09:35] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet [02:09:40] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet [02:10:12] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:11:58] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97333 and previous config saved to /var/cache/conftool/dbconfig/20261105-021157-ladsgroup.json [02:12:28] 06SRE, 10SRE-Access-Requests, 06Data-Engineering: Requesting access to Superset for brookecamarda - https://phabricator.wikimedia.org/T441893#12520637 (10andrea.denisse) a:05Brookecamarda→03andrea.denisse [02:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:17:40] (03PS1) 10Andrea Denisse: admin: Add brookecamarda to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1363546 (https://phabricator.wikimedia.org/T441893) [02:18:14] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 13Patch-For-Review: Requesting access to Superset for brookecamarda - https://phabricator.wikimedia.org/T441893#12520646 (10andrea.denisse) [02:22:00] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2228 (T441633)', diff saved to https://phabricator.wikimedia.org/P97334 and previous config saved to /var/cache/conftool/dbconfig/20261105-022159-ladsgroup.json [02:23:44] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet [02:23:45] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet [02:23:50] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet [02:24:22] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet [02:25:20] (03PS1) 10DDesouza: miscweb(design-landing-page): bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363548 (https://phabricator.wikimedia.org/T344471) [02:29:09] (03CR) 10DDesouza: [C:03+2] miscweb(design-landing-page): bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363548 (https://phabricator.wikimedia.org/T344471) (owner: 10DDesouza) [02:31:06] (03Merged) 10jenkins-bot: miscweb(design-landing-page): bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363548 (https://phabricator.wikimedia.org/T344471) (owner: 10DDesouza) [02:32:20] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet [02:32:21] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet [02:32:26] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet [02:34:43] !log dani@deploy2003 helmfile [staging] START helmfile.d/services/miscweb: apply [02:34:59] !log dani@deploy2003 helmfile [staging] DONE helmfile.d/services/miscweb: apply [02:35:00] !log dani@deploy2003 helmfile [eqiad] START helmfile.d/services/miscweb: apply [02:35:15] !log dani@deploy2003 helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [02:35:16] !log dani@deploy2003 helmfile [codfw] START helmfile.d/services/miscweb: apply [02:35:28] !log dani@deploy2003 helmfile [codfw] DONE helmfile.d/services/miscweb: apply [02:49:20] (03PS1) 10Jdrewniak: admin: Add FIDO SSH key for jdrewniak [puppet] - 10https://gerrit.wikimedia.org/r/1363564 (https://phabricator.wikimedia.org/T442758) [03:02:28] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet [03:10:27] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet [03:10:28] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet [03:10:33] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1039.eqiad.wmnet [03:11:06] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1039.eqiad.wmnet [03:16:24] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1039.eqiad.wmnet [03:16:25] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1039.eqiad.wmnet [03:16:31] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1040.eqiad.wmnet [03:20:44] 06SRE, 10LDAP-Access-Requests: Grant Access to [analytics resources: Superset, Jupyter, Growthbook, Turnilo, Test Kitchen UI] for cfoster - https://phabricator.wikimedia.org/T443699#12520768 (10kzimmerman) [03:20:45] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12520769 (10kzimmerman) [03:29:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.333% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [03:31:58] 06SRE, 06Data-Engineering, 10LDAP-Access-Requests: Grant Access to [analytics resources: Superset, Jupyter, Growthbook, Turnilo, Test Kitchen UI] for cfoster - https://phabricator.wikimedia.org/T443699#12520774 (10kzimmerman) [03:34:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [03:34:24] (03PS1) 10Lerickson: admin: Add FIDO SSH keys for lerickson [puppet] - 10https://gerrit.wikimedia.org/r/1363606 (https://phabricator.wikimedia.org/T442758) [03:34:58] (03PS2) 10Lerickson: admin: Add FIDO SSH key for lerickson [puppet] - 10https://gerrit.wikimedia.org/r/1363606 (https://phabricator.wikimedia.org/T442758) [03:39:23] 06SRE, 10SRE-Access-Requests, 06Data-Engineering: Requesting access to Superset for brookecamarda - https://phabricator.wikimedia.org/T441893#12520794 (10andrea.denisse) 05In progress→03Resolved Hi @Brookecamarda, I added you to analytics-privatedata-users with level 1 access, I didn't add your SSH k... [03:41:11] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for mgerlach - https://phabricator.wikimedia.org/T443038#12520797 (10andrea.denisse) 05Stalled→03Resolved [03:42:22] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for kartik - https://phabricator.wikimedia.org/T443182#12520800 (10andrea.denisse) Hi @KartikMistry , once you confirm that you can access using the Yubikey we can remove the old key. [03:44:06] 06SRE, 10SRE-Access-Requests, 06Product Safety and Integrity, 06Security-Team, 07SecTeam-Processed: Request IRC #mediawiki_security access for Novem Linguae - https://phabricator.wikimedia.org/T442747#12520802 (10andrea.denisse) 05Open→03Stalled [03:46:23] 06SRE, 10SRE-Access-Requests: Requesting access to Analytics Data Lake for Federico Ceratto - https://phabricator.wikimedia.org/T443235#12520804 (10andrea.denisse) Hi, the `ops` gruop requires FIDO keys, the one you added in the task is plain ed25519 key but I see you already have two FIDO keys present that gi... [03:46:33] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1040.eqiad.wmnet [03:48:47] 06SRE, 10SRE-Access-Requests: Requesting access to Analytics Data Lake for Federico Ceratto - https://phabricator.wikimedia.org/T443235#12520806 (10andrea.denisse) [03:49:08] 06SRE, 10SRE-Access-Requests: Requesting access to Analytics Data Lake for Federico Ceratto - https://phabricator.wikimedia.org/T443235#12520807 (10andrea.denisse) Hi @HShaikh, do you approve of this request?? [03:50:36] 06SRE, 10SRE-Access-Requests, 06Data-Engineering: Requesting access to Analytics Data Lake for Federico Ceratto - https://phabricator.wikimedia.org/T443235#12520808 (10andrea.denisse) 05Open→03In progress [03:51:55] (03PS1) 10Andrea Denisse: admin: Add fceratto to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/1363637 (https://phabricator.wikimedia.org/T443235) [03:51:55] (03CR) 10Andrea Denisse: [C:04-2] "Waiting on manager's approval on main task." [puppet] - 10https://gerrit.wikimedia.org/r/1363637 (https://phabricator.wikimedia.org/T443235) (owner: 10Andrea Denisse) [03:52:35] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 13Patch-For-Review: Requesting access to Analytics Data Lake for Federico Ceratto - https://phabricator.wikimedia.org/T443235#12520812 (10andrea.denisse) 05In progress→03Stalled [03:55:00] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for Southparkfan - https://phabricator.wikimedia.org/T443141#12520814 (10andrea.denisse) 05Open→03In progress a:03andrea.denisse [03:56:04] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1040.eqiad.wmnet [03:56:05] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1040.eqiad.wmnet [03:56:11] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1041.eqiad.wmnet [04:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:10:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.542% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [04:11:03] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2018.codfw.wmnet, repooling both afterwards [04:11:06] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [04:11:22] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards [04:15:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.333% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [04:17:22] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [04:18:40] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2018.codfw.wmnet, repooling both afterwards [04:18:42] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2019.codfw.wmnet, repooling both afterwards [04:18:43] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [04:19:10] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards [04:19:12] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1026.eqiad.wmnet, repooling both afterwards [04:19:58] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for Southparkfan - https://phabricator.wikimedia.org/T443141#12520836 (10andrea.denisse) [04:20:19] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for Southparkfan - https://phabricator.wikimedia.org/T443141#12520837 (10andrea.denisse) 05In progress→03Resolved Closing as resolved, feel free to reopen if required. [04:23:04] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for snowick - https://phabricator.wikimedia.org/T442951#12520842 (10andrea.denisse) 05In progress→03Invalid Duplicate of T442756. [04:24:34] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for ttaylor - https://phabricator.wikimedia.org/T443457#12520845 (10andrea.denisse) 05Open→03Invalid Hi @ttaylor , closing this task as we're managing these requests trough T442756 . [04:26:14] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1041.eqiad.wmnet [04:26:17] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2019.codfw.wmnet, repooling both afterwards [04:26:18] !log ryankemper@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2020.codfw.wmnet, repooling both afterwards [04:26:20] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [04:27:03] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1026.eqiad.wmnet, repooling both afterwards [04:27:15] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for aranyap - https://phabricator.wikimedia.org/T443041#12520852 (10andrea.denisse) 05In progress→03Stalled [04:31:36] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1041.eqiad.wmnet [04:31:37] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1041.eqiad.wmnet [04:31:43] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1042.eqiad.wmnet [04:33:44] !log btullis@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1042.eqiad.wmnet [04:33:51] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2020.codfw.wmnet, repooling both afterwards [04:33:54] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [04:41:02] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1042.eqiad.wmnet [04:41:04] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1042.eqiad.wmnet [04:41:09] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1043.eqiad.wmnet [04:41:56] 06SRE, 10LDAP-Access-Requests: Grant Access to ldap/nda for Aroraakhil - https://phabricator.wikimedia.org/T443439#12520866 (10andrea.denisse) 05Open→03Resolved a:03andrea.denisse Hi!! I confirm @Aroraakhil is in the NDA spreasheet however, their NDA expires on Nov 10. Is it going to be extended @MGe... [04:43:06] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for Rmaung - https://phabricator.wikimedia.org/T443666#12520871 (10andrea.denisse) 05Open→03Invalid Hi @Rmaung , closing this task as we're processing these requests in T442756. [04:46:26] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for kartik - https://phabricator.wikimedia.org/T443182#12520877 (10andrea.denisse) 05Open→03Stalled [04:48:40] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for fab - https://phabricator.wikimedia.org/T442665#12520881 (10andrea.denisse) 05Open→03Stalled Hi @fkaelin , once you confirm you can access using the Yubikey we can remove the old key. [04:50:07] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12520885 (10andrea.denisse) 05Stalled→03In progress [04:57:05] (03PS1) 10Andrea Denisse: admin: Add cfoster to analytics-product-users [puppet] - 10https://gerrit.wikimedia.org/r/1363740 (https://phabricator.wikimedia.org/T442648) [05:05:08] (03PS1) 10Andrea Denisse: admin: Make Corbin Foster main approver of analytics-product-users [puppet] - 10https://gerrit.wikimedia.org/r/1363756 [05:08:11] !log denisse@cumin1004 DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Cfoster out of all services on: 4 hosts [05:09:26] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12520908 (10andrea.denisse) 05In progress→03Resolved a:03andrea.denisse Closing as resolved, feel f... [05:11:12] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1043.eqiad.wmnet [05:14:32] 06SRE, 06Data-Engineering, 10LDAP-Access-Requests: Grant Access to [analytics resources: Superset, Jupyter, Growthbook, Turnilo, Test Kitchen UI] for cfoster - https://phabricator.wikimedia.org/T443699#12520913 (10andrea.denisse) 05Open→03Resolved a:03andrea.denisse Hi @cfoster, you have the necess... [05:15:35] 06SRE, 10SRE-Access-Requests, 06Wikimedia Enterprise: Request for additional rights for SanchezA-WMF to manage analytics - https://phabricator.wikimedia.org/T443636#12520919 (10andrea.denisse) 05Open→03Stalled [05:15:56] (03PS1) 10Kevin Bazira: airflow-ml: add an envoy listener for the TTS section generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363777 (https://phabricator.wikimedia.org/T441987) [05:18:30] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1043.eqiad.wmnet [05:18:31] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1043.eqiad.wmnet [05:18:32] (03PS2) 10Kevin Bazira: airflow-ml: add an envoy listener for the TTS section generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363777 (https://phabricator.wikimedia.org/T441987) [05:18:37] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1044.eqiad.wmnet [05:19:52] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to releasers-mediawiki for osleger - https://phabricator.wikimedia.org/T443655#12520927 (10andrea.denisse) 05Open→03Stalled Hi, please use the SRE Access Request form for your request. https://phabricator.wikimedia.org/maniphest/task/edi... [05:20:08] 06SRE, 10SRE-Access-Requests, 06Wikimedia Enterprise: Request for additional rights for SanchezA-WMF to manage analytics - https://phabricator.wikimedia.org/T443636#12520931 (10andrea.denisse) Hi, please use the SRE Access Request form for your request. https://phabricator.wikimedia.org/maniphest/task/edit/f... [05:20:38] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH key for sbassett - https://phabricator.wikimedia.org/T443508#12520932 (10andrea.denisse) 05Open→03In progress [05:21:26] !log denisse@cumin1004 DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Cfoster out of all services on: 4 hosts [05:31:07] 06SRE: Document current expected SSH fingerprint for releases2003.codfw.wmnet - https://phabricator.wikimedia.org/T443537#12520935 (10andrea.denisse) 05Open→03In progress p:05Triage→03High I checked the fingerprints on releases2003 and they're the ones you see which is fine: ` +---------+---------+----... [05:35:19] 06SRE, 06Collaboration-Services: Document current expected SSH fingerprint for releases2003.codfw.wmnet - https://phabricator.wikimedia.org/T443537#12520939 (10andrea.denisse) [05:37:36] (03PS1) 10Gmodena: trafficserver: route WDQSv2 through REST Gateway [puppet] - 10https://gerrit.wikimedia.org/r/1363813 (https://phabricator.wikimedia.org/T443756) [05:48:51] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1044.eqiad.wmnet [05:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 94.48% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [05:55:39] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1044.eqiad.wmnet [05:55:40] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1044.eqiad.wmnet [05:55:40] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad [06:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [06:30:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.958% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [06:40:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [06:43:13] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12520993 (10Marostegui) @CWilliams-WMF let's repool? [06:48:29] !log marostegui@cumin1004 START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie [06:48:38] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: dbproxy2007 down - https://phabricator.wikimedia.org/T442357#12520994 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1004 for host dbprov2007.codfw.wmnet with OS trixie [06:49:25] !log marostegui@cumin1004 START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (push config only) [06:49:29] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (push config only) [07:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T0700) [07:00:05] marostegui, cezmunsta, and federico3: gettimeofday() says it's time for Primary database switchover. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T0700) [07:00:24] 10SRE-Access-Requests: Add FIDO-backed SSH keys for ecarg - https://phabricator.wikimedia.org/T443758 (10ecarg) 03NEW [07:01:31] !log marostegui@cumin1004 START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (remove config only) [07:01:36] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (remove config only) [07:02:24] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12521007 (10Marostegui) Thanks Papaul! I've also switched db1262 (T442800#12515024) [07:03:56] !log marostegui@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage [07:07:36] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage [07:10:00] FIRING: PingLossPercent: Blackbox probe packet loss 4.583% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [07:12:59] 10SRE-Access-Requests: Add FIDO-backed SSH keys for ecarg - https://phabricator.wikimedia.org/T443758#12521013 (10ecarg) [07:14:32] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for ecarg - https://phabricator.wikimedia.org/T443758#12521015 (10ecarg) [07:23:02] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS trixie [07:23:14] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: dbproxy2007 down - https://phabricator.wikimedia.org/T442357#12521032 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1004 for host dbprov2007.codfw.wmnet with OS trixie completed: - dbprov2007 (**PASS**) - Downtimed on Icing... [07:26:45] !log Drop thread_reaction tables T442289 [07:26:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:26:48] T442289: Drop thread_reaction tables from production - https://phabricator.wikimedia.org/T442289 [07:26:56] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for Southparkfan - https://phabricator.wikimedia.org/T443141#12521047 (10Southparkfan) 05Resolved→03Open Hello @andrea.denisse, thank you for processing this request! Unfortunately, I cannot log in yet. I couldn't find a changeset in Gerrit con... [07:30:52] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org [07:32:06] (03PS1) 10Andrea Denisse: admin: Grant southparkfan shell and deployment access [puppet] - 10https://gerrit.wikimedia.org/r/1363935 (https://phabricator.wikimedia.org/T443141) [07:34:23] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for Southparkfan - https://phabricator.wikimedia.org/T443141#12521061 (10andrea.denisse) 05Open→03Resolved >>! In T443141#12521047, @Southparkfan wrote: > Hello @andrea.denisse, thank you for processing this request! Un... [07:37:24] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org [07:44:07] (03PS1) 10DCausse: Restore the semantic highlighting batch ext [extensions/CirrusSearch] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363978 [07:44:23] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 05 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/CirrusSearch] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363978 (owner: 10DCausse) [07:53:21] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org [07:54:35] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for Southparkfan - https://phabricator.wikimedia.org/T443141#12521077 (10Southparkfan) >>! In T443141#12521061, @andrea.denisse wrote: >>>! In T443141#12521047, @Southparkfan wrote: >> Hello @andrea.denisse, thank you for p... [07:56:37] !log installing expat security updates [07:56:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:59:32] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org [08:00:04] Amir1, urbanecm, and awight: UTC morning backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T0800). Please do the needful. [08:00:04] dcausse: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [08:00:14] o/ [08:00:18] I can deploy [08:00:46] (03PS1) 10Marostegui: mariadb: Decommission clouddb1014 [puppet] - 10https://gerrit.wikimedia.org/r/1363988 (https://phabricator.wikimedia.org/T441395) [08:01:54] !log marostegui@cumin1004 START - Cookbook sre.mysql.decommission [08:02:04] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dcausse@deploy2003 using scap backport" [extensions/CirrusSearch] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363978 (owner: 10DCausse) [08:02:07] !log marostegui@cumin1004 START - Cookbook sre.hosts.decommission for hosts clouddb1014.eqiad.wmnet [08:02:27] (03CR) 10Marostegui: [C:03+2] mariadb: Decommission clouddb1014 [puppet] - 10https://gerrit.wikimedia.org/r/1363988 (https://phabricator.wikimedia.org/T441395) (owner: 10Marostegui) [08:03:19] (03Merged) 10jenkins-bot: Restore the semantic highlighting batch ext [extensions/CirrusSearch] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363978 (owner: 10DCausse) [08:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:06:19] !log dcausse@deploy2003 Started scap sync-world: Backport for [[gerrit:1363978|Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) [08:07:05] (03CR) 10Arnaudb: "we could probably do that. First I'd like to close the 443 port on the non cloud instance, happy to follow up after that!" [puppet] - 10https://gerrit.wikimedia.org/r/1357232 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [08:07:21] (03CR) 10Aqu: [C:03+1] "Thanks for reclaiming disk space." [puppet] - 10https://gerrit.wikimedia.org/r/1360975 (https://phabricator.wikimedia.org/T438056) (owner: 10Xcollazo) [08:07:48] !log marostegui@cumin1004 START - Cookbook sre.dns.netbox [08:09:06] (03CR) 10Arnaudb: "great! thanks for jumping ahead and merging and testing that change ❤️" [puppet] - 10https://gerrit.wikimedia.org/r/1308081 (https://phabricator.wikimedia.org/T431398) (owner: 10Arnaudb) [08:10:56] !log dcausse@deploy2003 dcausse: Backport for [[gerrit:1363978|Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:11:12] (03PS1) 10Marostegui: check_private_data_report: Remove clouddb1014 [puppet] - 10https://gerrit.wikimedia.org/r/1363993 (https://phabricator.wikimedia.org/T441395) [08:11:31] !log marostegui@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004" [08:11:44] !log marostegui@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004" [08:11:45] !log marostegui@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [08:11:45] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1014.eqiad.wmnet [08:11:48] !log marostegui@cumin1004 Removing clouddb1014 from zarcillo T441395 [08:11:51] T441395: decommission clouddb1014.eqiad.wmnet - https://phabricator.wikimedia.org/T441395 [08:11:53] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) [08:11:58] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission clouddb1014.eqiad.wmnet - https://phabricator.wikimedia.org/T441395#12521119 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by marostegui@cumin1004 for hosts: `clouddb1014.eqiad.wmnet` - clouddb1014.eqiad.wmnet (**PASS**)... [08:11:59] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission clouddb1014.eqiad.wmnet - https://phabricator.wikimedia.org/T441395#12521120 (10ops-monitoring-bot) clouddb1014 has been deleted from zarcillo [08:12:03] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission clouddb1014.eqiad.wmnet - https://phabricator.wikimedia.org/T441395#12521122 (10ops-monitoring-bot) clouddb1014 has been decommissioned by Data Persistence [08:12:04] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission clouddb1014.eqiad.wmnet - https://phabricator.wikimedia.org/T441395#12521123 (10ops-monitoring-bot) a:05Marostegui→03None This host is ready for DC-Ops to decommission [08:12:39] (03CR) 10Marostegui: [C:03+2] check_private_data_report: Remove clouddb1014 [puppet] - 10https://gerrit.wikimedia.org/r/1363993 (https://phabricator.wikimedia.org/T441395) (owner: 10Marostegui) [08:14:20] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 1 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [08:14:52] !log dcausse@deploy2003 dcausse: Continuing with deployment [08:17:22] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [08:22:23] !log dcausse@deploy2003 Finished scap sync-world: Backport for [[gerrit:1363978|Restore the semantic highlighting batch ext]] (1.48.0-wmf.3) (duration: 16m 04s) [08:25:42] (03PS1) 10Muehlenhoff: thumbor-plugins: Rebuild against latest security updates in Trixie [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1364007 [08:27:00] (03PS2) 10Fabfur: profile::cache,hiera: enable kapow score everywhere and log the var [puppet] - 10https://gerrit.wikimedia.org/r/1363149 (https://phabricator.wikimedia.org/T435799) [08:29:24] 10ops-eqiad, 06DC-Ops: Unresponsive management for backup1010.mgmt:22 - https://phabricator.wikimedia.org/T443762 (10phaultfinder) 03NEW [08:30:27] 10ops-eqiad, 06DC-Ops: Unresponsive management for cirrussearch1090.mgmt:22 - https://phabricator.wikimedia.org/T443764 (10phaultfinder) 03NEW [08:30:28] 10ops-eqiad, 06DC-Ops: Unresponsive management for an-worker1181.mgmt:22 - https://phabricator.wikimedia.org/T443763 (10phaultfinder) 03NEW [08:30:40] (03PS1) 10Dpogorzelski: admin_ng: add KEDA, on ml-staging-codfw only [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364011 (https://phabricator.wikimedia.org/T436650) [08:31:30] 10ops-eqiad, 06DC-Ops: Unresponsive management for dse-k8s-worker1005.mgmt:22 - https://phabricator.wikimedia.org/T443767 (10phaultfinder) 03NEW [08:31:30] 10ops-eqiad, 06DC-Ops: Unresponsive management for an-worker1235.mgmt:22 - https://phabricator.wikimedia.org/T443766 (10phaultfinder) 03NEW [08:36:44] (03PS1) 10Brouberol: mediawiki-dumps-legacy/dump-toolbox: update image tag [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364016 [08:40:45] 06SRE, 10LDAP-Access-Requests: Grant Access to ldap/nda for Aroraakhil - https://phabricator.wikimedia.org/T443439#12521208 (10MGerlach) >>! In T443439#12520867, @andrea.denisse wrote: > Hi!! I confirm @Aroraakhil is in the NDA spreasheet however, their NDA expires on Nov 10. Is it going to be extended @MG... [08:41:43] (03CR) 10Brouberol: [C:03+2] mediawiki-dumps-legacy/dump-toolbox: update image tag [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364016 (owner: 10Brouberol) [08:42:15] (03PS2) 10Dpogorzelski: admin_ng: add KEDA, on ml-staging-codfw only [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364011 (https://phabricator.wikimedia.org/T436650) [08:42:15] (03PS1) 10Dpogorzelski: keda: add the upstream KEDA 2.21.0 chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364019 (https://phabricator.wikimedia.org/T436650) [08:42:19] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply [08:42:30] !log restarting FPM on phab1005 to pick up Expat security updates [08:42:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:44:11] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply [08:48:04] (03PS1) 10Gehel: admin: adding Leon Ziemba as contact for sd [puppet] - 10https://gerrit.wikimedia.org/r/1364022 (https://phabricator.wikimedia.org/T442756) [08:48:29] (03CR) 10Klausman: [C:03+1] admin_ng: add KEDA, on ml-staging-codfw only [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364011 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [09:00:05] jnuche and jnuche: How many deployers does it take to do MediaWiki train deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T0900). [09:00:10] morning, I'll deploy the train in a bit [09:02:54] (03PS1) 10TrainBranchBot: group2 to 1.48.0-wmf.3 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364042 (https://phabricator.wikimedia.org/T442221) [09:02:57] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by jnuche@deploy2003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364042 (https://phabricator.wikimedia.org/T442221) (owner: 10TrainBranchBot) [09:03:47] (03Merged) 10jenkins-bot: group2 to 1.48.0-wmf.3 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364042 (https://phabricator.wikimedia.org/T442221) (owner: 10TrainBranchBot) [09:08:07] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user nettrom [09:09:11] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user nettrom [09:10:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [09:12:48] !log jnuche@deploy2003 rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.3 refs T442221 [09:12:51] T442221: 1.48.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T442221 [09:16:32] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user kgraessle [09:16:57] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user kgraessle [09:16:58] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy [09:17:19] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy [09:17:20] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user dmaza [09:17:26] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12521294 (10CWilliams-WMF) @Marostegui it still seems to be catching up on replication [09:17:29] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user dmaza [09:17:30] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user ksiebert [09:17:38] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user ksiebert [09:17:39] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user atieno [09:17:47] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user atieno [09:17:48] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user harrayo-wmf [09:17:56] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user harrayo-wmf [09:18:10] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12521296 (10Marostegui) Yeah, I meant once ready :) [09:21:44] (03CR) 10Muehlenhoff: [C:03+1] "Key has been verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1361030 (https://phabricator.wikimedia.org/T442817) (owner: 10Mstyles) [09:22:08] (03CR) 10Muehlenhoff: [C:03+2] admin: Remove private enclave key and add yubikey backup for mstyles [puppet] - 10https://gerrit.wikimedia.org/r/1361030 (https://phabricator.wikimedia.org/T442817) (owner: 10Mstyles) [09:23:52] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy [09:24:01] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy [09:26:22] (03CR) 10Jelto: [C:03+2] deployment_server: Add editcheck-headless to deploy_service catalog [puppet] - 10https://gerrit.wikimedia.org/r/1363348 (https://phabricator.wikimedia.org/T430712) (owner: 10Ottomata) [09:29:13] (03CR) 10JMeybohm: [C:04-1] "Please import the upstream chart verbatim first and stack additions/changes on top so they can be properly reviewed and tracked. See https" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364019 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [09:31:01] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user suecarmol [09:31:10] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user suecarmol [09:31:11] (03PS1) 10Muehlenhoff: Add FIDO key for akhatun [puppet] - 10https://gerrit.wikimedia.org/r/1364070 (https://phabricator.wikimedia.org/T442758) [09:33:25] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for akhatun [puppet] - 10https://gerrit.wikimedia.org/r/1364070 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [09:35:31] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user andreawest [09:35:52] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user andreawest [09:35:53] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jmoore111 [09:36:01] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jmoore111 [09:36:02] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jynus [09:36:11] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jynus [09:37:51] (03CR) 10Ladsgroup: [C:03+2] thumbor-plugins: Rebuild against latest security updates in Trixie [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1364007 (owner: 10Muehlenhoff) [09:41:00] (03Merged) 10jenkins-bot: thumbor-plugins: Rebuild against latest security updates in Trixie [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1364007 (owner: 10Muehlenhoff) [09:43:46] (03PS1) 10Muehlenhoff: Add FIDO key for andrewtavis-wmde [puppet] - 10https://gerrit.wikimedia.org/r/1364089 (https://phabricator.wikimedia.org/T442763) [09:44:16] (03PS1) 10Marostegui: table-catalog: Remove globalnames [puppet] - 10https://gerrit.wikimedia.org/r/1364090 (https://phabricator.wikimedia.org/T442241) [09:44:37] (03CR) 10CI reject: [V:04-1] Add FIDO key for andrewtavis-wmde [puppet] - 10https://gerrit.wikimedia.org/r/1364089 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [09:45:33] (03CR) 10Ladsgroup: [C:03+1] table-catalog: Remove globalnames [puppet] - 10https://gerrit.wikimedia.org/r/1364090 (https://phabricator.wikimedia.org/T442241) (owner: 10Marostegui) [09:45:51] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org [09:47:33] (03CR) 10Marostegui: [C:03+2] table-catalog: Remove globalnames [puppet] - 10https://gerrit.wikimedia.org/r/1364090 (https://phabricator.wikimedia.org/T442241) (owner: 10Marostegui) [09:48:00] (03PS2) 10Muehlenhoff: Add FIDO key for andrewtavis-wmde [puppet] - 10https://gerrit.wikimedia.org/r/1364089 (https://phabricator.wikimedia.org/T442763) [09:49:08] (03Abandoned) 10Zabe: tables-catalog: Drop globalnames table [puppet] - 10https://gerrit.wikimedia.org/r/1358274 (https://phabricator.wikimedia.org/T442241) (owner: 10Zabe) [09:49:35] (03Abandoned) 10Fabfur: Revert "profile:haproxy: kapow score setup" [puppet] - 10https://gerrit.wikimedia.org/r/1363054 (owner: 10Fabfur) [09:49:40] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org [09:50:37] (03CR) 10Muehlenhoff: [C:03+1] "Verified out of band." [puppet] - 10https://gerrit.wikimedia.org/r/1364089 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [09:50:39] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for andrewtavis-wmde [puppet] - 10https://gerrit.wikimedia.org/r/1364089 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [09:50:40] (03PS1) 10Marostegui: eqiad.yaml: Remove clouddb1015 [puppet] - 10https://gerrit.wikimedia.org/r/1364098 (https://phabricator.wikimedia.org/T443768) [09:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 95.66% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [09:53:19] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ldap-replica1005.wikimedia.org [09:53:29] (03CR) 10FNegri: [C:03+1] eqiad.yaml: Remove clouddb1015 [puppet] - 10https://gerrit.wikimedia.org/r/1364098 (https://phabricator.wikimedia.org/T443768) (owner: 10Marostegui) [09:53:49] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance [09:53:51] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97335 and previous config saved to /var/cache/conftool/dbconfig/20261105-095350-ladsgroup.json [09:57:11] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1005.wikimedia.org [10:02:10] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97336 and previous config saved to /var/cache/conftool/dbconfig/20261105-100209-ladsgroup.json [10:02:16] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ldap-replica1006.wikimedia.org [10:03:17] (03CR) 10Marostegui: [C:03+2] eqiad.yaml: Remove clouddb1015 [puppet] - 10https://gerrit.wikimedia.org/r/1364098 (https://phabricator.wikimedia.org/T443768) (owner: 10Marostegui) [10:04:25] !log cwilliams@cumin1004 START - Cookbook sre.mysql.pool pool db1262: Maintenance [10:04:29] !log cwilliams@cumin1004 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1262: Maintenance [10:06:17] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1006.wikimedia.org [10:06:21] (03PS1) 10CWilliams: db1262: enabling notifications [puppet] - 10https://gerrit.wikimedia.org/r/1364118 (https://phabricator.wikimedia.org/T442800) [10:08:09] (03PS2) 10CWilliams: db1262: enabling notifications [puppet] - 10https://gerrit.wikimedia.org/r/1364118 (https://phabricator.wikimedia.org/T442800) [10:09:13] (03CR) 10CWilliams: [C:03+2] db1262: enabling notifications [puppet] - 10https://gerrit.wikimedia.org/r/1364118 (https://phabricator.wikimedia.org/T442800) (owner: 10CWilliams) [10:12:12] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97337 and previous config saved to /var/cache/conftool/dbconfig/20261105-101211-ladsgroup.json [10:13:20] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [10:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [10:16:37] 06SRE, 06Infrastructure-Foundations, 10netops: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12521571 (10cmooney) [10:16:40] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12521570 (10cmooney) [10:16:58] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sretest1006.eqiad.wmnet with reason: test on other vlan [10:17:11] 06SRE, 06Infrastructure-Foundations, 10netops: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12521573 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=351d61a3-389f-44b4-9fe8-5fea6f1228f9) set by cmooney@cumin1004 for 1:00:00 on 1 host(s) and the... [10:17:37] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364126 [10:18:44] (03PS1) 10Marostegui: db2250: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1364127 (https://phabricator.wikimedia.org/T424541) [10:19:04] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2250.codfw.wmnet with reason: Reimage [10:20:00] (03CR) 10Marostegui: [C:03+2] db2250: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1364127 (https://phabricator.wikimedia.org/T424541) (owner: 10Marostegui) [10:20:42] !log marostegui@cumin1004 START - Cookbook sre.hosts.reimage for host db2250.codfw.wmnet with OS trixie [10:21:27] !log marostegui@cumin1004 START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (push config only) [10:21:31] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (push config only) [10:21:47] (03PS1) 10Muehlenhoff: Add FIDO key for bd808 [puppet] - 10https://gerrit.wikimedia.org/r/1364131 (https://phabricator.wikimedia.org/T440887) [10:22:13] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97338 and previous config saved to /var/cache/conftool/dbconfig/20261105-102213-ladsgroup.json [10:25:08] (03CR) 10Muehlenhoff: "Verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1364131 (https://phabricator.wikimedia.org/T440887) (owner: 10Muehlenhoff) [10:25:16] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for bd808 [puppet] - 10https://gerrit.wikimedia.org/r/1364131 (https://phabricator.wikimedia.org/T440887) (owner: 10Muehlenhoff) [10:26:02] !log cmooney@cumin1004 START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie [10:27:03] !log cmooney@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only) [10:27:07] !log cmooney@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only) [10:28:39] (03CR) 10Muehlenhoff: admin: adding Leon Ziemba as contact for sd (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1364022 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [10:30:44] (03PS1) 10Muehlenhoff: Record extended MOU dates for aarora and ankita97531 [puppet] - 10https://gerrit.wikimedia.org/r/1364145 [10:31:15] (03PS1) 10Tchanders: admin: Add FIDO SSH key for tchanders [puppet] - 10https://gerrit.wikimedia.org/r/1364146 (https://phabricator.wikimedia.org/T442758) [10:32:04] !log installing libgit2 security updates [10:32:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:32:06] !log marostegui@cumin1004 START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (remove config only) [10:32:11] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (remove config only) [10:32:15] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97339 and previous config saved to /var/cache/conftool/dbconfig/20261105-103215-ladsgroup.json [10:32:33] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance [10:32:35] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1192 (T441633)', diff saved to https://phabricator.wikimedia.org/P97340 and previous config saved to /var/cache/conftool/dbconfig/20261105-103234-ladsgroup.json [10:32:46] FIRING: [2x] CertAlmostExpired: Certificate for service kubestagemaster2005:6443 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#kubestagemaster2005:6443 - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:33:49] (03PS1) 10Btullis: Update ownership details of dumps distribution servers to Data Platform [puppet] - 10https://gerrit.wikimedia.org/r/1364149 (https://phabricator.wikimedia.org/T441202) [10:33:51] (03PS1) 10Btullis: Deprecate and de-populate the dumps-roots group [puppet] - 10https://gerrit.wikimedia.org/r/1364150 (https://phabricator.wikimedia.org/T440471) [10:34:39] (03PS1) 10Muehlenhoff: Add library hint for libgit2 [puppet] - 10https://gerrit.wikimedia.org/r/1364152 [10:34:40] !log marostegui@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on db2250.codfw.wmnet with reason: host reimage [10:35:03] !log cwilliams@cumin1004 START - Cookbook sre.mysql.pool pool db1262: Maintenance [10:35:04] (03CR) 10Btullis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1364150 (https://phabricator.wikimedia.org/T440471) (owner: 10Btullis) [10:35:16] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12521631 (10ops-monitoring-bot) Starting pool of db1262 by cwilliams@cumin1004: Maintenance [10:36:05] (03CR) 10Btullis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1364149 (https://phabricator.wikimedia.org/T441202) (owner: 10Btullis) [10:38:07] (03CR) 10Muehlenhoff: [C:03+2] Add library hint for libgit2 [puppet] - 10https://gerrit.wikimedia.org/r/1364152 (owner: 10Muehlenhoff) [10:38:26] !log btullis@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-codfw [10:38:29] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2250.codfw.wmnet with reason: host reimage [10:38:29] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2001.codfw.wmnet [10:38:31] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2001.codfw.wmnet [10:40:48] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1192 (T441633)', diff saved to https://phabricator.wikimedia.org/P97342 and previous config saved to /var/cache/conftool/dbconfig/20261105-104047-ladsgroup.json [10:43:25] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2001.codfw.wmnet [10:43:27] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2001.codfw.wmnet [10:43:33] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2002.codfw.wmnet [10:43:34] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2002.codfw.wmnet [10:46:18] btullis@cumin1004 roll-restart-reboot-brokers (PID 796320) is awaiting input [10:47:29] (03PS2) 10Btullis: Update ownership details of dumps distribution servers to Data Platform [puppet] - 10https://gerrit.wikimedia.org/r/1364149 (https://phabricator.wikimedia.org/T441202) [10:47:29] (03PS2) 10Btullis: Deprecate and de-populate the dumps-roots group [puppet] - 10https://gerrit.wikimedia.org/r/1364150 (https://phabricator.wikimedia.org/T440471) [10:47:47] (03CR) 10Btullis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1364149 (https://phabricator.wikimedia.org/T441202) (owner: 10Btullis) [10:48:38] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2002.codfw.wmnet [10:48:40] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2002.codfw.wmnet [10:48:40] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-codfw [10:49:05] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12521677 (10Marostegui) @wiki_willy @VRiley-WMF can we push Dell again for this? [10:50:50] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97344 and previous config saved to /var/cache/conftool/dbconfig/20261105-105049-ladsgroup.json [10:51:28] (03CR) 10Atsuko: [C:03+1] airflow-ml: add an envoy listener for the TTS section generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363777 (https://phabricator.wikimedia.org/T441987) (owner: 10Kevin Bazira) [10:51:38] !log btullis@cumin1004 START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad [10:51:53] !log btullis@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-eqiad [10:51:56] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1001.eqiad.wmnet [10:51:57] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1001.eqiad.wmnet [10:52:42] (03PS1) 10Ladsgroup: thumbor: Bump to head [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364174 [10:56:49] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1001.eqiad.wmnet [10:56:51] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1001.eqiad.wmnet [10:56:56] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1002.eqiad.wmnet [10:56:57] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1002.eqiad.wmnet [10:57:23] RECOVERY - Host an-worker1154 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [10:57:43] (03PS1) 10Kamila Součková: Enroll 5% of client sessions in PHP 8.5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364180 (https://phabricator.wikimedia.org/T441689) [10:57:59] (03CR) 10Ladsgroup: [C:03+2] thumbor: Bump to head [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364174 (owner: 10Ladsgroup) [10:58:33] (03CR) 10Kamila Součková: "logs show only deprecation warnings, no new bugs, so OK to continue" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364180 (https://phabricator.wikimedia.org/T441689) (owner: 10Kamila Součková) [10:59:36] (03Merged) 10jenkins-bot: thumbor: Bump to head [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364174 (owner: 10Ladsgroup) [11:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1100) [11:00:52] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97346 and previous config saved to /var/cache/conftool/dbconfig/20261105-110051-ladsgroup.json [11:01:48] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1002.eqiad.wmnet [11:01:50] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1002.eqiad.wmnet [11:01:50] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-eqiad [11:04:38] !log ladsgroup@deploy2003 helmfile [staging] START helmfile.d/services/thumbor: apply [11:05:48] !log ladsgroup@deploy2003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [11:06:22] (03CR) 10Muehlenhoff: "Looks good, one note inline" [puppet] - 10https://gerrit.wikimedia.org/r/1364150 (https://phabricator.wikimedia.org/T440471) (owner: 10Btullis) [11:08:20] (03PS3) 10Btullis: Deprecate and de-populate the dumps-roots group [puppet] - 10https://gerrit.wikimedia.org/r/1364150 (https://phabricator.wikimedia.org/T440471) [11:09:06] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2250.codfw.wmnet with OS trixie [11:09:08] (03CR) 10Btullis: Deprecate and de-populate the dumps-roots group (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1364150 (https://phabricator.wikimedia.org/T440471) (owner: 10Btullis) [11:10:05] (03CR) 10Atsuko: [C:03+2] airflow-ml: add an envoy listener for the TTS section generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363777 (https://phabricator.wikimedia.org/T441987) (owner: 10Kevin Bazira) [11:10:54] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1192 (T441633)', diff saved to https://phabricator.wikimedia.org/P97348 and previous config saved to /var/cache/conftool/dbconfig/20261105-111053-ladsgroup.json [11:11:12] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance [11:11:13] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1209 (T441633)', diff saved to https://phabricator.wikimedia.org/P97349 and previous config saved to /var/cache/conftool/dbconfig/20261105-111113-ladsgroup.json [11:11:14] (03CR) 10Atsuko: [C:03+2] "Test plan" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363777 (https://phabricator.wikimedia.org/T441987) (owner: 10Kevin Bazira) [11:11:38] (03Merged) 10jenkins-bot: airflow-ml: add an envoy listener for the TTS section generator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363777 (https://phabricator.wikimedia.org/T441987) (owner: 10Kevin Bazira) [11:12:07] !log ladsgroup@deploy2003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [11:13:37] !log ladsgroup@deploy2003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [11:14:38] !log ladsgroup@deploy2003 helmfile [codfw] START helmfile.d/services/thumbor: apply [11:15:12] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply [11:15:26] (03CR) 10Tiziano Fogli: [C:03+1] karma: strip some useless labels from panel display [puppet] - 10https://gerrit.wikimedia.org/r/1311029 (https://phabricator.wikimedia.org/T431980) (owner: 10Hnowlan) [11:16:14] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply [11:16:58] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [11:17:58] !log ladsgroup@deploy2003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [11:18:20] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1209 (T441633)', diff saved to https://phabricator.wikimedia.org/P97350 and previous config saved to /var/cache/conftool/dbconfig/20261105-111819-ladsgroup.json [11:18:22] (03PS1) 10Muehlenhoff: Add FIDO key for ifrahkh [puppet] - 10https://gerrit.wikimedia.org/r/1364200 (https://phabricator.wikimedia.org/T442763) [11:19:24] (03PS1) 10Marostegui: Revert "db2250: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1364202 [11:20:12] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Maintenance [11:20:19] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12521765 (10ops-monitoring-bot) Completed pooling of db1262 by cwilliams@cumin1004: Maintenance [11:20:21] (03CR) 10Marostegui: [C:03+2] Revert "db2250: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1364202 (owner: 10Marostegui) [11:22:26] cmooney@cumin1004 netbox (PID 803059) is awaiting input [11:22:33] !log btullis@cumin1004 END (ERROR) - Cookbook sre.hadoop.reboot-workers (exit_code=97) for Hadoop analytics cluster [11:24:38] (03PS1) 10STran: Enable SuggestedInvestigations case generation on nlwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364209 (https://phabricator.wikimedia.org/T443646) [11:26:32] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for ifrahkh [puppet] - 10https://gerrit.wikimedia.org/r/1364200 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [11:26:38] (03PS1) 10Vgutierrez: varnish: Drop external requests to ESI fragments [puppet] - 10https://gerrit.wikimedia.org/r/1364210 (https://phabricator.wikimedia.org/T443704) [11:26:42] (03PS2) 10Atsuko: airflow-dumps: live-debug to a separate config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364192 (https://phabricator.wikimedia.org/T443252) [11:27:08] (03CR) 10Sergio Gimeno: [C:03+1] "Dependency aside this looks good to me" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361861 (https://phabricator.wikimedia.org/T443176) (owner: 10Michael Große) [11:27:34] !log btullis@cumin1004 END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad [11:27:37] (03CR) 10Majavah: [C:03+2] wmcs: use bd808 hardware-backed ssh key [puppet] - 10https://gerrit.wikimedia.org/r/1363482 (https://phabricator.wikimedia.org/T443002) (owner: 10BryanDavis) [11:27:46] (03CR) 10Atsuko: [C:03+2] airflow-dumps: live-debug to a separate config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364192 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [11:27:55] (03CR) 10STran: "Tables will be created as part of this" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364209 (https://phabricator.wikimedia.org/T443646) (owner: 10STran) [11:28:22] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97352 and previous config saved to /var/cache/conftool/dbconfig/20261105-112821-ladsgroup.json [11:28:27] (03PS2) 10Vgutierrez: varnish: Drop external requests to ESI fragments [puppet] - 10https://gerrit.wikimedia.org/r/1364210 (https://phabricator.wikimedia.org/T443704) [11:29:02] (03Merged) 10jenkins-bot: airflow-dumps: live-debug to a separate config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364192 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [11:31:22] (03CR) 10Mszwarc: [C:03+1] "Good to deploy once tables are created" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364209 (https://phabricator.wikimedia.org/T443646) (owner: 10STran) [11:31:47] (03CR) 10Btullis: [C:03+1] airflow-dumps: live-debug to a separate config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364192 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [11:33:43] (03PS1) 10Muehlenhoff: Add FIDO key for nicholusmuwonge [puppet] - 10https://gerrit.wikimedia.org/r/1364214 [11:33:56] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 05 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364209 (https://phabricator.wikimedia.org/T443646) (owner: 10STran) [11:34:39] (03CR) 10CI reject: [V:04-1] Add FIDO key for nicholusmuwonge [puppet] - 10https://gerrit.wikimedia.org/r/1364214 (owner: 10Muehlenhoff) [11:36:16] (03PS3) 10Vgutierrez: varnish: Drop external requests to ESI fragments [puppet] - 10https://gerrit.wikimedia.org/r/1364210 (https://phabricator.wikimedia.org/T443704) [11:38:24] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97353 and previous config saved to /var/cache/conftool/dbconfig/20261105-113823-ladsgroup.json [11:38:34] (03PS2) 10Muehlenhoff: Add FIDO key for nicholusmuwonge [puppet] - 10https://gerrit.wikimedia.org/r/1364214 (https://phabricator.wikimedia.org/T442763) [11:38:57] (03PS1) 10Daniel Kertesz: lvs: add prometheus::node_sysctl to collect rp_filter metrics [puppet] - 10https://gerrit.wikimedia.org/r/1364220 (https://phabricator.wikimedia.org/T443771) [11:38:58] (03CR) 10Ladsgroup: "out of abundance of caution, I'm going to run php package update on all stat machines first (even though we did it on the same day) and th" [puppet] - 10https://gerrit.wikimedia.org/r/1362194 (https://phabricator.wikimedia.org/T443525) (owner: 10WMDE-leszek) [11:39:32] (03PS1) 10Vgutierrez: admin: Add FIDO-backed SSH key for ecarg [puppet] - 10https://gerrit.wikimedia.org/r/1364221 (https://phabricator.wikimedia.org/T443758) [11:39:33] (03CR) 10Daniel Kertesz: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1364220 (https://phabricator.wikimedia.org/T443771) (owner: 10Daniel Kertesz) [11:39:54] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for ecarg - https://phabricator.wikimedia.org/T443758#12521791 (10Vgutierrez) 05Open→03In progress p:05Triage→03Medium [11:41:42] (03PS2) 10STran: Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364209 (https://phabricator.wikimedia.org/T443646) [11:42:00] (03PS3) 10STran: Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364209 (https://phabricator.wikimedia.org/T443646) [11:43:14] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for nicholusmuwonge [puppet] - 10https://gerrit.wikimedia.org/r/1364214 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [11:45:30] (03CR) 10Mszwarc: [C:03+1] Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364209 (https://phabricator.wikimedia.org/T443646) (owner: 10STran) [11:45:56] (03CR) 10Vgutierrez: [C:03+1] lvs: add prometheus::node_sysctl to collect rp_filter metrics [puppet] - 10https://gerrit.wikimedia.org/r/1364220 (https://phabricator.wikimedia.org/T443771) (owner: 10Daniel Kertesz) [11:47:07] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004" [11:47:09] !log cmooney@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only) [11:47:10] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004" [11:47:11] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:47:13] !log cmooney@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only) [11:47:13] !log cmooney@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1006.eqiad.wmnet with OS trixie [11:47:56] (03PS1) 10Southparkfan: LabsServices: Migrate ms-fe to service RR [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364235 (https://phabricator.wikimedia.org/T436476) [11:48:25] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1209 (T441633)', diff saved to https://phabricator.wikimedia.org/P97354 and previous config saved to /var/cache/conftool/dbconfig/20261105-114825-ladsgroup.json [11:48:43] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance [11:48:46] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1214 (T441633)', diff saved to https://phabricator.wikimedia.org/P97355 and previous config saved to /var/cache/conftool/dbconfig/20261105-114845-ladsgroup.json [11:49:12] !log cmooney@cumin1004 START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors [11:49:15] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors [11:49:19] (03CR) 10Daniel Kertesz: [C:03+2] lvs: add prometheus::node_sysctl to collect rp_filter metrics [puppet] - 10https://gerrit.wikimedia.org/r/1364220 (https://phabricator.wikimedia.org/T443771) (owner: 10Daniel Kertesz) [11:50:05] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 05 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364235 (https://phabricator.wikimedia.org/T436476) (owner: 10Southparkfan) [11:50:08] (03CR) 10Vgutierrez: "varnishtests are happy: `0 tests failed, 0 tests skipped, 45 tests passed`" [puppet] - 10https://gerrit.wikimedia.org/r/1364210 (https://phabricator.wikimedia.org/T443704) (owner: 10Vgutierrez) [11:51:56] !log cmooney@cumin1004 START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie [11:52:04] !log cmooney@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only) [11:52:07] !log cmooney@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only) [11:52:52] (03CR) 10Daniel Kertesz: [C:03+1] varnish: Drop external requests to ESI fragments [puppet] - 10https://gerrit.wikimedia.org/r/1364210 (https://phabricator.wikimedia.org/T443704) (owner: 10Vgutierrez) [11:55:50] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1214 (T441633)', diff saved to https://phabricator.wikimedia.org/P97356 and previous config saved to /var/cache/conftool/dbconfig/20261105-115549-ladsgroup.json [11:56:37] (03CR) 10Muehlenhoff: [C:03+1] "Key has been verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1364146 (https://phabricator.wikimedia.org/T442758) (owner: 10Tchanders) [11:56:41] (03CR) 10Muehlenhoff: [C:03+2] admin: Add FIDO SSH key for tchanders [puppet] - 10https://gerrit.wikimedia.org/r/1364146 (https://phabricator.wikimedia.org/T442758) (owner: 10Tchanders) [11:58:08] (03CR) 10Vgutierrez: [C:03+2] varnish: Drop external requests to ESI fragments [puppet] - 10https://gerrit.wikimedia.org/r/1364210 (https://phabricator.wikimedia.org/T443704) (owner: 10Vgutierrez) [11:59:37] (03PS1) 10Muehlenhoff: Remove legacy key for gmodena [puppet] - 10https://gerrit.wikimedia.org/r/1364254 (https://phabricator.wikimedia.org/T442758) [12:00:06] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:00:16] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:00:20] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:00:30] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:00:33] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [12:00:46] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [12:01:27] (03CR) 10Mforns: profile::cache,hiera: enable kapow score everywhere and log the var (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363149 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [12:01:53] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy key for gmodena [puppet] - 10https://gerrit.wikimedia.org/r/1364254 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [12:03:51] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1364261 (owner: 10L10n-bot) [12:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:05:06] (03PS2) 10Dwisehaupt: Add FIDO SSH key for eileen [puppet] - 10https://gerrit.wikimedia.org/r/1363505 [12:05:51] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97357 and previous config saved to /var/cache/conftool/dbconfig/20261105-120551-ladsgroup.json [12:06:00] (03CR) 10CI reject: [V:04-1] Add FIDO SSH key for eileen [puppet] - 10https://gerrit.wikimedia.org/r/1363505 (owner: 10Dwisehaupt) [12:07:12] !log cmooney@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only) [12:07:16] !log cmooney@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only) [12:09:20] !log cmooney@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage [12:10:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [12:11:35] !log installing alsa-lib security updates [12:11:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:12:05] (03PS3) 10Muehlenhoff: Add FIDO SSH key for eileen [puppet] - 10https://gerrit.wikimedia.org/r/1363505 (https://phabricator.wikimedia.org/T442759) (owner: 10Dwisehaupt) [12:13:47] !log cmooney@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage [12:15:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [12:15:53] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97358 and previous config saved to /var/cache/conftool/dbconfig/20261105-121552-ladsgroup.json [12:17:23] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [12:20:25] (03PS1) 10Muehlenhoff: Update key for mpostoronca [puppet] - 10https://gerrit.wikimedia.org/r/1364289 (https://phabricator.wikimedia.org/T442758) [12:21:48] (03CR) 10Muehlenhoff: [C:03+2] Update key for mpostoronca [puppet] - 10https://gerrit.wikimedia.org/r/1364289 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [12:22:26] (03CR) 10Klausman: [C:03+1] Deprecate and de-populate the dumps-roots group [puppet] - 10https://gerrit.wikimedia.org/r/1364150 (https://phabricator.wikimedia.org/T440471) (owner: 10Btullis) [12:24:17] (03CR) 10Klausman: [C:03+1] Update ownership details of dumps distribution servers to Data Platform [puppet] - 10https://gerrit.wikimedia.org/r/1364149 (https://phabricator.wikimedia.org/T441202) (owner: 10Btullis) [12:25:55] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1214 (T441633)', diff saved to https://phabricator.wikimedia.org/P97359 and previous config saved to /var/cache/conftool/dbconfig/20261105-122554-ladsgroup.json [12:26:13] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance [12:26:15] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1226 (T441633)', diff saved to https://phabricator.wikimedia.org/P97360 and previous config saved to /var/cache/conftool/dbconfig/20261105-122614-ladsgroup.json [12:26:58] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for ecarg - https://phabricator.wikimedia.org/T443758#12521963 (10Vgutierrez) 05In progress→03Stalled waiting for OOB key confirmation [12:29:20] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1364150 (https://phabricator.wikimedia.org/T440471) (owner: 10Btullis) [12:31:22] !log cmooney@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie [12:34:25] jouncebot: nowandnext [12:34:25] No deployments scheduled for the next 0 hour(s) and 25 minute(s) [12:34:25] In 0 hour(s) and 25 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1300) [12:34:37] (03CR) 10Btullis: [C:03+2] Update ownership details of dumps distribution servers to Data Platform [puppet] - 10https://gerrit.wikimedia.org/r/1364149 (https://phabricator.wikimedia.org/T441202) (owner: 10Btullis) [12:34:42] (03CR) 10Btullis: [C:03+2] Deprecate and de-populate the dumps-roots group [puppet] - 10https://gerrit.wikimedia.org/r/1364150 (https://phabricator.wikimedia.org/T440471) (owner: 10Btullis) [12:35:19] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1226 (T441633)', diff saved to https://phabricator.wikimedia.org/P97361 and previous config saved to /var/cache/conftool/dbconfig/20261105-123519-ladsgroup.json [12:35:20] going to bump mw-web to 5% on php 8.5 unless anyone objects [12:36:37] (03PS1) 10Mpostoronca: WikimediaAntiAbuse: Map the virtual domain to x1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364300 (https://phabricator.wikimedia.org/T443475) [12:37:34] !log installing alsa-lib security updates [12:37:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:37:37] (03PS1) 10Matthias Mullie: Squashed diff to master [extensions/ReaderExperiments] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1364301 [12:38:00] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 05 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [extensions/ReaderExperiments] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1364301 (owner: 10Matthias Mullie) [12:38:58] (03CR) 10Kamila Součková: "going to self-merge this as it just changes a constant within an obviously safe range" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364180 (https://phabricator.wikimedia.org/T441689) (owner: 10Kamila Součková) [12:40:25] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kamila@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364180 (https://phabricator.wikimedia.org/T441689) (owner: 10Kamila Součková) [12:41:34] (03PS1) 10Daniel Kertesz: team-traffic: add rp_filter_enabled alert for lvs nodes [alerts] - 10https://gerrit.wikimedia.org/r/1364303 (https://phabricator.wikimedia.org/T443771) [12:41:41] (03Merged) 10jenkins-bot: Enroll 5% of client sessions in PHP 8.5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364180 (https://phabricator.wikimedia.org/T441689) (owner: 10Kamila Součková) [12:41:58] !log kamila@deploy2003 Started scap sync-world: Backport for [[gerrit:1364180|Enroll 5% of client sessions in PHP 8.5 (T441689)]] [12:42:01] T441689: [PHP 8.5] Migrate production mw-web service to PHP 8.5 (stage 1: cookie-based) - https://phabricator.wikimedia.org/T441689 [12:42:44] 10SRE-SLO, 10Citoid, 06Editing-team, 06SRE Observability, 07Sustainability (Incident Followup): Create probe for citoid that tests whether requests can access the outside internet - https://phabricator.wikimedia.org/T381372#12522000 (10hnowlan) [12:45:21] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97362 and previous config saved to /var/cache/conftool/dbconfig/20261105-124521-ladsgroup.json [12:45:30] (03PS1) 10Lucian: Add lucian to the ops-limited group [puppet] - 10https://gerrit.wikimedia.org/r/1364304 [12:46:08] !log kamila@deploy2003 kamila: Backport for [[gerrit:1364180|Enroll 5% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:46:10] (03CR) 10Sergio Gimeno: [C:03+1] "Maybe we could also do a pass to https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/vagrant/+/refs/heads/master/puppet/modules/role/" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363328 (https://phabricator.wikimedia.org/T443648) (owner: 10Urbanecm) [12:46:42] !log kamila@deploy2003 kamila: Continuing with deployment [12:51:49] (03CR) 10Vgutierrez: team-traffic: add rp_filter_enabled alert for lvs nodes (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1364303 (https://phabricator.wikimedia.org/T443771) (owner: 10Daniel Kertesz) [12:52:48] !log kamila@deploy2003 Finished scap sync-world: Backport for [[gerrit:1364180|Enroll 5% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 50s) [12:52:52] T441689: [PHP 8.5] Migrate production mw-web service to PHP 8.5 (stage 1: cookie-based) - https://phabricator.wikimedia.org/T441689 [12:54:13] (03CR) 10Dreamy Jazz: [C:03+1] WikimediaAntiAbuse: Map the virtual domain to x1 (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364300 (https://phabricator.wikimedia.org/T443475) (owner: 10Mpostoronca) [12:54:25] (03PS2) 10Daniel Kertesz: team-traffic: add rp_filter_enabled alert for lvs nodes [alerts] - 10https://gerrit.wikimedia.org/r/1364303 (https://phabricator.wikimedia.org/T443771) [12:54:49] (03CR) 10Daniel Kertesz: team-traffic: add rp_filter_enabled alert for lvs nodes (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1364303 (https://phabricator.wikimedia.org/T443771) (owner: 10Daniel Kertesz) [12:55:23] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97363 and previous config saved to /var/cache/conftool/dbconfig/20261105-125522-ladsgroup.json [12:55:45] (03PS1) 10Kamila Součková: deployment_server: bump mw-(parsoid,misc,experimental) to php 8.5 [puppet] - 10https://gerrit.wikimedia.org/r/1364315 (https://phabricator.wikimedia.org/T439955) [12:57:13] (03PS1) 10Mforns: mforns - add yubikey ssh public key [puppet] - 10https://gerrit.wikimedia.org/r/1364317 (https://phabricator.wikimedia.org/T442758) [12:57:58] (03PS1) 10Effie Mouzeli: osm: update bootstrap-tile-storage [puppet] - 10https://gerrit.wikimedia.org/r/1364318 (https://phabricator.wikimedia.org/T443438) [12:58:47] (03CR) 10CI reject: [V:04-1] osm: update bootstrap-tile-storage [puppet] - 10https://gerrit.wikimedia.org/r/1364318 (https://phabricator.wikimedia.org/T443438) (owner: 10Effie Mouzeli) [13:00:05] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1300) [13:01:28] (03PS2) 10Effie Mouzeli: osm: update bootstrap-tile-storage [puppet] - 10https://gerrit.wikimedia.org/r/1364318 (https://phabricator.wikimedia.org/T443438) [13:01:39] PROBLEM - ganeti-wconfd running on ganeti-test2003 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 110 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [13:02:16] (03CR) 10CI reject: [V:04-1] osm: update bootstrap-tile-storage [puppet] - 10https://gerrit.wikimedia.org/r/1364318 (https://phabricator.wikimedia.org/T443438) (owner: 10Effie Mouzeli) [13:03:34] (03PS2) 10Kamila Součková: deployment_server: bump mw-{misc,experimental} to php 8.5 [puppet] - 10https://gerrit.wikimedia.org/r/1364315 (https://phabricator.wikimedia.org/T439955) [13:03:54] (03CR) 10Effie Mouzeli: [C:03+1] deployment_server: bump mw-{misc,experimental} to php 8.5 [puppet] - 10https://gerrit.wikimedia.org/r/1364315 (https://phabricator.wikimedia.org/T439955) (owner: 10Kamila Součková) [13:05:25] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1226 (T441633)', diff saved to https://phabricator.wikimedia.org/P97364 and previous config saved to /var/cache/conftool/dbconfig/20261105-130524-ladsgroup.json [13:05:43] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1281.eqiad.wmnet with reason: Maintenance [13:05:51] PROBLEM - Host sretest1006 is DOWN: PING CRITICAL - Packet loss = 100% [13:06:06] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance [13:06:08] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1281 (T441633)', diff saved to https://phabricator.wikimedia.org/P97365 and previous config saved to /var/cache/conftool/dbconfig/20261105-130607-ladsgroup.json [13:06:29] (03PS3) 10Effie Mouzeli: osm: update bootstrap-tile-storage [puppet] - 10https://gerrit.wikimedia.org/r/1364318 (https://phabricator.wikimedia.org/T443438) [13:08:48] (03CR) 10Muehlenhoff: [C:03+1] "Verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1364317 (https://phabricator.wikimedia.org/T442758) (owner: 10Mforns) [13:08:49] (03CR) 10Muehlenhoff: [C:03+2] mforns - add yubikey ssh public key [puppet] - 10https://gerrit.wikimedia.org/r/1364317 (https://phabricator.wikimedia.org/T442758) (owner: 10Mforns) [13:08:55] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364325 [13:09:07] 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20), 07Essential-Work: Follow up on multiple RAID / drive issues - https://phabricator.wikimedia.org/T426610#12522076 (10BTullis) 05Open→03Resolved I'm going to resolve this ticket, for now. We know that `an-worker1[187-208]` are particular... [13:11:21] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1281 (T441633)', diff saved to https://phabricator.wikimedia.org/P97366 and previous config saved to /var/cache/conftool/dbconfig/20261105-131120-ladsgroup.json [13:11:32] (03PS1) 10Muehlenhoff: Remove legacy key for akhatun [puppet] - 10https://gerrit.wikimedia.org/r/1364327 (https://phabricator.wikimedia.org/T442758) [13:13:45] (03PS1) 10Effie Mouzeli: maps: Disable replication timers in eqiad/bookworm [puppet] - 10https://gerrit.wikimedia.org/r/1364328 (https://phabricator.wikimedia.org/T443438) [13:14:13] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [13:14:20] (03PS1) 10JMeybohm: Add debian packaging from v1.34 branch [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/1364329 [13:14:20] (03PS1) 10JMeybohm: Update debian/ to gbp/pbuilder builds [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/1364330 [13:17:21] (03PS2) 10JMeybohm: Add debian packaging from v1.34 branch [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/1364329 (https://phabricator.wikimedia.org/T443455) [13:17:23] (03PS2) 10JMeybohm: Update debian/ to gbp/pbuilder builds [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/1364330 (https://phabricator.wikimedia.org/T443455) [13:18:16] !log cmooney@cumin1004 START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie [13:19:19] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004" [13:19:21] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004" [13:19:22] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:19:29] !log cmooney@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only) [13:19:32] !log cmooney@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only) [13:19:59] !log cmooney@cumin1004 START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors [13:20:02] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors [13:21:22] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97367 and previous config saved to /var/cache/conftool/dbconfig/20261105-132122-ladsgroup.json [13:26:44] !log cmooney@cumin1004 START - Cookbook sre.hosts.provision for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:30:03] (03PS1) 10Eevans: linked-artifacts: deploy Hoarde v1.5.4 (prod) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364344 (https://phabricator.wikimedia.org/T441744) [13:31:20] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12522159 (10cmooney) [13:31:24] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97368 and previous config saved to /var/cache/conftool/dbconfig/20261105-133123-ladsgroup.json [13:32:25] (03CR) 10Eevans: [C:03+2] linked-artifacts: deploy Hoarde v1.5.4 (prod) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364344 (https://phabricator.wikimedia.org/T441744) (owner: 10Eevans) [13:33:06] !log cmooney@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [13:33:46] (03CR) 10Vgutierrez: [C:03+1] "looks good, we could need to adjust it in the future if we stop configuring some NICs after L2 adjancency is not required anymore" [alerts] - 10https://gerrit.wikimedia.org/r/1364303 (https://phabricator.wikimedia.org/T443771) (owner: 10Daniel Kertesz) [13:33:47] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12522178 (10cmooney) [13:33:59] (03Merged) 10jenkins-bot: linked-artifacts: deploy Hoarde v1.5.4 (prod) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364344 (https://phabricator.wikimedia.org/T441744) (owner: 10Eevans) [13:34:52] !log eevans@deploy2003 helmfile [staging] START helmfile.d/services/linked-artifacts: apply [13:35:00] !log eevans@deploy2003 helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply [13:35:07] !log eevans@deploy2003 helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply [13:35:27] !log eevans@deploy2003 helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply [13:35:41] !log eevans@deploy2003 helmfile [codfw] START helmfile.d/services/linked-artifacts: apply [13:35:56] !log eevans@deploy2003 helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply [13:41:26] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1281 (T441633)', diff saved to https://phabricator.wikimedia.org/P97369 and previous config saved to /var/cache/conftool/dbconfig/20261105-134125-ladsgroup.json [13:41:44] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1286.eqiad.wmnet with reason: Maintenance [13:41:46] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1286 (T441633)', diff saved to https://phabricator.wikimedia.org/P97370 and previous config saved to /var/cache/conftool/dbconfig/20261105-134145-ladsgroup.json [13:42:10] (03CR) 10Filippo Giunchedi: [C:03+1] errorpage: Hide Wikimedia branding outside production [puppet] - 10https://gerrit.wikimedia.org/r/1363420 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [13:44:17] (03PS3) 10JMeybohm: Update debian/ to gbp/pbuilder builds [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/1364330 (https://phabricator.wikimedia.org/T443455) [13:45:51] !log cmooney@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only) [13:45:55] !log cmooney@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only) [13:46:50] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1286 (T441633)', diff saved to https://phabricator.wikimedia.org/P97371 and previous config saved to /var/cache/conftool/dbconfig/20261105-134649-ladsgroup.json [13:48:00] !log cmooney@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage [13:48:55] 06SRE, 06DC-Ops, 06tools-infrastructure-team, 06cloud-services-team (Hardware), and 2 others: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12522241 (10xcollazo) Quick update: when I read doubling storage to 320TB I got *very* excited about the possibilities! De... [13:50:40] PROBLEM - Host sretest1006 is DOWN: PING CRITICAL - Packet loss = 100% [13:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 98.86% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [13:53:31] !log cmooney@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage [13:55:42] RECOVERY - Host sretest1006 is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms [13:55:52] (03CR) 10Kamila Součková: [C:03+2] deployment_server: bump mw-{misc,experimental} to php 8.5 [puppet] - 10https://gerrit.wikimedia.org/r/1364315 (https://phabricator.wikimedia.org/T439955) (owner: 10Kamila Součková) [13:56:14] (03PS3) 10Huei Tan: ULS: Use the new language selector on Timeless on testwiki and mediawiki.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363037 (https://phabricator.wikimedia.org/T439715) [13:56:15] (03CR) 10STran: "`" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364209 (https://phabricator.wikimedia.org/T443646) (owner: 10STran) [13:56:19] (03PS1) 10Huei Tan: ULS: Use the new language selector on MonoBook on testwiki and mediawiki.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363074 (https://phabricator.wikimedia.org/T439718) [13:56:52] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97372 and previous config saved to /var/cache/conftool/dbconfig/20261105-135651-ladsgroup.json [13:58:24] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364353 [13:59:33] jouncebot: nowandnext [13:59:33] For the next 0 hour(s) and 0 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1300) [13:59:33] In 0 hour(s) and 0 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1400) [14:00:04] Lucas_WMDE, urbanecm, and TheresNoTime: Time to snap out of that daydream and deploy UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1400). [14:00:05] Tran and matthiasmullie: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [14:00:06] o/ [14:00:08] o/ [14:00:18] I'm going to do my own deploy as I have a private deploy component to it after [14:00:49] (03PS1) 10Huei Tan: ULS: Use the new language selector on Vector legacy on testwiki and mediawiki.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364354 (https://phabricator.wikimedia.org/T439718) [14:02:11] Tran: daylight confusion strikes again, I didn't get my infra change in before the deploy window... you might see an "83 -> 85" diff on mw-misc and mw-experimental, it's safe to deploy [14:02:34] ack [14:02:35] (I really need to figure out that deploy windows are an hour earlier now '^^) [14:02:40] going to start then [14:02:45] (03CR) 10TrainBranchBot: [C:03+2] "Approved by stran@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364209 (https://phabricator.wikimedia.org/T443646) (owner: 10STran) [14:03:10] * TheresNoTime is about [14:03:18] matthiasmullie: will you self-deploy? [14:03:25] yep! [14:03:30] (03Merged) 10jenkins-bot: Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364209 (https://phabricator.wikimedia.org/T443646) (owner: 10STran) [14:03:30] cool :D [14:03:34] * TheresNoTime goes back to coffee [14:03:46] !log stran@deploy2003 Started scap sync-world: Backport for [[gerrit:1364209|Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] [14:03:51] T443646: Enable SI on nlwiki - https://phabricator.wikimedia.org/T443646 [14:03:51] T443147: Enable SuggestedInvestigations on enwikivoyage - https://phabricator.wikimedia.org/T443147 [14:03:56] * Raine mmm coffeeeeeeee [14:06:54] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97373 and previous config saved to /var/cache/conftool/dbconfig/20261105-140653-ladsgroup.json [14:07:22] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364364 [14:07:39] (03CR) 10Muehlenhoff: [C:03+1] "Verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1363606 (https://phabricator.wikimedia.org/T442758) (owner: 10Lerickson) [14:07:41] (03CR) 10Muehlenhoff: [C:03+2] admin: Add FIDO SSH key for lerickson [puppet] - 10https://gerrit.wikimedia.org/r/1363606 (https://phabricator.wikimedia.org/T442758) (owner: 10Lerickson) [14:08:06] !log stran@deploy2003 stran: Backport for [[gerrit:1364209|Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:09:23] !log stran@deploy2003 stran: Continuing with deployment [14:09:41] (03CR) 10Muehlenhoff: [C:03+1] "Verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1363564 (https://phabricator.wikimedia.org/T442758) (owner: 10Jdrewniak) [14:09:43] (03CR) 10Muehlenhoff: [C:03+2] admin: Add FIDO SSH key for jdrewniak [puppet] - 10https://gerrit.wikimedia.org/r/1363564 (https://phabricator.wikimedia.org/T442758) (owner: 10Jdrewniak) [14:09:51] (03CR) 10Dbrant: [C:03+2] mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364364 (owner: 10PipelineBot) [14:10:00] (03Abandoned) 10Dbrant: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364353 (owner: 10PipelineBot) [14:11:15] (03Merged) 10jenkins-bot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364364 (owner: 10PipelineBot) [14:11:29] !log cmooney@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie [14:12:02] (03CR) 10Muehlenhoff: [C:03+2] Record extended MOU dates for aarora and ankita97531 [puppet] - 10https://gerrit.wikimedia.org/r/1364145 (owner: 10Muehlenhoff) [14:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [14:15:20] !log stran@deploy2003 Finished scap sync-world: Backport for [[gerrit:1364209|Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)]] (duration: 11m 34s) [14:15:25] T443646: Enable SI on nlwiki - https://phabricator.wikimedia.org/T443646 [14:15:25] T443147: Enable SuggestedInvestigations on enwikivoyage - https://phabricator.wikimedia.org/T443147 [14:15:35] config deploy done, doing private deploy now [14:16:56] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1286 (T441633)', diff saved to https://phabricator.wikimedia.org/P97374 and previous config saved to /var/cache/conftool/dbconfig/20261105-141655-ladsgroup.json [14:17:14] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance [14:17:23] Tran: can you ping me when you're done? [14:17:40] (03CR) 10Muehlenhoff: [C:03+1] "Verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1363478 (https://phabricator.wikimedia.org/T443508) (owner: 10SBassett) [14:17:42] (03CR) 10Muehlenhoff: [C:03+2] admin: Add and update FIDO SSH key for sbassett [puppet] - 10https://gerrit.wikimedia.org/r/1363478 (https://phabricator.wikimedia.org/T443508) (owner: 10SBassett) [14:18:35] np, will do [14:19:57] (03PS1) 10Muehlenhoff: Remove legacy key for mpostoronca [puppet] - 10https://gerrit.wikimedia.org/r/1364379 (https://phabricator.wikimedia.org/T442758) [14:20:25] (03PS2) 10Mpostoronca: WikimediaAntiAbuse: Map the virtual domain to x1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364300 (https://phabricator.wikimedia.org/T443475) [14:20:51] (03CR) 10Mpostoronca: WikimediaAntiAbuse: Map the virtual domain to x1 (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364300 (https://phabricator.wikimedia.org/T443475) (owner: 10Mpostoronca) [14:21:26] (03CR) 10Vgutierrez: trafficserver: route WDQSv2 through REST Gateway (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1363813 (https://phabricator.wikimedia.org/T443756) (owner: 10Gmodena) [14:21:51] (03CR) 10Ayounsi: Add _get_bgp_config (034 comments) [software/homer/deploy] - 10https://gerrit.wikimedia.org/r/1360446 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [14:22:15] (03PS2) 10Ayounsi: Config Transit BGP from netbox-bgp [homer/public] - 10https://gerrit.wikimedia.org/r/1360452 (https://phabricator.wikimedia.org/T439408) [14:22:32] (03CR) 10Ssingh: "Thanks for the patch. I do think these are separate cookbooks so should probably be separate patches?" [cookbooks] - 10https://gerrit.wikimedia.org/r/1344697 (https://phabricator.wikimedia.org/T439025) (owner: 10Arnaudb) [14:23:07] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy key for mpostoronca [puppet] - 10https://gerrit.wikimedia.org/r/1364379 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [14:24:04] (03CR) 10Ssingh: [C:03+2] "Rebased since last two code reviews, no code change." [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 (https://phabricator.wikimedia.org/T440168) (owner: 10Ssingh) [14:26:08] (03CR) 10Arnaudb: "good idea, I'm working in that direction and will let you know when this is a bit more reviewable" [cookbooks] - 10https://gerrit.wikimedia.org/r/1344697 (https://phabricator.wikimedia.org/T439025) (owner: 10Arnaudb) [14:27:19] (03PS3) 10Ayounsi: Add _get_bgp_config [software/homer/deploy] - 10https://gerrit.wikimedia.org/r/1360446 (https://phabricator.wikimedia.org/T439408) [14:27:19] (03PS3) 10Ayounsi: BGP: implement 'dont_export' on circuits [software/homer/deploy] - 10https://gerrit.wikimedia.org/r/1362088 (https://phabricator.wikimedia.org/T439408) [14:27:42] (03Merged) 10jenkins-bot: sre.dns.admin: add option to depool DNS hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1335049 (https://phabricator.wikimedia.org/T440168) (owner: 10Ssingh) [14:28:49] (03CR) 10Ayounsi: "I can take care of adding support for "prepend_out" in a future CR if we think it's needed." [software/homer/deploy] - 10https://gerrit.wikimedia.org/r/1362088 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [14:29:31] (03CR) 10Daniel Kertesz: [C:03+2] team-traffic: add rp_filter_enabled alert for lvs nodes [alerts] - 10https://gerrit.wikimedia.org/r/1364303 (https://phabricator.wikimedia.org/T443771) (owner: 10Daniel Kertesz) [14:32:15] !log Deployed patch for T443598 [14:32:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:32:23] matthiasmullie: I'm done [14:32:27] (03Merged) 10jenkins-bot: team-traffic: add rp_filter_enabled alert for lvs nodes [alerts] - 10https://gerrit.wikimedia.org/r/1364303 (https://phabricator.wikimedia.org/T443771) (owner: 10Daniel Kertesz) [14:32:43] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy2003 using scap backport" [extensions/ReaderExperiments] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1364301 (owner: 10Matthias Mullie) [14:32:46] FIRING: [2x] CertAlmostExpired: Certificate for service kubestagemaster2005:6443 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#kubestagemaster2005:6443 - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:34:18] (03Merged) 10jenkins-bot: Squashed diff to master [extensions/ReaderExperiments] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1364301 (owner: 10Matthias Mullie) [14:34:35] !log mlitn@deploy2003 Started scap sync-world: Backport for [[gerrit:1364301|Squashed diff to master]] (1.48.0-wmf.3) [14:38:44] !log mlitn@deploy2003 mlitn: Backport for [[gerrit:1364301|Squashed diff to master]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:38:52] (03CR) 10Jgiannelos: [C:03+1] "Nit comments, overall looks OK." [puppet] - 10https://gerrit.wikimedia.org/r/1364318 (https://phabricator.wikimedia.org/T443438) (owner: 10Effie Mouzeli) [14:41:16] !log mlitn@deploy2003 mlitn: Continuing with deployment [14:41:29] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 05 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362175 (https://phabricator.wikimedia.org/T443449) (owner: 10Codename Noreste) [14:41:54] (03CR) 10Fabfur: "OK for X-Kapow-Estimate (header)" [puppet] - 10https://gerrit.wikimedia.org/r/1363149 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [14:42:02] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1362175 [14:42:19] see discussion linked on T443449 [14:42:19] T443449: Disable page creation for anonymous users on English Wikibooks - https://phabricator.wikimedia.org/T443449 [14:42:30] but I need the heads up before we deploy [14:43:10] (03PS3) 10Fabfur: profile::cache,hiera: enable kapow score everywhere and log the var [puppet] - 10https://gerrit.wikimedia.org/r/1363149 (https://phabricator.wikimedia.org/T435799) [14:43:21] (03CR) 10Fabfur: profile::cache,hiera: enable kapow score everywhere and log the var (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363149 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [14:44:13] !log dbrant@deploy2003 helmfile [staging] START helmfile.d/services/mobileapps: apply [14:44:18] (03PS1) 10Thibaut Le Page: striker: (toolsbeta) bump container version [puppet] - 10https://gerrit.wikimedia.org/r/1364398 (https://phabricator.wikimedia.org/T443773) [14:44:39] !log dbrant@deploy2003 helmfile [staging] DONE helmfile.d/services/mobileapps: apply [14:44:56] (03PS1) 10STran: Enable Special:SuggestedInvestigations for nlwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364399 (https://phabricator.wikimedia.org/T443646) [14:45:08] (03CR) 10FNegri: [C:03+2] striker: (toolsbeta) bump container version [puppet] - 10https://gerrit.wikimedia.org/r/1364398 (https://phabricator.wikimedia.org/T443773) (owner: 10Thibaut Le Page) [14:45:35] (03CR) 10Ayounsi: Config Transit BGP from netbox-bgp (032 comments) [homer/public] - 10https://gerrit.wikimedia.org/r/1360452 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [14:45:46] jouncebot: nowandnext [14:45:46] For the next 0 hour(s) and 14 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1400) [14:45:46] In 0 hour(s) and 44 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1530) [14:45:49] (03CR) 10STran: Enable Special:SuggestedInvestigations for nlwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364399 (https://phabricator.wikimedia.org/T443646) (owner: 10STran) [14:45:52] (03PS3) 10Ayounsi: Config Transit BGP from netbox-bgp [homer/public] - 10https://gerrit.wikimedia.org/r/1360452 (https://phabricator.wikimedia.org/T439408) [14:46:18] (03PS1) 10Kamila Součková: Revert^3 "mw-parsoid: temporarily override image to PHP 8.5 in eqiad" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364402 [14:46:35] !log akhatun@deploy2003 helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply [14:46:42] !log akhatun@deploy2003 helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply [14:47:07] !log dbrant@deploy2003 helmfile [eqiad] START helmfile.d/services/mobileapps: apply [14:47:14] !log mlitn@deploy2003 Finished scap sync-world: Backport for [[gerrit:1364301|Squashed diff to master]] (1.48.0-wmf.3) (duration: 12m 38s) [14:47:56] !log dbrant@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [14:48:05] !log dbrant@deploy2003 helmfile [codfw] START helmfile.d/services/mobileapps: apply [14:48:25] I'm done; anyone else in this window? [14:48:32] Do I have time to sneak in another config deploy? [14:48:44] !log dbrant@deploy2003 helmfile [codfw] DONE helmfile.d/services/mobileapps: apply [14:48:46] codenamenoreste: looks like you're up next - will you self-deploy, or do you need help? [14:49:12] I have a patch to see but I do not have deploy rights [14:49:17] Tran: I guess we can combine with codenamenoreste? [14:49:29] it's for enwikibooks removing createpage from anonymous users [14:49:37] (03CR) 10Mszwarc: [C:03+1] Enable Special:SuggestedInvestigations for nlwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364399 (https://phabricator.wikimedia.org/T443646) (owner: 10STran) [14:49:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12522546 (10phaultfinder) [14:49:47] however, we need to see if that goes against the limits for configuration changes... [14:51:22] i'm back, internet was choppy and had to go to my main account [14:52:48] codenamenoreste: so can your config patch be deployed? Or is it awaiting approval? [14:53:17] (03PS9) 10Arnaudb: gerrit: read the replica flag from the loaded unit [cookbooks] - 10https://gerrit.wikimedia.org/r/1344698 (https://phabricator.wikimedia.org/T439025) [14:53:17] (03PS9) 10Arnaudb: gerrit: move duplicated code into shared helpers [cookbooks] - 10https://gerrit.wikimedia.org/r/1344699 (https://phabricator.wikimedia.org/T439025) [14:53:17] (03PS3) 10Arnaudb: gerrit: reindex the new primary offline before it starts [cookbooks] - 10https://gerrit.wikimedia.org/r/1357493 (https://phabricator.wikimedia.org/T440177) [14:53:36] it can theoretically be deployed, but there is a limit against disabling page creations for non-autoconfirmed users, however I am not sure if it applies the same to anonymous users [14:54:24] so should it not be deployed while someone confirms that or...? [14:54:53] go ahead probably, some wikis do disable page creations for new users [14:55:02] we'll see [14:55:51] well I guess we'll see, sorry if someone has to ping me asking why we did this. Going to deploy it with my config patch. [14:55:59] (03PS9) 10Arnaudb: gerrit: add a cookbook for the emergency DNS edit [cookbooks] - 10https://gerrit.wikimedia.org/r/1344697 (https://phabricator.wikimedia.org/T439025) [14:55:59] (03CR) 10Arnaudb: "done!" [cookbooks] - 10https://gerrit.wikimedia.org/r/1344697 (https://phabricator.wikimedia.org/T439025) (owner: 10Arnaudb) [14:55:59] (03PS9) 10Ayounsi: Add cookbook to sync BGP sessions with Netbox [cookbooks] - 10https://gerrit.wikimedia.org/r/1347441 (https://phabricator.wikimedia.org/T439408) [14:56:00] !log akhatun@deploy2003 helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply [14:56:17] (03PS10) 10Ayounsi: Add cookbook to sync BGP sessions with Netbox [cookbooks] - 10https://gerrit.wikimedia.org/r/1347441 (https://phabricator.wikimedia.org/T439408) [14:56:19] !log akhatun@deploy2003 helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply [14:56:26] (03CR) 10TrainBranchBot: [C:03+2] "Approved by stran@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362175 (https://phabricator.wikimedia.org/T443449) (owner: 10Codename Noreste) [14:56:27] (03CR) 10TrainBranchBot: [C:03+2] "Approved by stran@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364399 (https://phabricator.wikimedia.org/T443646) (owner: 10STran) [14:56:49] (03CR) 10Jgiannelos: [C:03+1] maps: Disable replication timers in eqiad/bookworm [puppet] - 10https://gerrit.wikimedia.org/r/1364328 (https://phabricator.wikimedia.org/T443438) (owner: 10Effie Mouzeli) [14:57:01] (03CR) 10Ayounsi: Add cookbook to sync BGP sessions with Netbox (034 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1347441 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [14:57:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:57:32] (03PS1) 10Arnaudb: gerrit: add a cookbook to demote a former primary [cookbooks] - 10https://gerrit.wikimedia.org/r/1364415 (https://phabricator.wikimedia.org/T439025) [14:57:42] (03PS1) 10Klausman: hiera: Switch LW staging to using IPIP for services [puppet] - 10https://gerrit.wikimedia.org/r/1364397 (https://phabricator.wikimedia.org/T441352) [14:57:47] (03PS1) 10Arnaudb: gerrit: add the failover cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1364416 (https://phabricator.wikimedia.org/T439025) [14:57:59] (03Merged) 10jenkins-bot: Round 2 on disabling createpage for anonymous users on English Wikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362175 (https://phabricator.wikimedia.org/T443449) (owner: 10Codename Noreste) [14:58:02] (03Merged) 10jenkins-bot: Enable Special:SuggestedInvestigations for nlwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364399 (https://phabricator.wikimedia.org/T443646) (owner: 10STran) [14:58:20] !log stran@deploy2003 Started scap sync-world: Backport for [[gerrit:1362175|Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399|Enable Special:SuggestedInvestigations for nlwiki (T443646)]] [14:58:25] T443449: Disable page creation for anonymous users on English Wikibooks - https://phabricator.wikimedia.org/T443449 [14:58:26] T443646: Enable SI on nlwiki - https://phabricator.wikimedia.org/T443646 [15:01:49] (03CR) 10Vgutierrez: [C:03+1] "config is OK and realservers are handling IPIP traffic as expected:" [puppet] - 10https://gerrit.wikimedia.org/r/1364397 (https://phabricator.wikimedia.org/T441352) (owner: 10Klausman) [15:02:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [15:02:18] !log jhancock@cumin2003 START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [15:02:29] !log jhancock@cumin2003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [15:02:36] !log stran@deploy2003 stran, codenamenoreste: Backport for [[gerrit:1362175|Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399|Enable Special:SuggestedInvestigations for nlwiki (T443646)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:02:51] !log jhancock@cumin2003 START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [15:03:58] codenamenoreste: your config is on test servers, could you test? [15:04:37] using wikimediadebug, createpage is NOT in the anonymous user toolkit, while registered users CAN do so [15:04:49] thanks, continuing [15:04:56] !log stran@deploy2003 stran, codenamenoreste: Continuing with deployment [15:05:51] (03CR) 10Klausman: [V:03+2 C:03+2] hiera: Switch LW staging to using IPIP for services [puppet] - 10https://gerrit.wikimedia.org/r/1364397 (https://phabricator.wikimedia.org/T441352) (owner: 10Klausman) [15:08:28] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [15:09:14] (03PS2) 10Gmodena: trafficserver: route WDQSv2 through REST Gateway [puppet] - 10https://gerrit.wikimedia.org/r/1363813 (https://phabricator.wikimedia.org/T443756) [15:09:38] (03PS1) 10Hashar: Add some shellcheck shell hint [puppet] - 10https://gerrit.wikimedia.org/r/1364428 [15:10:49] !log stran@deploy2003 Finished scap sync-world: Backport for [[gerrit:1362175|Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449)]], [[gerrit:1364399|Enable Special:SuggestedInvestigations for nlwiki (T443646)]] (duration: 12m 28s) [15:10:54] T443449: Disable page creation for anonymous users on English Wikibooks - https://phabricator.wikimedia.org/T443449 [15:10:54] T443646: Enable SI on nlwiki - https://phabricator.wikimedia.org/T443646 [15:10:54] done [15:12:46] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs2014.codfw.wmnet} and A:lvs [15:13:20] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs2014.codfw.wmnet} and A:lvs [15:13:42] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs2013.codfw.wmnet} and A:lvs [15:13:52] (03CR) 10Gmodena: trafficserver: route WDQSv2 through REST Gateway (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1363813 (https://phabricator.wikimedia.org/T443756) (owner: 10Gmodena) [15:14:28] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs2013.codfw.wmnet} and A:lvs [15:14:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12522738 (10phaultfinder) [15:15:25] (03CR) 10Vgutierrez: trafficserver: route WDQSv2 through REST Gateway (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363813 (https://phabricator.wikimedia.org/T443756) (owner: 10Gmodena) [15:18:00] 10SRE-tools, 06Infrastructure-Foundations, 13Patch-For-Review: Debmonitor: backend-changeable settings are stored in the browser's session storage - https://phabricator.wikimedia.org/T240457#12522764 (10Lizimedia) @Volans we should either merge this speculatively or close it, I think [15:22:57] (03PS4) 10Effie Mouzeli: osm: update bootstrap-tile-storage [puppet] - 10https://gerrit.wikimedia.org/r/1364318 (https://phabricator.wikimedia.org/T443438) [15:23:14] (03PS1) 10Jgiannelos: imposm: Skip water lines import because source mirror is down [puppet] - 10https://gerrit.wikimedia.org/r/1364444 [15:24:12] (03CR) 10Effie Mouzeli: osm: update bootstrap-tile-storage (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1364318 (https://phabricator.wikimedia.org/T443438) (owner: 10Effie Mouzeli) [15:24:52] (03PS2) 10Jgiannelos: imposm: Skip water lines import because source mirror is down [puppet] - 10https://gerrit.wikimedia.org/r/1364444 (https://phabricator.wikimedia.org/T443438) [15:25:47] (03CR) 10Aghirelli: [C:03+1] REST gateway: add support for /api/ routes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1354040 (https://phabricator.wikimedia.org/T433547) (owner: 10Daniel Kinzler) [15:27:01] !log sukhe@cumin1004 START - Cookbook sre.dns.admin DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified] [15:27:08] !log sukhe@cumin1004 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified] [15:27:42] (03PS5) 10Effie Mouzeli: osm: update bootstrap-tile-storage [puppet] - 10https://gerrit.wikimedia.org/r/1364318 (https://phabricator.wikimedia.org/T443438) [15:28:04] !log sukhe@cumin1004 START - Cookbook sre.dns.admin DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified] [15:28:06] !log sukhe@cumin1004 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified] [15:29:34] (03CR) 10Effie Mouzeli: [C:03+2] osm: update bootstrap-tile-storage [puppet] - 10https://gerrit.wikimedia.org/r/1364318 (https://phabricator.wikimedia.org/T443438) (owner: 10Effie Mouzeli) [15:29:42] (03CR) 10Jgiannelos: [C:03+1] osm: update bootstrap-tile-storage [puppet] - 10https://gerrit.wikimedia.org/r/1364318 (https://phabricator.wikimedia.org/T443438) (owner: 10Effie Mouzeli) [15:30:04] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1530) [15:35:25] (03PS1) 10Kevin Bazira: airflow-ml: enable internet access for the TTS pipeline [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364448 (https://phabricator.wikimedia.org/T441987) [15:40:27] (03CR) 10Xcollazo: [C:03+1] "@btullis@wikimedia.org please merge if you agree. thanks." [puppet] - 10https://gerrit.wikimedia.org/r/1360975 (https://phabricator.wikimedia.org/T438056) (owner: 10Xcollazo) [15:41:27] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps: hw troubleshooting: power-on for wikikube-worker2337.codfw.wmnet - https://phabricator.wikimedia.org/T442714#12522905 (10Jhancock.wm) this one still refuses to boot. might be an issue with the CPU but it shows healthy. found something online that the BMC gets stu... [15:42:46] FIRING: [4x] CertAlmostExpired: Certificate for service kubestagemaster2003:6443 is about to expire - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [15:45:45] 07Puppet, 06SRE, 06Release-Engineering-Team, 10Continuous-Integration-Config: Upgrade operations/puppet CI from Bullseye to Bookworm - https://phabricator.wikimedia.org/T443798 (10hashar) 03NEW [15:45:53] 07Puppet, 06SRE, 06Release-Engineering-Team, 10Continuous-Integration-Config: Upgrade operations/puppet CI from Bullseye to Bookworm - https://phabricator.wikimedia.org/T443798#12522973 (10hashar) [15:47:13] (03PS1) 10Hashar: icinga: remove check_iostat [puppet] - 10https://gerrit.wikimedia.org/r/1364450 (https://phabricator.wikimedia.org/T443798) [15:47:31] (03PS2) 10Hashar: Add some shellcheck shell hint [puppet] - 10https://gerrit.wikimedia.org/r/1364428 (https://phabricator.wikimedia.org/T443798) [15:55:04] (03PS1) 10Thibaut Le Page: striker: (production) bump container version [puppet] - 10https://gerrit.wikimedia.org/r/1364453 (https://phabricator.wikimedia.org/T443773) [15:59:14] (03CR) 10Mforns: [C:03+1] "Thank you! +1" [puppet] - 10https://gerrit.wikimedia.org/r/1363149 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [16:00:46] (03PS1) 10Muehlenhoff: Add FIDO key for jsn [puppet] - 10https://gerrit.wikimedia.org/r/1364457 (https://phabricator.wikimedia.org/T442758) [16:02:28] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for jsn [puppet] - 10https://gerrit.wikimedia.org/r/1364457 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [16:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:04:50] (03CR) 10Kamila Součková: [C:03+2] Revert^3 "mw-parsoid: temporarily override image to PHP 8.5 in eqiad" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364402 (owner: 10Kamila Součková) [16:04:56] (03CR) 10Kamila Součková: [C:03+2] deployment_server: switch mw-parsoid to PHP 8.5 [puppet] - 10https://gerrit.wikimedia.org/r/1357273 (https://phabricator.wikimedia.org/T440321) (owner: 10Kamila Součková) [16:06:31] (03Merged) 10jenkins-bot: Revert^3 "mw-parsoid: temporarily override image to PHP 8.5 in eqiad" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364402 (owner: 10Kamila Součková) [16:08:03] (03PS1) 10Muehlenhoff: Remove legacy key for mforns [puppet] - 10https://gerrit.wikimedia.org/r/1364465 (https://phabricator.wikimedia.org/T442758) [16:09:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12523289 (10phaultfinder) [16:11:23] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy key for mforns [puppet] - 10https://gerrit.wikimedia.org/r/1364465 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [16:11:35] (03PS1) 10Daniel Kertesz: team-traffic: add the deploy-site tag to the lvs_rp_filter alert [alerts] - 10https://gerrit.wikimedia.org/r/1364468 (https://phabricator.wikimedia.org/T443771) [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:14:20] (03CR) 10DCausse: [C:03+1] cirrus: Provide short cluster names [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363463 (https://phabricator.wikimedia.org/T440376) (owner: 10Ebernhardson) [16:15:08] (03PS1) 10Muehlenhoff: Add FIDO key for rscout [puppet] - 10https://gerrit.wikimedia.org/r/1364470 (https://phabricator.wikimedia.org/T442758) [16:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:46] (03PS1) 10Hashar: pybal: migrate check_pybal_ipvs_diff to Python 3 [puppet] - 10https://gerrit.wikimedia.org/r/1364471 (https://phabricator.wikimedia.org/T443798) [16:17:20] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for rscout [puppet] - 10https://gerrit.wikimedia.org/r/1364470 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [16:17:23] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:18:30] (03CR) 10Vgutierrez: [C:03+1] team-traffic: add the deploy-site tag to the lvs_rp_filter alert [alerts] - 10https://gerrit.wikimedia.org/r/1364468 (https://phabricator.wikimedia.org/T443771) (owner: 10Daniel Kertesz) [16:19:42] (03CR) 10Daniel Kertesz: [C:03+2] team-traffic: add the deploy-site tag to the lvs_rp_filter alert [alerts] - 10https://gerrit.wikimedia.org/r/1364468 (https://phabricator.wikimedia.org/T443771) (owner: 10Daniel Kertesz) [16:20:13] 06SRE, 06Collaboration-Services: Document current expected SSH fingerprint for releases2003.codfw.wmnet - https://phabricator.wikimedia.org/T443537#12523397 (10Dzahn) a:03Dzahn [16:21:06] (03PS1) 10Eric Gardner: ShareCard: Log client errors with the topic "error.sharecard" [extensions/WikimediaCustomizations] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1364473 (https://phabricator.wikimedia.org/T439163) [16:21:25] 06SRE, 06Collaboration-Services: Document current expected SSH fingerprint for releases2003.codfw.wmnet - https://phabricator.wikimedia.org/T443537#12523406 (10Dzahn) I'll take care of this shortly. [16:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:23:03] (03Merged) 10jenkins-bot: team-traffic: add the deploy-site tag to the lvs_rp_filter alert [alerts] - 10https://gerrit.wikimedia.org/r/1364468 (https://phabricator.wikimedia.org/T443771) (owner: 10Daniel Kertesz) [16:27:30] (03CR) 10Jelto: [V:03+1 C:03+1] "looks reasonable, I did a successful test build on `build2004` and `kubernetes-client134_1.34.12-2_amd64.deb` installs fine on my machine" [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/1364330 (https://phabricator.wikimedia.org/T443455) (owner: 10JMeybohm) [16:32:22] (03PS2) 10Dzahn: ci: Configure docker to use the containerd snapshotter [puppet] - 10https://gerrit.wikimedia.org/r/1363509 (https://phabricator.wikimedia.org/T440668) (owner: 10Dduvall) [16:32:55] (03CR) 10FNegri: [C:03+2] striker: (production) bump container version [puppet] - 10https://gerrit.wikimedia.org/r/1364453 (https://phabricator.wikimedia.org/T443773) (owner: 10Thibaut Le Page) [16:33:36] (03CR) 10Jelto: [C:03+1] Add debian packaging from v1.34 branch [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/1364329 (https://phabricator.wikimedia.org/T443455) (owner: 10JMeybohm) [16:34:45] !log kamila@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [16:35:03] (03CR) 10Dzahn: "switching this looks like it will make every agent re-pull and old data stays around until pruned. so needs to have enough disk space for " [puppet] - 10https://gerrit.wikimedia.org/r/1363509 (https://phabricator.wikimedia.org/T440668) (owner: 10Dduvall) [16:35:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [16:35:20] !log installing krb5 security updates [16:35:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:35:37] !log kamila@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [16:35:59] (03PS1) 10Ssingh: service: set IPIP for sophroid [puppet] - 10https://gerrit.wikimedia.org/r/1364477 (https://phabricator.wikimedia.org/T441429) [16:36:05] jouncebot: nowandnext [16:36:05] No deployments scheduled for the next 0 hour(s) and 23 minute(s) [16:36:05] In 0 hour(s) and 23 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1700) [16:36:07] (03CR) 10Dduvall: "Just FYI, integration-agent-docker-1070.integration.eqiad1.wikimedia.cloud doesn't exist anymore. You can do `integration-agent-docker-109" [puppet] - 10https://gerrit.wikimedia.org/r/1363509 (https://phabricator.wikimedia.org/T440668) (owner: 10Dduvall) [16:36:27] going to do a sync-world to poke mw-parsoid [16:37:17] !log kamila@deploy2003 Started scap sync-world: switch mw-parsoid to php 8.5 - T439955 [16:37:20] T439955: [PHP 8.5 Upgrade] Ramp up production traffic - https://phabricator.wikimedia.org/T439955 [16:39:04] (03PS2) 10Eric Gardner: Add Creative Commons license icons for the share card [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358851 (https://phabricator.wikimedia.org/T439162) [16:39:22] (03CR) 10Jelto: [C:03+1] "wikikube should be fine for now. So the patch can stay as-is" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356269 (https://phabricator.wikimedia.org/T440988) (owner: 10Hnowlan) [16:39:33] (03PS3) 10Eric Gardner: testwiki: Enable the share card [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358852 (https://phabricator.wikimedia.org/T439162) [16:39:47] (03CR) 10Dduvall: "> switching this looks like it will make every agent re-pull and old data stays around until pruned. so needs to have enough disk space fo" [puppet] - 10https://gerrit.wikimedia.org/r/1363509 (https://phabricator.wikimedia.org/T440668) (owner: 10Dduvall) [16:39:57] (03PS3) 10Eric Gardner: Enable the share card on group1 Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358853 (https://phabricator.wikimedia.org/T439162) [16:40:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [16:40:16] (03PS3) 10Eric Gardner: Enable the share card on all Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358854 (https://phabricator.wikimedia.org/T439162) [16:40:20] !log kamila@deploy2003 Finished scap sync-world: switch mw-parsoid to php 8.5 - T439955 (duration: 03m 03s) [16:40:39] !log installing krb5 security updates [16:40:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:41:02] !log kamila@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [16:42:05] !log kamila@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [16:43:37] !log kamila@deploy2003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [16:44:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [16:44:26] !log kamila@deploy2003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [16:47:29] !log sukhe@cumin1004 START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::worker@eqiad [16:48:11] !log sukhe@cumin1004 END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: aux_k8s::worker@eqiad [16:49:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [16:49:11] (03PS1) 10Muehlenhoff: Add FIDO keys for milimetric and dsantamaria [puppet] - 10https://gerrit.wikimedia.org/r/1364488 (https://phabricator.wikimedia.org/T442758) [16:51:33] (03PS2) 10Muehlenhoff: Add FIDO keys for milimetric and dsantamaria [puppet] - 10https://gerrit.wikimedia.org/r/1364488 (https://phabricator.wikimedia.org/T442758) [16:52:46] FIRING: [6x] CertAlmostExpired: Certificate for service kubestagemaster2003:6443 is about to expire - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:53:34] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO keys for milimetric and dsantamaria [puppet] - 10https://gerrit.wikimedia.org/r/1364488 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [16:54:46] (03PS3) 10Dzahn: ci: Configure docker to use the containerd snapshotter [puppet] - 10https://gerrit.wikimedia.org/r/1363509 (https://phabricator.wikimedia.org/T440668) (owner: 10Dduvall) [16:54:51] (03CR) 10Dzahn: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1363509 (https://phabricator.wikimedia.org/T440668) (owner: 10Dduvall) [16:54:58] (03CR) 10Ssingh: [C:03+1] "Thanks for the patch! Looks good; will merge in a bit." [puppet] - 10https://gerrit.wikimedia.org/r/1364471 (https://phabricator.wikimedia.org/T443798) (owner: 10Hashar) [16:57:31] (03PS1) 10Hashar: utils: vcl_ec2_nets [puppet] - 10https://gerrit.wikimedia.org/r/1364492 (https://phabricator.wikimedia.org/T443798) [16:58:10] (03PS1) 10JavierMonton: ssh-key: add yubikey [puppet] - 10https://gerrit.wikimedia.org/r/1364493 [16:59:47] (03PS2) 10Hashar: utils: vcl_ec2_nets [puppet] - 10https://gerrit.wikimedia.org/r/1364492 (https://phabricator.wikimedia.org/T443798) [17:00:05] jhathaway and rzl: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for Puppet request window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1700). [17:00:05] dancy: A patch you scheduled for Puppet request window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [17:00:12] o/ [17:00:49] jhathaway: unusually, both of us are in the same meeting, sorry -- we might have to get back to you in a bit if that's okay [17:01:01] er [17:01:02] dancy: ^ [17:01:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:01:23] yep, that's fine. [17:04:24] !log installing haproxykafka 0.4.0 on cp7002 for testing (T443632) (T443640) [17:04:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:04:29] T443632: Support x-kapow-score header in HaproxyKafka - https://phabricator.wikimedia.org/T443632 [17:04:29] T443640: Upgrade HaproxyKafka to go1.27 - https://phabricator.wikimedia.org/T443640 [17:06:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:07:31] FIRING: [2x] Not accepting/receiving prefixes from anycast BGP peer: Alert for device asw1-b3-magru.mgmt.magru.wmnet - Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [17:07:36] (03CR) 10Muehlenhoff: [C:03+1] "Verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1364493 (owner: 10JavierMonton) [17:07:43] (03CR) 10Muehlenhoff: [C:03+2] ssh-key: add yubikey [puppet] - 10https://gerrit.wikimedia.org/r/1364493 (owner: 10JavierMonton) [17:11:45] (03PS1) 10Mpostoronca: tables-catalog: Add AbuseReview tables [puppet] - 10https://gerrit.wikimedia.org/r/1364499 (https://phabricator.wikimedia.org/T443206) [17:12:02] 10ops-codfw, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: move wikikube-worker servers in same rack - https://phabricator.wikimedia.org/T431585#12523731 (10Jhancock.wm) @Clement_Goubert hey wasn't on the original list (which i'll get to after the sre meetup) but can i move w... [17:12:48] 10ops-codfw, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: move wikikube-worker servers in same rack - https://phabricator.wikimedia.org/T431585#12523733 (10Jhancock.wm) [17:14:04] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [17:16:56] (03CR) 10Ssingh: "I will run the migrate-service-ipip cookbook but since it asks if this patch has been merged, will wait for that." [puppet] - 10https://gerrit.wikimedia.org/r/1364477 (https://phabricator.wikimedia.org/T441429) (owner: 10Ssingh) [17:18:37] (03PS1) 10Btullis: wdqs: Remove hosts that we will rename from firewall lists [puppet] - 10https://gerrit.wikimedia.org/r/1364508 (https://phabricator.wikimedia.org/T443792) [17:18:39] (03PS1) 10Btullis: dse_k8s: Add site.pp and preseed entries for six new workers [puppet] - 10https://gerrit.wikimedia.org/r/1364509 (https://phabricator.wikimedia.org/T443792) [17:18:41] (03PS1) 10Btullis: wdqs: Remove the wdqs::test role [puppet] - 10https://gerrit.wikimedia.org/r/1364510 (https://phabricator.wikimedia.org/T443792) [17:18:43] (03PS1) 10Btullis: dse_k8s: Add four former wdqs hosts to the clusters [puppet] - 10https://gerrit.wikimedia.org/r/1364511 (https://phabricator.wikimedia.org/T443792) [17:18:45] (03PS1) 10Btullis: dse_k8s: Remove the wdqs test nodes from the clusters [puppet] - 10https://gerrit.wikimedia.org/r/1364512 (https://phabricator.wikimedia.org/T443792) [17:18:48] (03PS1) 10Btullis: dse_k8s: Add the two former wdqs test hosts to the clusters [puppet] - 10https://gerrit.wikimedia.org/r/1364513 (https://phabricator.wikimedia.org/T443792) [17:19:04] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [17:19:42] (03CR) 10Btullis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1364508 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [17:19:42] (03CR) 10Hnowlan: [C:03+2] "Great, thanks!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356269 (https://phabricator.wikimedia.org/T440988) (owner: 10Hnowlan) [17:20:35] (03CR) 10Dzahn: "is there a integration-agent-docker-9999 ?a test instance on bookworm just created?" [puppet] - 10https://gerrit.wikimedia.org/r/1363509 (https://phabricator.wikimedia.org/T440668) (owner: 10Dduvall) [17:22:37] (03CR) 10Btullis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1364509 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [17:23:19] (03PS1) 10Hashar: backy2: set sql.py to python 3 [puppet] - 10https://gerrit.wikimedia.org/r/1364514 (https://phabricator.wikimedia.org/T443798) [17:23:56] (03PS2) 10Mpostoronca: tables-catalog: Add AbuseReview tables [puppet] - 10https://gerrit.wikimedia.org/r/1364499 (https://phabricator.wikimedia.org/T443206) [17:24:29] (03PS2) 10Hashar: backy2: set sql.py to python 3 [puppet] - 10https://gerrit.wikimedia.org/r/1364514 (https://phabricator.wikimedia.org/T443798) [17:28:03] (03CR) 10Muehlenhoff: [C:03+1] "Key has been verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1363504 (https://phabricator.wikimedia.org/T442759) (owner: 10Dwisehaupt) [17:28:05] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO SSH key for dwisehaupt [puppet] - 10https://gerrit.wikimedia.org/r/1363504 (https://phabricator.wikimedia.org/T442759) (owner: 10Dwisehaupt) [17:28:07] (03Merged) 10jenkins-bot: miscweb: add apps.wikipedia.org configuration [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356269 (https://phabricator.wikimedia.org/T440988) (owner: 10Hnowlan) [17:30:39] jayme@cumin2003 decommission (PID 3763457) is awaiting input [17:31:53] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need to test things on switch [17:32:00] 06SRE, 06Infrastructure-Foundations, 10netops: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12523829 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=37a3a0c5-db75-4a5f-b3af-203b15319a02) set by cmooney@cumin1004 for 3 days, 0:00:00 on 1 host(s)... [17:32:24] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad with reason: need to test things on switch [17:32:31] 06SRE, 06Infrastructure-Foundations, 10netops: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12523837 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a759d84c-b17a-40ad-9d3b-e8171fe3782b) set by cmooney@cumin1004 for 3 days, 0:00:00 on 1 host(s)... [17:33:03] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on ssw1-a[1,8]-eqiad with reason: need to test things on switch [17:33:17] 06SRE, 06Infrastructure-Foundations, 10netops: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12523840 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=83c9e035-90af-41e4-bc00-e2734e08a539) set by cmooney@cumin1004 for 5:00:00 on 2 host(s) and the... [17:34:39] !log jayme@cumin2003 START - Cookbook sre.hosts.decommission for hosts parsoidtest1001.eqiad.wmnet [17:34:44] jouncebot: nowandnext [17:34:44] For the next 0 hour(s) and 25 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1700) [17:34:44] In 0 hour(s) and 25 minute(s): Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1800) [17:34:44] In 0 hour(s) and 25 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1800) [17:34:54] (03PS3) 10Hashar: backy2: set sql.py to python 3 [puppet] - 10https://gerrit.wikimedia.org/r/1364514 (https://phabricator.wikimedia.org/T443798) [17:35:06] (03PS1) 10Hashar: burrow: migrate check_kafka_consumer_lag to Python3 [puppet] - 10https://gerrit.wikimedia.org/r/1364521 (https://phabricator.wikimedia.org/T443798) [17:35:25] 06SRE, 06Infrastructure-Foundations, 10netops: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12523870 (10cmooney) FWIW I did some more testing today, starting with no network-instances, irb sub-interface etc configured, and built the config up bit by bit. See P973... [17:35:34] !log hnowlan@deploy1003 helmfile [staging] START helmfile.d/services/miscweb: apply [17:35:46] !log hnowlan@deploy1003 helmfile [staging] DONE helmfile.d/services/miscweb: apply [17:35:58] (03CR) 10CI reject: [V:04-1] burrow: migrate check_kafka_consumer_lag to Python3 [puppet] - 10https://gerrit.wikimedia.org/r/1364521 (https://phabricator.wikimedia.org/T443798) (owner: 10Hashar) [17:36:44] (03PS1) 10JMeybohm: Decom parsoidtest1001.eqiad.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/1364522 (https://phabricator.wikimedia.org/T421484) [17:39:13] (03CR) 10Dduvall: "Oh I hadn't realized that integration-agent-qemu-1004 had docker. I can create a new trixie based instance to replace that today. In the m" [puppet] - 10https://gerrit.wikimedia.org/r/1363509 (https://phabricator.wikimedia.org/T440668) (owner: 10Dduvall) [17:39:14] (03PS1) 10Hashar: mediawiki: set firejail python script shebang to python3 [puppet] - 10https://gerrit.wikimedia.org/r/1364525 (https://phabricator.wikimedia.org/T443798) [17:39:23] !log reboot lsw1-b3-eqiad to downgrade to SR-Linux 26.3.3 T443687 [17:39:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:39:26] T443687: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687 [17:39:37] (03PS4) 10Dduvall: ci: Configure docker to use the containerd snapshotter [puppet] - 10https://gerrit.wikimedia.org/r/1363509 (https://phabricator.wikimedia.org/T440668) [17:40:00] !log hnowlan@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [17:40:14] !log jayme@cumin2003 START - Cookbook sre.dns.netbox [17:40:42] !log hnowlan@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [17:40:55] (03CR) 10Dduvall: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1363509 (https://phabricator.wikimedia.org/T440668) (owner: 10Dduvall) [17:41:28] jouncebot: nowandnext [17:41:28] For the next 0 hour(s) and 18 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1700) [17:41:28] In 0 hour(s) and 18 minute(s): Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1800) [17:41:28] In 0 hour(s) and 18 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1800) [17:43:08] !log hnowlan@deploy1003 helmfile [eqiad] START helmfile.d/admin 'apply'. [17:43:41] !log hnowlan@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'apply'. [17:45:04] !log jayme@cumin2003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003" [17:45:24] !log hnowlan@deploy1003 helmfile [codfw] START helmfile.d/admin 'apply'. [17:45:29] !log jayme@cumin2003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003" [17:45:29] !log jayme@cumin2003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [17:45:31] !log jayme@cumin2003 END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts parsoidtest1001.eqiad.wmnet [17:45:56] (03CR) 10JMeybohm: [C:03+2] Decom parsoidtest1001.eqiad.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/1364522 (https://phabricator.wikimedia.org/T421484) (owner: 10JMeybohm) [17:46:26] !log hnowlan@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'apply'. [17:46:46] (03PS2) 10Hashar: burrow: migrate check_kafka_consumer_lag to Python3 [puppet] - 10https://gerrit.wikimedia.org/r/1364521 (https://phabricator.wikimedia.org/T443798) [17:47:01] 10ops-eqiad, 06DC-Ops, 10decommission-hardware, 06Infrastructure-Foundations, and 3 others: Reclaim parsoidtest1001 as a Ganeti node - https://phabricator.wikimedia.org/T421484#12523907 (10JMeybohm) a:05jasmine_→03None I don't think I fully understand the history around this but since the server has be... [17:48:17] 10ops-eqiad, 06DC-Ops, 10decommission-hardware, 06Infrastructure-Foundations, 13Patch-For-Review: Reclaim parsoidtest1001 as a Ganeti node - https://phabricator.wikimedia.org/T421484#12523928 (10JMeybohm) [17:51:30] (03PS2) 10Jforrester: tables-catalogue: List abstractwiki_integrations [puppet] - 10https://gerrit.wikimedia.org/r/1362124 (https://phabricator.wikimedia.org/T443295) [17:51:36] (03CR) 10Ladsgroup: [V:03+2 C:03+2] tables-catalogue: List abstractwiki_integrations [puppet] - 10https://gerrit.wikimedia.org/r/1362124 (https://phabricator.wikimedia.org/T443295) (owner: 10Jforrester) [17:51:46] (03PS1) 10Bking: WIP: Introduce auth proxy to Matomo chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364530 (https://phabricator.wikimedia.org/T436186) [17:51:51] !log hnowlan@deploy1003 helmfile [eqiad] START helmfile.d/services/miscweb: apply [17:51:58] jouncebot: nowandnext [17:51:58] For the next 0 hour(s) and 8 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1700) [17:51:58] In 0 hour(s) and 8 minute(s): Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1800) [17:51:59] In 0 hour(s) and 8 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1800) [17:52:02] !log hnowlan@deploy1003 helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [17:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 98.85% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [17:52:12] (03PS2) 10WMDE-leszek: Restore running WMDE Analytics scripts [puppet] - 10https://gerrit.wikimedia.org/r/1362194 (https://phabricator.wikimedia.org/T443525) [17:52:39] (03CR) 10Ladsgroup: [C:03+2] "I have checked and the php packages on all stat machines are the latest. I'm pushing this but again, please migrate to DAGs" [puppet] - 10https://gerrit.wikimedia.org/r/1362194 (https://phabricator.wikimedia.org/T443525) (owner: 10WMDE-leszek) [17:52:42] (03CR) 10Ladsgroup: [V:03+2 C:03+2] Restore running WMDE Analytics scripts [puppet] - 10https://gerrit.wikimedia.org/r/1362194 (https://phabricator.wikimedia.org/T443525) (owner: 10WMDE-leszek) [17:52:51] !log hnowlan@deploy1003 helmfile [codfw] START helmfile.d/services/miscweb: apply [17:53:06] !log hnowlan@deploy1003 helmfile [codfw] DONE helmfile.d/services/miscweb: apply [17:53:09] (03PS3) 10Ladsgroup: errorpage: Hide Wikimedia branding outside production [puppet] - 10https://gerrit.wikimedia.org/r/1363420 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [17:53:19] (03CR) 10Ladsgroup: [C:03+2] errorpage: Hide Wikimedia branding outside production [puppet] - 10https://gerrit.wikimedia.org/r/1363420 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [17:53:24] (03CR) 10Ladsgroup: [V:03+2 C:03+2] errorpage: Hide Wikimedia branding outside production [puppet] - 10https://gerrit.wikimedia.org/r/1363420 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [17:54:28] (03Abandoned) 10Ladsgroup: mediawiki: Better error page layout on mobile devices [puppet] - 10https://gerrit.wikimedia.org/r/405058 (https://phabricator.wikimedia.org/T182247) (owner: 10Phantom42) [17:54:34] PROBLEM - Host lsw1-b3-eqiad IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [17:54:34] PROBLEM - Host lsw1-b3-eqiad.mgmt is DOWN: PING CRITICAL - Packet loss = 100% [17:57:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [17:59:36] RECOVERY - Host lsw1-b3-eqiad.mgmt is UP: PING OK - Packet loss = 0%, RTA = 2.16 ms [17:59:36] RECOVERY - Host lsw1-b3-eqiad IPv6 is UP: PING OK - Packet loss = 0%, RTA = 0.77 ms [17:59:55] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad IPv6,lsw1-b3-eqiad.mgmt with reason: need to test things on switch [18:00:05] 06SRE, 06Infrastructure-Foundations, 10netops: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12523955 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=866cbc1d-696d-4894-9d5d-2aa4aa1c50da) set by cmooney@cumin1004 for 3 days, 0:00:00 on 2 host(s)... [18:00:05] bd808: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1800). [18:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1800) [18:00:08] ^^ sry I should have downtimed this swtich for v6 and mgmt too, I've been working on it [18:00:42] I have 1-2 things for my window today. [18:02:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:03:48] !log ladsgroup@deploy2003 Started scap sync-world: Push small changes for error pages [18:04:55] !log ladsgroup@deploy2003 ladsgroup: Push small changes for error pages synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [18:05:24] FIRING: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:06:42] (03CR) 10MusikAnimal: admin: adding Leon Ziemba as contact for sd (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1364022 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [18:06:42] !log ladsgroup@deploy2003 ladsgroup: Continuing with deployment [18:08:11] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance [18:08:13] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2154 (T441633)', diff saved to https://phabricator.wikimedia.org/P97377 and previous config saved to /var/cache/conftool/dbconfig/20261105-180812-ladsgroup.json [18:08:47] !log ladsgroup@deploy2003 Finished scap sync-world: Push small changes for error pages (duration: 05m 08s) [18:09:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:09:09] (03CR) 10Klausman: [C:03+1] dse_k8s: Add the two former wdqs test hosts to the clusters [puppet] - 10https://gerrit.wikimedia.org/r/1364513 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [18:09:22] (03PS4) 10JMeybohm: Update debian/ to gbp/pbuilder builds [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/1364330 (https://phabricator.wikimedia.org/T443455) [18:09:22] (03PS1) 10JMeybohm: Set Node.Spec.Unschedulable registering with unschedulable taint [debs/kubernetes] - 10https://gerrit.wikimedia.org/r/1364546 (https://phabricator.wikimedia.org/T443455) [18:09:28] (03CR) 10Klausman: [C:03+1] dse_k8s: Remove the wdqs test nodes from the clusters [puppet] - 10https://gerrit.wikimedia.org/r/1364512 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [18:10:24] FIRING: [2x] SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:10:25] (03CR) 10Klausman: [C:03+1] dse_k8s: Add four former wdqs hosts to the clusters [puppet] - 10https://gerrit.wikimedia.org/r/1364511 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [18:11:09] (03CR) 10Klausman: [C:03+1] wdqs: Remove the wdqs::test role [puppet] - 10https://gerrit.wikimedia.org/r/1364510 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [18:11:50] (03CR) 10Klausman: [C:03+1] dse_k8s: Add site.pp and preseed entries for six new workers [puppet] - 10https://gerrit.wikimedia.org/r/1364509 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [18:12:23] (03CR) 10Klausman: [C:03+1] wdqs: Remove hosts that we will rename from firewall lists [puppet] - 10https://gerrit.wikimedia.org/r/1364508 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [18:13:24] !log dancy@deploy2003 Started scap sync-world: Testing T411063 [18:14:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [18:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [18:15:25] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2154 (T441633)', diff saved to https://phabricator.wikimedia.org/P97378 and previous config saved to /var/cache/conftool/dbconfig/20261105-181524-ladsgroup.json [18:16:45] (03CR) 10Bking: [C:03+1] wdqs: Remove hosts that we will rename from firewall lists [puppet] - 10https://gerrit.wikimedia.org/r/1364508 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [18:23:15] (03PS1) 10BryanDavis: developer-portal: Bump container to 2026-11-05-120739-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364568 [18:25:26] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97379 and previous config saved to /var/cache/conftool/dbconfig/20261105-182526-ladsgroup.json [18:26:48] (03CR) 10BryanDavis: [C:03+2] developer-portal: Bump container to 2026-11-05-120739-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364568 (owner: 10BryanDavis) [18:28:04] (03Merged) 10jenkins-bot: developer-portal: Bump container to 2026-11-05-120739-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364568 (owner: 10BryanDavis) [18:28:33] (03PS1) 10Sbisson: admin: add fido key for sbisson [puppet] - 10https://gerrit.wikimedia.org/r/1364574 (https://phabricator.wikimedia.org/T442758) [18:31:13] !log bd808@deploy2003 helmfile [staging] START helmfile.d/services/developer-portal: apply [18:31:34] !log bd808@deploy2003 helmfile [staging] DONE helmfile.d/services/developer-portal: apply [18:31:50] !log bd808@deploy2003 helmfile [codfw] START helmfile.d/services/developer-portal: apply [18:32:08] !log bd808@deploy2003 helmfile [codfw] DONE helmfile.d/services/developer-portal: apply [18:32:19] !log bd808@deploy2003 helmfile [eqiad] START helmfile.d/services/developer-portal: apply [18:32:33] !log bd808@deploy2003 helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply [18:35:22] !log dancy@deploy2003 dancy: Testing T411063 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [18:35:28] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97380 and previous config saved to /var/cache/conftool/dbconfig/20261105-183527-ladsgroup.json [18:41:03] (03CR) 10Dzahn: [C:03+1] "sounds good to me. ready to merge then?" [puppet] - 10https://gerrit.wikimedia.org/r/1363509 (https://phabricator.wikimedia.org/T440668) (owner: 10Dduvall) [18:42:33] !log dancy@deploy2003 dancy: Continuing with deployment [18:43:21] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 06Infrastructure-Foundations: Reclaim parsoidtest1001 as a Ganeti node - https://phabricator.wikimedia.org/T421484#12524140 (10Dzahn) The host still appeared in other configs including CDN which is why it had not been decom'ed yet. Per comments on h... [18:45:28] (03CR) 10Andrew Bogott: "For context: this is a file forked from upstream with only a couple of lines changed. So we'd like to keep the diff minimal." [puppet] - 10https://gerrit.wikimedia.org/r/1364514 (https://phabricator.wikimedia.org/T443798) (owner: 10Hashar) [18:45:30] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2154 (T441633)', diff saved to https://phabricator.wikimedia.org/P97381 and previous config saved to /var/cache/conftool/dbconfig/20261105-184529-ladsgroup.json [18:45:38] (03CR) 10Andrew Bogott: [C:03+2] backy2: set sql.py to python 3 [puppet] - 10https://gerrit.wikimedia.org/r/1364514 (https://phabricator.wikimedia.org/T443798) (owner: 10Hashar) [18:45:49] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance [18:45:51] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2163 (T441633)', diff saved to https://phabricator.wikimedia.org/P97382 and previous config saved to /var/cache/conftool/dbconfig/20261105-184550-ladsgroup.json [18:49:30] abogott: Would you mind taking a look at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1361577 ? [18:50:55] (03CR) 10Andrew Bogott: "On second thought... maybe we should just remove it from linting. I can't decide." [puppet] - 10https://gerrit.wikimedia.org/r/1364514 (https://phabricator.wikimedia.org/T443798) (owner: 10Hashar) [18:51:31] (03PS1) 10Dzahn: codesearch: clone from external code hosts via the webproxy in production [puppet] - 10https://gerrit.wikimedia.org/r/1364598 (https://phabricator.wikimedia.org/T268199) [18:52:19] (03CR) 10CI reject: [V:04-1] codesearch: clone from external code hosts via the webproxy in production [puppet] - 10https://gerrit.wikimedia.org/r/1364598 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [18:54:02] (03CR) 10Gmodena: [C:03+1] wdqs: Remove the wdqs::test role [puppet] - 10https://gerrit.wikimedia.org/r/1364510 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [18:54:09] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b3-eqiad [18:54:11] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2163 (T441633)', diff saved to https://phabricator.wikimedia.org/P97383 and previous config saved to /var/cache/conftool/dbconfig/20261105-185411-ladsgroup.json [18:54:29] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-eqiad [18:54:46] (03PS2) 10Dzahn: codesearch: clone from external code hosts via the webproxy in production [puppet] - 10https://gerrit.wikimedia.org/r/1364598 (https://phabricator.wikimedia.org/T268199) [18:56:35] (03CR) 10Dzahn: "don't forget the wdqs-test-roots user group itself in the admin module" [puppet] - 10https://gerrit.wikimedia.org/r/1364510 (https://phabricator.wikimedia.org/T443792) (owner: 10Btullis) [18:57:33] !log dancy@deploy2003 Finished scap sync-world: Testing T411063 (duration: 44m 09s) [19:00:05] jnuche and dduvall: gettimeofday() says it's time for MediaWiki train - Utc-7 Version. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T1900) [19:02:29] (03PS1) 10BryanDavis: toolhub: bump container to 2026-11-03-201105-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364618 [19:04:13] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97384 and previous config saved to /var/cache/conftool/dbconfig/20261105-190412-ladsgroup.json [19:05:37] (03CR) 10BryanDavis: [C:03+2] toolhub: bump container to 2026-11-03-201105-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364618 (owner: 10BryanDavis) [19:07:01] (03Merged) 10jenkins-bot: toolhub: bump container to 2026-11-03-201105-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1364618 (owner: 10BryanDavis) [19:10:37] !log bd808@deploy2003 helmfile [staging] START helmfile.d/services/toolhub: apply [19:11:11] !log bd808@deploy2003 helmfile [staging] DONE helmfile.d/services/toolhub: apply [19:12:06] !log bd808@deploy2003 helmfile [codfw] START helmfile.d/services/toolhub: apply [19:12:53] !log bd808@deploy2003 helmfile [codfw] DONE helmfile.d/services/toolhub: apply [19:13:51] !log bd808@deploy2003 helmfile [eqiad] START helmfile.d/services/toolhub: apply [19:14:15] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97385 and previous config saved to /var/cache/conftool/dbconfig/20261105-191414-ladsgroup.json [19:14:43] !log bd808@deploy2003 helmfile [eqiad] DONE helmfile.d/services/toolhub: apply [19:17:48] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for dse-k8s-worker1005.mgmt:22 - https://phabricator.wikimedia.org/T443767#12524237 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF E1 network switch is seemingly having issues. I have reseated the cable and it came back online [19:21:35] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for an-worker1235.mgmt:22 - https://phabricator.wikimedia.org/T443766#12524241 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF Replaced cable [19:24:17] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2163 (T441633)', diff saved to https://phabricator.wikimedia.org/P97386 and previous config saved to /var/cache/conftool/dbconfig/20261105-192416-ladsgroup.json [19:24:25] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance [19:24:27] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2164 (T441633)', diff saved to https://phabricator.wikimedia.org/P97387 and previous config saved to /var/cache/conftool/dbconfig/20261105-192426-ladsgroup.json [19:25:22] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for cirrussearch1090.mgmt:22 - https://phabricator.wikimedia.org/T443764#12524265 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF Replaced cable [19:28:09] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for an-worker1181.mgmt:22 - https://phabricator.wikimedia.org/T443763#12524282 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF Replaced cable [19:28:34] (03CR) 10Dduvall: "Yes, please and thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1363509 (https://phabricator.wikimedia.org/T440668) (owner: 10Dduvall) [19:29:48] (03CR) 10Hashar: "Given it is an upstream change and it is already running under python 3, it is fine to keep the `python` shebang." [puppet] - 10https://gerrit.wikimedia.org/r/1364514 (https://phabricator.wikimedia.org/T443798) (owner: 10Hashar) [19:31:45] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2164 (T441633)', diff saved to https://phabricator.wikimedia.org/P97388 and previous config saved to /var/cache/conftool/dbconfig/20261105-193144-ladsgroup.json [19:41:20] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for backup1010.mgmt:22 - https://phabricator.wikimedia.org/T443762#12524326 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF Replaced cable [19:41:47] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97389 and previous config saved to /var/cache/conftool/dbconfig/20261105-194146-ladsgroup.json [19:49:04] (03CR) 10Thcipriani: [C:03+1] LabsServices: Migrate ms-fe to service RR [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364235 (https://phabricator.wikimedia.org/T436476) (owner: 10Southparkfan) [19:51:49] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97390 and previous config saved to /var/cache/conftool/dbconfig/20261105-195148-ladsgroup.json [19:55:42] 10ops-eqiad, 06SRE, 06DC-Ops: Interface eno8303 on rdb1014 has the wrong speed - https://phabricator.wikimedia.org/T443165#12524351 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF Swapped out optic [19:55:52] !log bking@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [19:56:04] !log bking@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [20:01:51] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2164 (T441633)', diff saved to https://phabricator.wikimedia.org/P97391 and previous config saved to /var/cache/conftool/dbconfig/20261105-200150-ladsgroup.json [20:02:10] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance [20:02:12] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2165 (T441633)', diff saved to https://phabricator.wikimedia.org/P97392 and previous config saved to /var/cache/conftool/dbconfig/20261105-200211-ladsgroup.json [20:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:05:32] !log bking@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply [20:06:22] !log bking@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply [20:09:12] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2165 (T441633)', diff saved to https://phabricator.wikimedia.org/P97393 and previous config saved to /var/cache/conftool/dbconfig/20261105-200911-ladsgroup.json [20:12:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.542% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [20:17:23] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [20:19:14] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97394 and previous config saved to /var/cache/conftool/dbconfig/20261105-201913-ladsgroup.json [20:19:23] (03CR) 10Cathal Mooney: [C:03+1] "LGTM Nice!" [homer/public] - 10https://gerrit.wikimedia.org/r/1360452 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [20:20:38] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12524413 (10wiki_willy) @Marostegui - I just poked them again on email. Did you change the recommended bio settings that @Papaul brought up to Performance for this server already? [20:21:21] (03CR) 10Cathal Mooney: [C:03+1] "Nice one :)" [cookbooks] - 10https://gerrit.wikimedia.org/r/1347441 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [20:22:14] (03PS1) 10Bking: dse-k8s-eqiad: Add ATS mapping for matomo-next [puppet] - 10https://gerrit.wikimedia.org/r/1364735 (https://phabricator.wikimedia.org/T436186) [20:23:30] (03CR) 10Cathal Mooney: [C:03+1] "LGTM nice work!" [software/homer/deploy] - 10https://gerrit.wikimedia.org/r/1360446 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [20:25:28] 06SRE, 10LDAP-Access-Requests: Grant Access to ldap/nda for Aroraakhil - https://phabricator.wikimedia.org/T443439#12524449 (10KFrancis) Hi all, I'm in the process of extending Aroraakhil (@Aroraakhil)'s MOU/NDA. I'll confirm once it's signed. Thanks! [20:29:16] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97395 and previous config saved to /var/cache/conftool/dbconfig/20261105-202915-ladsgroup.json [20:30:39] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12524460 (10Marostegui) >>! In T431115#12524412, @wiki_willy wrote: > @Marostegui - I just poked them again on email. Did you change the recommended bio settings that @Papaul brought up to P... [20:31:48] 06SRE, 06Infrastructure-Foundations, 10netops: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12524464 (10cmooney) In an effort to try and work out what version things still worked in I downgraded lsw1-b3-eqiad to 26.7.1, same issue. I then went to 26.3.3 - same th... [20:31:52] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 05 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340815 (https://phabricator.wikimedia.org/T430731) (owner: 10Pppery) [20:32:16] (03CR) 10Hashar: "I have amended my change to indicate this script can be skipped https://gerrit.wikimedia.org/r/c/operations/puppet/+/1364487/6..7//COMMIT_" [puppet] - 10https://gerrit.wikimedia.org/r/1364514 (https://phabricator.wikimedia.org/T443798) (owner: 10Hashar) [20:32:21] (03Abandoned) 10Hashar: backy2: set sql.py to python 3 [puppet] - 10https://gerrit.wikimedia.org/r/1364514 (https://phabricator.wikimedia.org/T443798) (owner: 10Hashar) [20:34:44] 10ops-eqiad, 06DC-Ops: Unresponsive management for backup1010.mgmt:22 - https://phabricator.wikimedia.org/T443829 (10phaultfinder) 03NEW [20:38:41] 06SRE, 06Infrastructure-Foundations, 10netops: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12524495 (10cmooney) p:05High→03Medium [20:39:15] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: failover lvs1019 to lvs1020 to add new vlan sub-ints [20:39:18] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2165 (T441633)', diff saved to https://phabricator.wikimedia.org/P97396 and previous config saved to /var/cache/conftool/dbconfig/20261105-203917-ladsgroup.json [20:39:27] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12524496 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8099810e-6408-4be2-82b4-5240b5d6c531) set by cmooney@cumi... [20:39:37] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance [20:39:39] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97397 and previous config saved to /var/cache/conftool/dbconfig/20261105-203938-ladsgroup.json [20:42:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [20:42:56] 06SRE, 10SRE-Access-Requests, 06Wikimedia Enterprise: Request for additional rights for SanchezA-WMF to manage analytics - https://phabricator.wikimedia.org/T443636#12524512 (10SanchezA-WMF) Hi, where I can find the: L3 Wikimedia Server Access Responsibilities document that I need to sign to complete the SRE... [20:44:28] !log disable pybal service on lvs1019 to failover to lvs1020 ahead of adding new vlan ints to the former T440859 [20:44:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:44:31] T440859: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859 [20:46:42] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97398 and previous config saved to /var/cache/conftool/dbconfig/20261105-204641-ladsgroup.json [20:50:29] (03CR) 10Pppery: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361803 (https://phabricator.wikimedia.org/T443174) (owner: 10Rafka Aditia) [20:51:27] (03PS2) 10Cathal Mooney: lvs1019: add new vlan ints for rows a/b and move existing [puppet] - 10https://gerrit.wikimedia.org/r/1361872 (https://phabricator.wikimedia.org/T440859) [20:52:46] FIRING: [6x] CertAlmostExpired: Certificate for service kubestagemaster2003:6443 is about to expire - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [20:53:04] (03PS1) 10DLynch: VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364770 (https://phabricator.wikimedia.org/T442767) [20:53:18] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, November 05 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364770 (https://phabricator.wikimedia.org/T442767) (owner: 10DLynch) [20:56:37] !log mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_integrations.sql # T443295 [20:56:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:56:41] T443295: WikiLambda: New x1 table abstractwiki_integrations (virtual-abstractwiki domain) - https://phabricator.wikimedia.org/T443295 [20:56:44] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97399 and previous config saved to /var/cache/conftool/dbconfig/20261105-205643-ladsgroup.json [20:58:50] (03CR) 10Volans: [C:03+1] "LGTM" [software/debmonitor] - 10https://gerrit.wikimedia.org/r/1358866 (https://phabricator.wikimedia.org/T240457) (owner: 10Lizimedia) [20:59:32] 10SRE-tools, 06Infrastructure-Foundations, 13Patch-For-Review: Debmonitor: backend-changeable settings are stored in the browser's session storage - https://phabricator.wikimedia.org/T240457#12524621 (10Volans) Reviewed the code, LGTM. [21:00:04] (03PS1) 10Krinkle: varnish: Don't strip browser Cache-Control:maxage from action=raw [puppet] - 10https://gerrit.wikimedia.org/r/1364773 (https://phabricator.wikimedia.org/T279120) [21:00:05] RoanKattouw, urbanecm, TheresNoTime, and cjming: OwO what's this, a deployment window?? UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T2100). nyaa~ [21:00:05] Southparkfan, Pppery, and kemayo: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [21:00:09] Here [21:00:13] o/ [21:00:17] Hi [21:00:46] My change should be fast to deploy because it only touches Beta Cluster config - OK if I merge that one first? [21:00:50] (cc thcipriani) [21:00:56] o/ [21:01:03] (03CR) 10CI reject: [V:04-1] varnish: Don't strip browser Cache-Control:maxage from action=raw [puppet] - 10https://gerrit.wikimedia.org/r/1364773 (https://phabricator.wikimedia.org/T279120) (owner: 10Krinkle) [21:01:21] Mine has nothing to test on staging, so it should be fairly bundleable in with other config changes. [21:02:06] (03CR) 10TrainBranchBot: [C:03+2] "Approved by southparkfan@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364235 (https://phabricator.wikimedia.org/T436476) (owner: 10Southparkfan) [21:02:58] (03Merged) 10jenkins-bot: LabsServices: Migrate ms-fe to service RR [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364235 (https://phabricator.wikimedia.org/T436476) (owner: 10Southparkfan) [21:03:54] Alright, mine's finished now. Go ahead. [21:04:17] Southparkfan: tada! first click of the button is always the hardest click of the button :) [21:04:28] glad you're deploying! [21:04:58] Starting to get the hang of it :) [21:05:42] (03PS2) 10Krinkle: varnish: Don't strip browser Cache-Control:maxage from action=raw [puppet] - 10https://gerrit.wikimedia.org/r/1364773 (https://phabricator.wikimedia.org/T279120) [21:05:43] Kemayo: you deploying your own? Pppery you'll need a deployer, correct? [21:05:48] yep I will [21:06:02] (03PS3) 10Krinkle: varnish: Don't strip browser Cache-Control:maxage from action=raw [puppet] - 10https://gerrit.wikimedia.org/r/1364773 (https://phabricator.wikimedia.org/T279120) [21:06:03] ack [21:06:11] I am happy to deploy mine, if nobody wants to include it. I can throw in Pppery's with mine if they'd like. [21:06:23] go ahead [21:06:27] Kemayo: that'd be super <# [21:06:30] er <3 [21:06:35] I'll want to do some quick testing, and you may want to run namespaceDupes afterword [21:06:46] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97400 and previous config saved to /var/cache/conftool/dbconfig/20261105-210645-ladsgroup.json [21:06:59] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364770 (https://phabricator.wikimedia.org/T442767) (owner: 10DLynch) [21:07:00] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340815 (https://phabricator.wikimedia.org/T430731) (owner: 10Pppery) [21:07:31] FIRING: [2x] Not accepting/receiving prefixes from anycast BGP peer: Alert for device asw1-b3-magru.mgmt.magru.wmnet - Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [21:07:48] (03Merged) 10jenkins-bot: VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364770 (https://phabricator.wikimedia.org/T442767) (owner: 10DLynch) [21:07:51] (03Merged) 10jenkins-bot: Add new namespace for zh_min_nanwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1340815 (https://phabricator.wikimedia.org/T430731) (owner: 10Pppery) [21:08:05] !log kemayo@deploy2003 Started scap sync-world: Backport for [[gerrit:1364770|VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815|Add new namespace for zh_min_nanwiki (T430731)]] [21:08:10] T442767: Deploy Headless VE to a second Wikipedia - https://phabricator.wikimedia.org/T442767 [21:08:11] T430731: New namespace for zh-min-nan Wikipedia: Hàn-jī - https://phabricator.wikimedia.org/T430731 [21:12:08] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12524688 (10Papaul) @VRiley-WMF can you please get this host back online so we can access it for troubleshooting ? Thanks [21:12:52] Pppery: Just let me know when your tests are done. [21:13:00] Has it reached the testservers yet? [21:13:07] (I was expecting logmsgbot to point out when it did) [21:13:27] but my last time watching a deployment was months ago so maybe something changed [21:13:52] Oh, sorry, it's at 93% right now -- I mixed up my notifications and thought it had just announced being there. [21:13:58] !log kemayo@deploy2003 pppery, kemayo: Backport for [[gerrit:1364770|VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815|Add new namespace for zh_min_nanwiki (T430731)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:14:02] T442767: Deploy Headless VE to a second Wikipedia - https://phabricator.wikimedia.org/T442767 [21:14:03] T430731: New namespace for zh-min-nan Wikipedia: Hàn-jī - https://phabricator.wikimedia.org/T430731 [21:14:03] There we go. [21:14:05] indeed [21:14:08] speak of the spiderpig... [21:14:20] (03CR) 10Cathal Mooney: [C:03+2] lvs1019: add new vlan ints for rows a/b and move existing [puppet] - 10https://gerrit.wikimedia.org/r/1361872 (https://phabricator.wikimedia.org/T440859) (owner: 10Cathal Mooney) [21:15:10] (03CR) 10DLynch: [C:03+1] "Only real problem is that there'll be increased load during the switchover, I think, as the old queue drains while the new queue also exis" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362092 (https://phabricator.wikimedia.org/T443442) (owner: 10Ottomata) [21:15:13] Looks good [21:15:24] Very cool to be able to watch this happening in real-time in SpierPig [21:15:28] remember to run namespaceDupes when it's done deploying. I don't expect it to find anything, but better safe than sorry [21:15:29] *SpiderPig [21:15:32] !log kemayo@deploy2003 pppery, kemayo: Continuing with deployment [21:15:35] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12524723 (10VRiley-WMF) @Papaul can we coordinate a time on this? The problem is the number of times I have been able to get this server back online, it will just pull itself back off line fo... [21:16:16] It'll be my first time actually running namespaceDupes, but there's some fairly clearly spelled out instructions on the deployers pages, so it looks entirely doable. [21:16:47] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97401 and previous config saved to /var/cache/conftool/dbconfig/20261105-211647-ladsgroup.json [21:17:07] oh good, those instructions do look up-to-date :) [21:17:07] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance [21:17:09] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97402 and previous config saved to /var/cache/conftool/dbconfig/20261105-211708-ladsgroup.json [21:18:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:19:37] thcipriani: I had to pick between the ones on wikitech:Adding_namespaces and wikitech:Backport_windows/Deployers, and the latter looked more appropriate. (With the former being a little more comprehensive about what to do if it goes wrong, I guess.) [21:19:51] !log cmooney@cumin1004 START - Cookbook sre.hosts.reboot-single for host lvs1019.eqiad.wmnet [21:20:06] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12524731 (10ops-monitoring-bot) Host lvs1019.eqiad.wmnet rebooted by cmooney@cumin1004 with reason: reconfigure vlan ints [21:21:35] * thcipriani updates Adding_namespaces for future Kemayo s [21:21:43] (03CR) 10Lizimedia: [C:03+2] don't restore column settings [software/debmonitor] - 10https://gerrit.wikimedia.org/r/1358866 (https://phabricator.wikimedia.org/T240457) (owner: 10Lizimedia) [21:22:26] I figured I'd wait until I'd actually run it before updating anything. Didn't want to be *too* bold. 🤩 [21:22:30] (03Merged) 10jenkins-bot: don't restore column settings [software/debmonitor] - 10https://gerrit.wikimedia.org/r/1358866 (https://phabricator.wikimedia.org/T240457) (owner: 10Lizimedia) [21:23:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:24:27] !log kemayo@deploy2003 Finished scap sync-world: Backport for [[gerrit:1364770|VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815|Add new namespace for zh_min_nanwiki (T430731)]] (duration: 16m 22s) [21:24:27] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97403 and previous config saved to /var/cache/conftool/dbconfig/20261105-212426-ladsgroup.json [21:24:32] T442767: Deploy Headless VE to a second Wikipedia - https://phabricator.wikimedia.org/T442767 [21:24:32] T430731: New namespace for zh-min-nan Wikipedia: Hàn-jī - https://phabricator.wikimedia.org/T430731 [21:24:37] Okay, doing the dry-run of namespaceDupes now. [21:24:38] heh, fair, tho I don't think the old ways work in our k8s-y future. [21:24:54] our *current* k8s-y future [21:25:09] (03PS1) 10Jdrewniak: Enable ReadingLists custom lists on all Wikipedias. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364782 (https://phabricator.wikimedia.org/T441372) [21:25:42] Pppery: it feels it'll need to fix 22 pages. Do you want to look at anything before I tell it to go ahead? [21:25:56] I would like to see the output [21:26:25] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 09 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364782 (https://phabricator.wikimedia.org/T441372) (owner: 10Jdrewniak) [21:26:43] !log cmooney@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1019.eqiad.wmnet [21:26:54] Pppery: sure, see it at https://phabricator.wikimedia.org/P97404 [21:27:18] Oh that's just pagelinks entries, those are derived data. Go ahead [21:27:38] When you said "22 pages" I had thought you meant the first output line which is "0 pages to fix, 0 were resolvable." [21:27:41] That's the important one [21:28:28] !log kemayo@deploy2003 mwscript-k8s job started: namespaceDupes zh_min_nanwiki --fix # T430731 [21:28:52] Sorry, yeah, that was quite imprecise of me [21:29:00] Pppery: Anyway, done. [21:29:50] (03CR) 10Ottomata: "Hm. I was going to say no big deal but:" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362092 (https://phabricator.wikimedia.org/T443442) (owner: 10Ottomata) [21:30:24] And I think that concludes the window [21:30:40] !log re-enable BGP for pybal on lvs1019 to fail back from lvs1020 after vlan add T440859 [21:30:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:30:43] T440859: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859 [21:32:50] Kemayo: thanks for verifying the deployer page instructions are up-to-date :) [21:33:50] !log ssastry@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [21:34:15] (03CR) 10DLynch: [C:03+1] "I'd assume the spikes are from rapid bursts of editing on large pages. We've got few enough replicas that if a cluster of pages that take " [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362092 (https://phabricator.wikimedia.org/T443442) (owner: 10Ottomata) [21:34:29] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97405 and previous config saved to /var/cache/conftool/dbconfig/20261105-213428-ladsgroup.json [21:34:32] !log ssastry@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [21:35:36] (03CR) 10DLynch: [C:03+1] "It'll need to get rebased and updated for the current state of the config vis-a-vis the tests for T442767, but it seems reasonable otherwi" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362093 (https://phabricator.wikimedia.org/T443442) (owner: 10Ottomata) [21:39:31] (03PS1) 10Southparkfan: swift: Replace bullseye nodes in deployment-prep cluster [puppet] - 10https://gerrit.wikimedia.org/r/1364812 (https://phabricator.wikimedia.org/T436476) [21:40:39] (03CR) 10Dzahn: [C:03+2] ci: Configure docker to use the containerd snapshotter [puppet] - 10https://gerrit.wikimedia.org/r/1363509 (https://phabricator.wikimedia.org/T440668) (owner: 10Dduvall) [21:40:54] (03CR) 10Southparkfan: "As soon as https://gerrit.wikimedia.org/r/c/operations/puppet/+/1364812 has been merged, this cherry-pick can be removed" [puppet] - 10https://gerrit.wikimedia.org/r/1360569 (https://phabricator.wikimedia.org/T436476) (owner: 10Southparkfan) [21:42:11] (03PS1) 10JHathaway: fmt [puppet] - 10https://gerrit.wikimedia.org/r/1364820 [21:44:31] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json [21:45:28] (03CR) 10Southparkfan: "https://gitlab.wikimedia.org/repos/data_persistence/swift-ring/-/merge_requests/24 must also be merged, otherwise ring management breaks." [puppet] - 10https://gerrit.wikimedia.org/r/1360569 (https://phabricator.wikimedia.org/T436476) (owner: 10Southparkfan) [21:47:26] (03PS4) 10Krinkle: varnish: Don't strip browser Cache-Control:maxage from action=raw [puppet] - 10https://gerrit.wikimedia.org/r/1364773 (https://phabricator.wikimedia.org/T279120) [21:47:50] (03PS5) 10Krinkle: varnish: Don't strip browser Cache-Control:maxage from action=raw [puppet] - 10https://gerrit.wikimedia.org/r/1364773 (https://phabricator.wikimedia.org/T279120) [21:51:36] (03CR) 10JHathaway: [C:03+2] fmt [puppet] - 10https://gerrit.wikimedia.org/r/1364820 (owner: 10JHathaway) [21:51:39] !log ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4 (T440668) [21:51:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:51:42] T440668: ST6.1.4 Refactor Pipelinelib to use buildx and generate SBOMs - https://phabricator.wikimedia.org/T440668 [21:52:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 99.01% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [21:54:33] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97407 and previous config saved to /var/cache/conftool/dbconfig/20261105-215432-ladsgroup.json [21:54:52] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance [21:54:54] !log ci / contint1003/contint2003 - if "sudo docker info -f '{{.Driver}}'" says "overlay2" it's the classic way. after restart when it says "overlayfs2" that will be the NEW way! (T440668) [21:54:54] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97408 and previous config saved to /var/cache/conftool/dbconfig/20261105-215453-ladsgroup.json [21:54:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:57:46] (03PS1) 10Dduvall: ci: Restart containerd upon reconfiguration [puppet] - 10https://gerrit.wikimedia.org/r/1364843 (https://phabricator.wikimedia.org/T440668) [21:57:59] Hi all. When the readers window begins in a few minutes I have a few patches I'm planning to backport. [21:58:36] (03CR) 10Dduvall: "Note I already manually restarted containerd on all the new agent instances I created. This should only effect new instances from this poi" [puppet] - 10https://gerrit.wikimedia.org/r/1364843 (https://phabricator.wikimedia.org/T440668) (owner: 10Dduvall) [22:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261105T2200) [22:00:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by egardner@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358851 (https://phabricator.wikimedia.org/T439162) (owner: 10Eric Gardner) [22:00:24] EricGardner, the security team also has some deployments to do [22:00:53] (03Merged) 10jenkins-bot: Add Creative Commons license icons for the share card [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358851 (https://phabricator.wikimedia.org/T439162) (owner: 10Eric Gardner) [22:01:13] !log egardner@deploy2003 Started scap sync-world: Backport for [[gerrit:1358851|Add Creative Commons license icons for the share card (T439162)]] [22:01:16] T439162: ShareCard: Launch Checklist - https://phabricator.wikimedia.org/T439162 [22:01:20] Hi maryum – I'm doing one config patch, and then I have a set of 2 code backports. Then I'll be done. Does that work for you all? [22:01:38] EricGardner that's fine [22:02:20] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97409 and previous config saved to /var/cache/conftool/dbconfig/20261105-220220-ladsgroup.json [22:02:30] (03PS6) 10Krinkle: varnish: Don't strip browser Cache-Control:maxage from action=raw [puppet] - 10https://gerrit.wikimedia.org/r/1364773 (https://phabricator.wikimedia.org/T279120) [22:05:35] (03CR) 10Krinkle: varnish: Don't strip browser Cache-Control:maxage from action=raw (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1364773 (https://phabricator.wikimedia.org/T279120) (owner: 10Krinkle) [22:07:35] !log egardner@deploy2003 egardner: Backport for [[gerrit:1358851|Add Creative Commons license icons for the share card (T439162)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:07:38] T439162: ShareCard: Launch Checklist - https://phabricator.wikimedia.org/T439162 [22:08:02] !log egardner@deploy2003 egardner: Continuing with deployment [22:10:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.542% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [22:10:24] FIRING: [2x] SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:10:36] (03PS1) 10Dr0ptp4kt: Add yubikey 1 ssh pub key for dr0ptp4kt [puppet] - 10https://gerrit.wikimedia.org/r/1364857 [22:12:22] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97410 and previous config saved to /var/cache/conftool/dbconfig/20261105-221221-ladsgroup.json [22:13:03] (03CR) 10Dr0ptp4kt: "Thanks in advance." [puppet] - 10https://gerrit.wikimedia.org/r/1364857 (owner: 10Dr0ptp4kt) [22:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [22:15:32] !log egardner@deploy2003 Finished scap sync-world: Backport for [[gerrit:1358851|Add Creative Commons license icons for the share card (T439162)]] (duration: 14m 19s) [22:15:35] T439162: ShareCard: Launch Checklist - https://phabricator.wikimedia.org/T439162 [22:15:53] (03PS1) 10Lizimedia: add netbox-rw-access [puppet] - 10https://gerrit.wikimedia.org/r/1364863 (https://phabricator.wikimedia.org/T341581) [22:15:57] (03CR) 10TrainBranchBot: [C:03+2] "Approved by egardner@deploy2003 using scap backport" [extensions/WikimediaCustomizations] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1364473 (https://phabricator.wikimedia.org/T439163) (owner: 10Eric Gardner) [22:15:58] (03CR) 10TrainBranchBot: [C:03+2] "Approved by egardner@deploy2003 using scap backport" [extensions/WikimediaCustomizations] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363474 (https://phabricator.wikimedia.org/T442968) (owner: 10Eric Gardner) [22:17:24] (03Merged) 10jenkins-bot: ShareCard: Log client errors with the topic "error.sharecard" [extensions/WikimediaCustomizations] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1364473 (https://phabricator.wikimedia.org/T439163) (owner: 10Eric Gardner) [22:18:52] (03Merged) 10jenkins-bot: ShareCard: Show animated GIFs as a still image [extensions/WikimediaCustomizations] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363474 (https://phabricator.wikimedia.org/T442968) (owner: 10Eric Gardner) [22:19:11] !log egardner@deploy2003 Started scap sync-world: Backport for [[gerrit:1364473|ShareCard: Log client errors with the topic "error.sharecard" (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474|ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) [22:19:15] T439163: ShareCard: Product Analytics - https://phabricator.wikimedia.org/T439163 [22:19:15] T442968: [Share Card] Bug bashing - https://phabricator.wikimedia.org/T442968 [22:20:37] 06SRE, 10SRE-Access-Requests: Requesting access to Analytics Data Lake for sancheza/SanchezA-WMF - https://phabricator.wikimedia.org/T443850 (10SanchezA-WMF) 03NEW [22:20:47] 06SRE, 10LDAP-Access-Requests: Grant Access to ldap/nda for Aroraakhil - https://phabricator.wikimedia.org/T443439#12525039 (10KFrancis) Hi all, the extension agreement is complete. Thanks! [22:22:18] !log ssastry@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [22:22:24] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97411 and previous config saved to /var/cache/conftool/dbconfig/20261105-222223-ladsgroup.json [22:22:50] !log ssastry@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [22:23:04] 06SRE, 10SRE-Access-Requests, 06Wikimedia Enterprise: Request for additional rights for SanchezA-WMF to manage analytics - https://phabricator.wikimedia.org/T443636#12525042 (10SanchezA-WMF) I have completed the SRE Access Request form for this request. Thank you! [22:23:28] !log egardner@deploy2003 egardner: Backport for [[gerrit:1364473|ShareCard: Log client errors with the topic "error.sharecard" (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474|ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:24:07] !log egardner@deploy2003 egardner: Continuing with deployment [22:24:29] (03PS1) 10JHathaway: Add a script to set the in_service_date [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1364871 (https://phabricator.wikimedia.org/T441507) [22:26:28] (03CR) 10CI reject: [V:04-1] Add a script to set the in_service_date [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1364871 (https://phabricator.wikimedia.org/T441507) (owner: 10JHathaway) [22:27:34] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for Southparkfan - https://phabricator.wikimedia.org/T443141#12525050 (10Southparkfan) Hi @andrea.denisse! Could you add me to the [[ https://lists.wikimedia.org/postorius/lists/ops.lists.wikimedia.org/ | ops list ]]? Rights have been propagated... [22:28:19] (03CR) 10JHathaway: "Not sure if this should live in this repo, or the puppet repo, but it is vaguely similar to the netbox ganeti script, so putting it here." [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1364871 (https://phabricator.wikimedia.org/T441507) (owner: 10JHathaway) [22:29:53] (03PS1) 10Ahmon Dancy: buildkit: Bump v0.34.2 [puppet] - 10https://gerrit.wikimedia.org/r/1364875 (https://phabricator.wikimedia.org/T443845) [22:30:57] !log egardner@deploy2003 Finished scap sync-world: Backport for [[gerrit:1364473|ShareCard: Log client errors with the topic "error.sharecard" (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474|ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) (duration: 11m 46s) [22:31:01] T439163: ShareCard: Product Analytics - https://phabricator.wikimedia.org/T439163 [22:31:02] T442968: [Share Card] Bug bashing - https://phabricator.wikimedia.org/T442968 [22:32:26] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97412 and previous config saved to /var/cache/conftool/dbconfig/20261105-223225-ladsgroup.json [22:32:46] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance [22:32:48] maryum and anyone else still waiting to deploy, that's it for me. Feel free to take over [22:32:48] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2195 (T441633)', diff saved to https://phabricator.wikimedia.org/P97413 and previous config saved to /var/cache/conftool/dbconfig/20261105-223248-ladsgroup.json [22:34:06] (03CR) 10Jdlrobson: Enable ReadingLists custom lists on all Wikipedias. (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1364782 (https://phabricator.wikimedia.org/T441372) (owner: 10Jdrewniak) [22:35:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.333% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [22:37:24] EricGardner thanks!! [22:39:48] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: power-on failure for druid-internal1004.eqiad.wmnet - https://phabricator.wikimedia.org/T441866#12525134 (10VRiley-WMF) They requested the following information To assist with identifying the exact issue, kindly provide the following logs and screenshots: I... [22:39:57] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2195 (T441633)', diff saved to https://phabricator.wikimedia.org/P97414 and previous config saved to /var/cache/conftool/dbconfig/20261105-223956-ladsgroup.json [22:44:00] FIRING: PingLossPercent: Blackbox probe packet loss 3.958% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [22:44:17] 06SRE, 10SRE-tools, 10Ganeti, 06Infrastructure-Foundations: Cookbook to failover the Ganeti master - https://phabricator.wikimedia.org/T283320#12525135 (10Lizimedia) a:03Lizimedia [22:44:31] (03PS7) 10Krinkle: varnish: Don't strip browser Cache-Control:maxage from action=raw [puppet] - 10https://gerrit.wikimedia.org/r/1364773 (https://phabricator.wikimedia.org/T279120) [22:45:44] 10ops-eqiad, 06SRE, 06DC-Ops, 06ServiceOps: Rerack mc1057, mc1061, mc1069 - https://phabricator.wikimedia.org/T443209#12525138 (10VRiley-WMF) @jijiki The only problem I see is with A5. Would it be possible to have mc1057 in any other row other than A/B? [22:49:59] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97415 and previous config saved to /var/cache/conftool/dbconfig/20261105-224958-ladsgroup.json [22:50:15] (03PS2) 10Dzahn: Add yubikey 1 ssh pub key for dr0ptp4kt [puppet] - 10https://gerrit.wikimedia.org/r/1364857 (https://phabricator.wikimedia.org/T442758) (owner: 10Dr0ptp4kt) [22:51:25] !log Deployed security fixes for T439878, T440323, T440657, T440656, T440655 [22:51:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:55:17] (03CR) 10Dreamy Jazz: [C:03+1] tables-catalog: Add AbuseReview tables [puppet] - 10https://gerrit.wikimedia.org/r/1364499 (https://phabricator.wikimedia.org/T443206) (owner: 10Mpostoronca) [22:59:02] preparing to run another scap [23:00:01] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json [23:00:47] (03CR) 10Dzahn: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1364598 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [23:02:20] 06SRE, 10LDAP-Access-Requests: Grant Access to ciadmin for Peter Wangai - https://phabricator.wikimedia.org/T443855 (10pwangai) 03NEW [23:05:23] (03CR) 10Dzahn: [V:03+1 C:03+1] "compiler shows codesearch in cloud VPS does not get the proxy change while prod does" [puppet] - 10https://gerrit.wikimedia.org/r/1364598 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [23:06:54] (03CR) 10Dzahn: [V:03+1 C:03+2] codesearch: clone from external code hosts via the webproxy in production [puppet] - 10https://gerrit.wikimedia.org/r/1364598 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [23:09:30] 06SRE, 10LDAP-Access-Requests: Grant Access to ciadmin for Peter Wangai - https://phabricator.wikimedia.org/T443855#12525220 (10Dzahn) @thcipriani does "ciadmin" LDAP group have an approval step? [23:10:03] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2195 (T441633)', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json [23:10:23] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance [23:17:56] 06SRE, 10LDAP-Access-Requests: Grant Access to ciadmin for Peter Wangai - https://phabricator.wikimedia.org/T443855#12525252 (10thcipriani) hrm, it's not one that's set up for request in idm, but if it were I suppose I'd be approver there. But I know that test platform is working on improving test performance,... [23:18:40] !log Deployed security fix for T443424, T443467, T443036 [23:18:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:19:57] scap finished [23:30:06] !log codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout. [23:30:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:31:31] 07sre-alert-triage, 06Abstract Wikipedia team, 10Wikifunctions: Alert in need of triage: ProbeDown (instance wikifunctions-python-evaluator-staging:30443) - https://phabricator.wikimedia.org/T429678#12525285 (10RLazarus) a:03RLazarus The problem is the probes are configured to hit the staging evaluator in... [23:32:28] 06SRE, 10LDAP-Access-Requests: Grant Access to ciadmin for Peter Wangai - https://phabricator.wikimedia.org/T443855#12525288 (10Dzahn) In this context also see https://phabricator.wikimedia.org/T440668 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363509 for a change we made just now, today. [23:34:19] (03CR) 10Dzahn: [C:03+2] buildkit: Bump v0.34.2 [puppet] - 10https://gerrit.wikimedia.org/r/1364875 (https://phabricator.wikimedia.org/T443845) (owner: 10Ahmon Dancy) [23:34:32] 06SRE, 10LDAP-Access-Requests: Grant Access to ciadmin for Peter Wangai - https://phabricator.wikimedia.org/T443855#12525301 (10pwangai) @thcipriani and @Dzahn, that is noted. I will make sure the new agents adhere to that. I am in communication with @dduvall, and he will guide me through the process for the f... [23:34:51] (03PS8) 10Krinkle: varnish: Don't strip browser Cache-Control:maxage from action=raw [puppet] - 10https://gerrit.wikimedia.org/r/1364773 (https://phabricator.wikimedia.org/T279120) [23:35:43] (03PS1) 10RLazarus: staging: Remove codfw from sites for wikifunctions-python-evaluator-staging [puppet] - 10https://gerrit.wikimedia.org/r/1364993 (https://phabricator.wikimedia.org/T424193) [23:36:02] (03PS2) 10RLazarus: services: Remove codfw from sites for wikifunctions-python-evaluator-staging [puppet] - 10https://gerrit.wikimedia.org/r/1364993 (https://phabricator.wikimedia.org/T424193) [23:44:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from eqiad to ulsfo - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=ulsfo - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [23:46:22] 07Puppet, 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Cannot rollout Trixie hosts in deployment-prep without rm -rf /var/lib/puppet/ssl - https://phabricator.wikimedia.org/T443373#12525336 (10Southparkfan) >As far as I know, agents on Bookworm hosts do not... [23:50:10] (03CR) 10CDanis: [C:03+1] "It would be neat if the appropriate sites entries for a staging service were a YAML anchor, perhaps" [puppet] - 10https://gerrit.wikimedia.org/r/1364993 (https://phabricator.wikimedia.org/T424193) (owner: 10RLazarus) [23:58:36] (03PS3) 10RLazarus: services: Remove codfw from sites for wikifunctions-python-evaluator-staging [puppet] - 10https://gerrit.wikimedia.org/r/1364993 (https://phabricator.wikimedia.org/T424193) [23:58:43] (03CR) 10RLazarus: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1364993 (https://phabricator.wikimedia.org/T424193) (owner: 10RLazarus)