[00:06:40] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:12:05] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1307653 [01:12:05] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1307653 (owner: 10TrainBranchBot) [01:19:49] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1307653 (owner: 10TrainBranchBot) [02:00:18] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:07:27] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 08s) [02:09:44] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:14:44] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [04:06:40] FIRING: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:18:26] (03CR) 10Marostegui: [C:03+2] production-m5.sql.erb: Remove ipoid grants [puppet] - 10https://gerrit.wikimedia.org/r/1307440 (https://phabricator.wikimedia.org/T431007) (owner: 10Marostegui) [05:19:45] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb1015.eqiad.wmnet with reason: cloning [05:20:07] !log marostegui@cumin1003 conftool action : set/pooled=no; selector: name=clouddb1015.eqiad.wmnet,service=s4 [05:20:10] !log marostegui@cumin1003 conftool action : set/pooled=no; selector: name=clouddb1015.eqiad.wmnet,service=s6 [05:24:15] (03PS1) 10Marostegui: mariadb: Productionize clouddb1028 [puppet] - 10https://gerrit.wikimedia.org/r/1307675 (https://phabricator.wikimedia.org/T409557) [05:27:56] (03CR) 10Marostegui: [C:03+2] mariadb: Productionize clouddb1028 [puppet] - 10https://gerrit.wikimedia.org/r/1307675 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [05:43:13] (03PS1) 10Marostegui: es1039: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1307678 (https://phabricator.wikimedia.org/T430764) [05:52:59] (03CR) 10Marostegui: tables-catalog: set betafeatures_user_counts to public visibility (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1298329 (https://phabricator.wikimedia.org/T402145) (owner: 10SD0001) [05:57:18] 10ops-eqiad, 06Data-Persistence, 06DBA, 06DC-Ops, and 3 others: es7 primary (es1039.eqiad.wmnet) crashed - https://phabricator.wikimedia.org/T430764#12088652 (10Marostegui) @VRiley-WMF this host crashed a few days ago and has been working fine ever since. However I just tried a normal reboot before placing... [05:59:32] (03CR) 10Marostegui: [C:04-2] "Host not ready" [puppet] - 10https://gerrit.wikimedia.org/r/1307678 (https://phabricator.wikimedia.org/T430764) (owner: 10Marostegui) [06:11:25] FIRING: [2x] SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:13:08] !log installing Linux 6.12.95 on trixie hosts [06:13:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:59:45] FIRING: WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [07:00:05] Amir1, urbanecm, and awight: UTC morning backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260706T0700). Please do the needful. [07:00:05] VadymTS1: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [07:00:14] o/ [07:00:19] I need deployer [07:02:13] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.5 point update - https://phabricator.wikimedia.org/T427072#12088752 (10MoritzMuehlenhoff) [07:04:45] FIRING: [3x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [07:09:45] FIRING: [5x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [07:09:59] Can someone deploy my change? [07:13:50] (03CR) 10Hashar: [C:03+1] cache: turn off caching for zuul.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1306353 (https://phabricator.wikimedia.org/T430462) (owner: 10Dzahn) [07:20:59] VadymTS1: are you still around? I can do it [07:21:06] Yes I'm here [07:21:15] sorry I forgot the window, I have been busy doing other duties :) [07:21:17] jouncebot: now [07:21:17] For the next 0 hour(s) and 38 minute(s): UTC morning backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260706T0700) [07:24:02] (03CR) 10TrainBranchBot: [C:03+2] "Approved by hashar@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307617 (https://phabricator.wikimedia.org/T431202) (owner: 10VadymTS1) [07:24:31] VadymTS1: your change is being processed! that might take a while because if I remember well the first change of a week requires a full deployment of MediaWiki which is a lot of data :) [07:24:56] (03Merged) 10jenkins-bot: Turn on PageImages in Author namespace for Ukrainian Wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307617 (https://phabricator.wikimedia.org/T431202) (owner: 10VadymTS1) [07:25:06] do you have a link we can use to verify PageImages work properly on the Author namespace? [07:25:38] !log hashar@deploy1003 Started scap sync-world: Backport for [[gerrit:1307617|Turn on PageImages in Author namespace for Ukrainian Wikisource (T431202)]] [07:25:41] T431202: Turn on PageImages in Author namespace for Ukrainian Wikisource - https://phabricator.wikimedia.org/T431202 [07:26:19] Unfortunately, no. [07:26:38] :-] [07:26:57] well if after the change has been deployed you see something is not working as expected please report back [07:27:08] yes I'm know [07:27:34] preferably here. I have subscribed to the task (T431202) but might miss notifications [07:28:16] PROBLEM - HAProxy HTTPS wikipedia25.org ECDSA on cp4038 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [07:28:18] PROBLEM - HAProxy HTTPS wikiworkshop.org ECDSA on cp4038 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [07:28:18] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp4038 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [07:28:22] PROBLEM - haproxy process on cp4038 is CRITICAL: PROCS CRITICAL: 0 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy [07:29:24] !log mvernon@cumin2003 START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw [07:30:03] !log fabfur@cumin1003 conftool action : set/pooled=yes; selector: name=cp4038.* [07:30:06] !log fabfur@cumin1003 conftool action : set/pooled=no; selector: name=cp4038.* [07:30:25] !log depooled cp4038 to investigate on possible maxmind failure [07:30:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:34:28] VadymTS1: the MediaWiki images are still building [07:37:51] !log mvernon@cumin2003 END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw [07:38:41] !log installing python-urllib3 security updates [07:38:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:40:28] (03PS1) 10Brouberol: dse-k8s-eqiad/mediawiki-dumps-legacy: add 5% free space to the dumps volume [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307687 (https://phabricator.wikimedia.org/T431244) [07:41:25] FIRING: [2x] SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:43:46] !log hashar@deploy1003 vadymts1, hashar: Backport for [[gerrit:1307617|Turn on PageImages in Author namespace for Ukrainian Wikisource (T431202)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [07:43:49] T431202: Turn on PageImages in Author namespace for Ukrainian Wikisource - https://phabricator.wikimedia.org/T431202 [07:44:24] 07:38:01 [root] Image builds completed [07:44:33] I am trying to understand what is blocking/waiting [07:45:00] (03PS1) 10Elukey: CHANGELOG: add changelogs for release v13.0.0 [software/spicerack] - 10https://gerrit.wikimedia.org/r/1307688 [07:45:14] !log hashar@deploy1003 vadymts1, hashar: Continuing with deployment [07:45:21] oh it was pending confirmation [07:47:19] !log installing openjpeg2 security updates [07:47:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:49:16] 06SRE, 10SRE-Access-Requests, 06Machine-Learning-Team: Requesting access to ml-lab-users for mfossati - https://phabricator.wikimedia.org/T429148#12088859 (10isarantopoulos) [07:49:45] FIRING: [5x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [07:50:18] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations, 06Machine-Learning-Team: Move the Docker Registry's /ml prefix to S3/apus - https://phabricator.wikimedia.org/T420978#12088888 (10isarantopoulos) [07:51:43] 06SRE, 10Lift-Wing, 06Machine-Learning-Team: Fix securityContext propagation in liftwing - https://phabricator.wikimedia.org/T423149#12088924 (10isarantopoulos) [07:51:57] 07sre-alert-triage, 06Machine-Learning-Team, 07Essential-Work: Alert in need of triage: HelmfileAdminNGPendingChanges (instance deploy1003:9100) - https://phabricator.wikimedia.org/T414971#12088936 (10isarantopoulos) [07:52:51] (03CR) 10Elukey: [V:03+2 C:03+2] CHANGELOG: add changelogs for release v13.0.0 [software/spicerack] - 10https://gerrit.wikimedia.org/r/1307688 (owner: 10Elukey) [07:53:41] VadymTS1: your change is being deployed ( 07:53:16 K8s deployment progress: 32% (ok: 627; fail: 0; left: 1310) ) [07:53:44] (03PS1) 10Elukey: Upstream release v13.0.0 [software/spicerack] (debian) - 10https://gerrit.wikimedia.org/r/1307691 [07:53:59] !log installing pyjwt security updates [07:54:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:54:07] (03CR) 10Elukey: [V:03+2 C:03+2] Upstream release v13.0.0 [software/spicerack] (debian) - 10https://gerrit.wikimedia.org/r/1307691 (owner: 10Elukey) [07:54:45] RESOLVED: [5x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [07:57:16] RECOVERY - HAProxy HTTPS wikipedia25.org ECDSA on cp4038 is OK: SSL OK - Certificate wikipedia25.org contains all required SANs:Certificate wikipedia25.org (ECDSA) valid until 2026-08-05 06:34:20 +0000 (expires in 29 days) https://wikitech.wikimedia.org/wiki/HTTPS [07:57:18] RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp4038 is OK: SSL OK - Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2026-09-10 03:24:50 +0000 (expires in 65 days) https://wikitech.wikimedia.org/wiki/HTTPS [07:57:18] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp4038 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-09-04 20:06:43 +0000 (expires in 60 days) https://wikitech.wikimedia.org/wiki/HTTPS [07:57:22] RECOVERY - haproxy process on cp4038 is OK: PROCS OK: 2 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy [07:57:26] !log fabfur@cumin1003 conftool action : set/pooled=yes; selector: name=cp4038.* [07:57:39] !log repooled cp4038 [07:57:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:58:16] (03PS1) 10Muehlenhoff: build2004: Disable build on production images for now [puppet] - 10https://gerrit.wikimedia.org/r/1307693 [07:58:32] !log hashar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1307617|Turn on PageImages in Author namespace for Ukrainian Wikisource (T431202)]] (duration: 32m 53s) [07:58:35] T431202: Turn on PageImages in Author namespace for Ukrainian Wikisource - https://phabricator.wikimedia.org/T431202 [07:58:47] VadymTS1: your change has been deployed! [07:59:00] Ok I check [08:00:38] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host cumin2002.codfw.wmnet [08:02:04] let me know here whether it works as expected and if that is the case feel free to mark the task resolved ( https://phabricator.wikimedia.org/T431202 ) [08:03:07] Alright, I'll wait for my cache to update so I can view the change; if anything is amiss, I'll let you know. [08:06:33] !log remove cloudvirt1046, cloudvirt1062, cloudvirt1074, cloudvirt1075 from maintenance aggregate and put them in network-ovs - T424802 [08:06:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:06:36] T424802: cloudvirt1075 in 'maintenance' aggregate - https://phabricator.wikimedia.org/T424802 [08:07:39] VadymTS1: sure thing! :] [08:08:07] VadymTS1: or maybe you can erase the cache using ?action=purge [08:08:41] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cumin2002.codfw.wmnet [08:09:22] for your local browser cache: v [08:09:23] https://en.wikipedia.org/wiki/Help:Purge#Purge_local_browser_cache [08:09:40] (that Help:Purge page has documentation has to how one can purge the page on the server side ) [08:11:30] Hmm... I don't see any changes at the moment; perhaps I should wait a bit longer. [08:20:08] 06SRE, 06Data-Engineering, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 07Sustainability (Incident Followup): The webrequest_sampled_live data pipeline and its query tools have become mission-critical and require re-engineering for resilience - https://phabricator.wikimedia.org/T431112#12089050 (10Gehel)... [08:21:11] hashar For some reason, I don't see any changes at the moment. Perhaps you could check if there is an image here: https://uk.wikisource.org/w/index.php?title=Автор:Богдан-Ігор_Антонич&action=info [08:23:07] (03PS1) 10Bartosz Wójtowicz: ml-services: Declare ephemeral-storage for llm-qwen3-14b, move to ml-serve1013 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307707 (https://phabricator.wikimedia.org/T431017) [08:24:31] (03PS1) 10Kevin Bazira: ml-services: deploy cope-b-a4b isvc image that emits vLLM metrics [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307708 (https://phabricator.wikimedia.org/T431136) [08:24:32] (03CR) 10JMeybohm: [C:03+2] Clean up istio configs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306327 (https://phabricator.wikimedia.org/T427401) (owner: 10JMeybohm) [08:24:34] (03PS2) 10Bartosz Wójtowicz: ml-services: Declare ephemeral-storage for llm-qwen3-14b, move to ml-serve1013 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307707 (https://phabricator.wikimedia.org/T431017) [08:24:50] (03PS1) 10Majavah: dynamicproxy: Do not allow renaming proxies [puppet] - 10https://gerrit.wikimedia.org/r/1307709 (https://phabricator.wikimedia.org/T431092) [08:25:15] VadymTS1: :-\ I don't know anything about the PageImages extension unfortunately. May you report it back on the task please? [08:25:24] I will try to find someone knowing about it [08:25:50] of course PageImages has no maintainer bah [08:25:55] (03CR) 10Atsuko: [C:03+1] dse-k8s-eqiad/mediawiki-dumps-legacy: add 5% free space to the dumps volume [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307687 (https://phabricator.wikimedia.org/T431244) (owner: 10Brouberol) [08:26:18] (03CR) 10Elukey: [C:03+1] build2004: Disable build on production images for now [puppet] - 10https://gerrit.wikimedia.org/r/1307693 (owner: 10Muehlenhoff) [08:26:35] (03Merged) 10jenkins-bot: Clean up istio configs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306327 (https://phabricator.wikimedia.org/T427401) (owner: 10JMeybohm) [08:26:41] (03CR) 10Majavah: [V:03+2 C:03+2] "merging per task and IRC discussion" [puppet] - 10https://gerrit.wikimedia.org/r/1307709 (https://phabricator.wikimedia.org/T431092) (owner: 10Majavah) [08:27:39] (03CR) 10Brouberol: [C:03+2] dse-k8s-eqiad/mediawiki-dumps-legacy: add 5% free space to the dumps volume [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307687 (https://phabricator.wikimedia.org/T431244) (owner: 10Brouberol) [08:28:18] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply [08:28:23] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply [08:28:46] (03CR) 10Muehlenhoff: [C:03+2] build2004: Disable build on production images for now [puppet] - 10https://gerrit.wikimedia.org/r/1307693 (owner: 10Muehlenhoff) [08:32:22] (03PS1) 10Majavah: dynamicproxy: Improve HTTP body parameter validation [puppet] - 10https://gerrit.wikimedia.org/r/1307711 [08:32:22] (03PS1) 10Majavah: dynamicproxy: Do not allow editing a proxy with the create endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1307712 (https://phabricator.wikimedia.org/T429960) [08:32:25] (03PS1) 10Majavah: dynamicproxy: Validate each proxy is linked to exactly one domain [puppet] - 10https://gerrit.wikimedia.org/r/1307713 (https://phabricator.wikimedia.org/T429960) [08:32:28] (03PS1) 10Majavah: dynamicproxy: Update schema to only support a single backend [puppet] - 10https://gerrit.wikimedia.org/r/1307714 (https://phabricator.wikimedia.org/T429960) [08:33:21] Hashar ok it's work [08:34:18] (03CR) 10CI reject: [V:04-1] dynamicproxy: Update schema to only support a single backend [puppet] - 10https://gerrit.wikimedia.org/r/1307714 (https://phabricator.wikimedia.org/T429960) (owner: 10Majavah) [08:34:27] (03CR) 10Kevin Bazira: [C:03+1] ml-services: Declare ephemeral-storage for llm-qwen3-14b, move to ml-serve1013 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307707 (https://phabricator.wikimedia.org/T431017) (owner: 10Bartosz Wójtowicz) [08:36:24] VadymTS1: I did purge the page and the wikidata entry, maybe that has caused a refresh [08:36:25] RESOLVED: SystemdUnitFailed: debian-weekly-rebuild.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:36:49] I have also started: `mwscript extensions/PageImages/maintenance/initImageData.php --wiki=ukwikisource --namespaces 102` [08:37:04] I think the script fix this [08:37:53] Sure. Thanks you hashar for deploying [08:43:29] (03PS1) 10Kosta Harlan: extension-list: Add WikimediaAntiAbuse [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307716 (https://phabricator.wikimedia.org/T431023) [08:43:32] (03PS1) 10Kosta Harlan: WikimediaAntiAbuse: Register wmgUse config and load extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307717 (https://phabricator.wikimedia.org/T431023) [08:43:40] (03CR) 10Ayounsi: [C:03+1] "thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1307092 (https://phabricator.wikimedia.org/T281095) (owner: 10Hnowlan) [08:44:12] (03CR) 10Kosta Harlan: [C:04-2] extension-list: Add WikimediaAntiAbuse [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307716 (https://phabricator.wikimedia.org/T431023) (owner: 10Kosta Harlan) [08:47:40] (03PS2) 10Kosta Harlan: WikimediaAntiAbuse: Register wmgUse config and load extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307717 (https://phabricator.wikimedia.org/T431023) [08:47:52] PROBLEM - ganeti-noded running on ganeti1055 is CRITICAL: PROCS CRITICAL: 3 processes with UID = 0 (root), command name ganeti-noded https://wikitech.wikimedia.org/wiki/Ganeti [08:48:52] RECOVERY - ganeti-noded running on ganeti1055 is OK: PROCS OK: 2 processes with UID = 0 (root), command name ganeti-noded https://wikitech.wikimedia.org/wiki/Ganeti [08:49:14] (03PS3) 10Arnaudb: trafficserver: raise ATS timeouts for the gerrit secondary backends [puppet] - 10https://gerrit.wikimedia.org/r/1304808 (https://phabricator.wikimedia.org/T429749) [08:49:23] (03PS2) 10Arnaudb: trafficserver: raise ATS timeouts for the gerrit primary backend [puppet] - 10https://gerrit.wikimedia.org/r/1304821 (https://phabricator.wikimedia.org/T429749) [08:51:07] 06SRE, 06Data-Engineering, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 07Sustainability (Incident Followup): The webrequest_sampled_live data pipeline and its query tools have become mission-critical and require re-engineering for resilience - https://phabricator.wikimedia.org/T431112#12089173 (10elukey... [08:52:34] 06SRE, 06Infrastructure-Foundations, 10netops: Blackbox probe for TLS cert expriy failing on multiple eqiad SR-Linux nodes - https://phabricator.wikimedia.org/T429242#12089176 (10ayounsi) I had a quick look at the doc and indeed I couldn't find a way to increase the check interval. [08:53:42] (03PS3) 10Arnaudb: trafficserver: raise ATS timeouts for the gerrit primary backend [puppet] - 10https://gerrit.wikimedia.org/r/1304821 (https://phabricator.wikimedia.org/T429749) [08:55:05] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: Declare ephemeral-storage for llm-qwen3-14b, move to ml-serve1013 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307707 (https://phabricator.wikimedia.org/T431017) (owner: 10Bartosz Wójtowicz) [08:56:25] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-pretrain: apply [08:56:42] !log blake@deploy1003 helmfile [codfw] START helmfile.d/services/mw-pretrain: apply [08:57:05] (03PS1) 10Elukey: debian: modify rules to allow unit tests and cov in auto-test [software/spicerack] (debian) - 10https://gerrit.wikimedia.org/r/1307720 (https://phabricator.wikimedia.org/T429699) [08:57:28] (03Merged) 10jenkins-bot: ml-services: Declare ephemeral-storage for llm-qwen3-14b, move to ml-serve1013 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307707 (https://phabricator.wikimedia.org/T431017) (owner: 10Bartosz Wójtowicz) [08:58:24] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [09:00:23] (03CR) 10Jforrester: "I believe that bd808 proved earlier this year that, contrary to the folk knowledge I've been guilty of spreading, this is actually safe to" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307716 (https://phabricator.wikimedia.org/T431023) (owner: 10Kosta Harlan) [09:00:28] (03PS4) 10Arnaudb: trafficserver: raise ATS timeouts for the gerrit secondary backends [puppet] - 10https://gerrit.wikimedia.org/r/1304808 (https://phabricator.wikimedia.org/T429749) [09:00:32] (03PS4) 10Arnaudb: trafficserver: raise ATS timeouts for the gerrit primary backend [puppet] - 10https://gerrit.wikimedia.org/r/1304821 (https://phabricator.wikimedia.org/T429749) [09:00:56] (03CR) 10Arnaudb: trafficserver: raise ATS timeouts for the gerrit secondary backends (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1304808 (https://phabricator.wikimedia.org/T429749) (owner: 10Arnaudb) [09:04:55] (03CR) 10Ayounsi: [C:03+1] dse-k8s: add new POD IPv4 block 10.68.0.0/17 to prefix lists [homer/public] - 10https://gerrit.wikimedia.org/r/1307396 (https://phabricator.wikimedia.org/T430658) (owner: 10Cathal Mooney) [09:07:01] !log blake@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-pretrain: apply [09:10:50] !log marostegui@cumin1003 conftool action : set/pooled=yes; selector: name=clouddb1015.eqiad.wmnet,service=s6 [09:10:54] !log marostegui@cumin1003 conftool action : set/pooled=yes; selector: name=clouddb1015.eqiad.wmnet,service=s4 [09:11:32] (03CR) 10Kosta Harlan: [C:04-2] "I don't mind either way. If we think this is safe, we could sync it today." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307716 (https://phabricator.wikimedia.org/T431023) (owner: 10Kosta Harlan) [09:11:43] (03CR) 10Kosta Harlan: [C:04-2] "Needs to wait for the extension to be on all wikis" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307717 (https://phabricator.wikimedia.org/T431023) (owner: 10Kosta Harlan) [09:12:02] (03PS3) 10Kosta Harlan: WikimediaAntiAbuse: Register wmgUse config and load extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307717 (https://phabricator.wikimedia.org/T431023) [09:12:59] (03PS1) 10Marostegui: installserver: Do not format clouddb1028 [puppet] - 10https://gerrit.wikimedia.org/r/1307722 (https://phabricator.wikimedia.org/T409557) [09:15:33] (03CR) 10Marostegui: [C:03+2] installserver: Do not format clouddb1028 [puppet] - 10https://gerrit.wikimedia.org/r/1307722 (https://phabricator.wikimedia.org/T409557) (owner: 10Marostegui) [09:16:43] (03CR) 10Santiago Faci: [C:03+1] T429269: Send logged-in experiment events to ins-502b [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1303490 (owner: 10Pushpaktiwari) [09:17:08] (03CR) 10Klausman: [C:03+1] "https://gitlab.wikimedia.org/repos/sre/preseed-test is a great tool for testing partman recipes, in case you haven't already." [puppet] - 10https://gerrit.wikimedia.org/r/1307368 (https://phabricator.wikimedia.org/T431017) (owner: 10Dpogorzelski) [09:18:16] (03PS1) 10Bartosz Wójtowicz: admin_ng: raise llm namespace compute quota on ml-serve [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307723 (https://phabricator.wikimedia.org/T431017) [09:18:47] (03PS1) 10Btullis: datahub: set SearchClientShim engine type explicitly [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307724 (https://phabricator.wikimedia.org/T402408) [09:18:49] (03PS1) 10Majavah: toolforge: bastion_proc_management: Do not try to set Icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/1307725 [09:19:24] (03PS1) 10Btullis: datahub-next: drop dead noCodeDataMigration values [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307726 (https://phabricator.wikimedia.org/T402408) [09:19:24] (03CR) 10CI reject: [V:04-1] toolforge: bastion_proc_management: Do not try to set Icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/1307725 (owner: 10Majavah) [09:19:28] (03PS1) 10Btullis: datahub-next: pin OpenSearch shim engine type [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307727 (https://phabricator.wikimedia.org/T402408) [09:19:54] (03PS2) 10Majavah: toolforge: bastion_proc_management: Do not try to set Icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/1307725 [09:20:19] !log upgrade all bullseye hosts to pywmflib 3.1 - T430552 [09:20:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:20:22] T430552: Deploy wmflib 3.0.0 to production - https://phabricator.wikimedia.org/T430552 [09:22:57] (03CR) 10David Caro: [C:03+1] "LGTM, are we alerting on failing systemd units on prometheus side?" [puppet] - 10https://gerrit.wikimedia.org/r/1307725 (owner: 10Majavah) [09:23:13] (03CR) 10Filippo Giunchedi: [C:03+1] toolforge: bastion_proc_management: Do not try to set Icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/1307725 (owner: 10Majavah) [09:23:15] (03CR) 10Majavah: [C:04-1] "AIUI this requires a `ensure => absent` first for the nrpe::monitor_service definition to remove the nrpe2nodexp timers" [puppet] - 10https://gerrit.wikimedia.org/r/1302764 (https://phabricator.wikimedia.org/T328502) (owner: 10Filippo Giunchedi) [09:23:24] (03CR) 10Majavah: [C:03+2] toolforge: bastion_proc_management: Do not try to set Icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/1307725 (owner: 10Majavah) [09:25:10] (03CR) 10CWilliams: "Thanks! It looks like the key details are made visible by this." [puppet] - 10https://gerrit.wikimedia.org/r/1307087 (https://phabricator.wikimedia.org/T430149) (owner: 10Hnowlan) [09:25:32] (03CR) 10Bartosz Wójtowicz: "@tklausmann@wikimedia.org @dpogorzelski@wikimedia.org Could you approve and later deploy whenever you find some free time?" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307723 (https://phabricator.wikimedia.org/T431017) (owner: 10Bartosz Wójtowicz) [09:29:57] (03PS1) 10Marostegui: mariadb: Move s4 hosts to x4 [puppet] - 10https://gerrit.wikimedia.org/r/1307730 (https://phabricator.wikimedia.org/T404715) [09:30:12] (03PS1) 10Majavah: hieradata: Remove unused hiera file [puppet] - 10https://gerrit.wikimedia.org/r/1307731 [09:30:29] (03CR) 10CI reject: [V:04-1] mariadb: Move s4 hosts to x4 [puppet] - 10https://gerrit.wikimedia.org/r/1307730 (https://phabricator.wikimedia.org/T404715) (owner: 10Marostegui) [09:30:45] (03CR) 10Majavah: [C:03+2] "not that i am aware of." [puppet] - 10https://gerrit.wikimedia.org/r/1307725 (owner: 10Majavah) [09:31:52] (03PS2) 10Marostegui: mariadb: Move s4 hosts to x4 [puppet] - 10https://gerrit.wikimedia.org/r/1307730 (https://phabricator.wikimedia.org/T404715) [09:32:06] (03PS1) 10Bartosz Wójtowicz: ml-services: declare ephemeral-storage for qwen3-14b, move to ml-serve1013 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307732 (https://phabricator.wikimedia.org/T431017) [09:32:58] (03CR) 10Marostegui: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1307730 (https://phabricator.wikimedia.org/T404715) (owner: 10Marostegui) [09:33:10] (03CR) 10Majavah: [C:03+2] Revert "P:dumps: rsync: Do not use LOAD_BALANCER_HEALTH_CHECKS" [puppet] - 10https://gerrit.wikimedia.org/r/1307435 (owner: 10Majavah) [09:33:46] (03CR) 10Majavah: [C:03+2] hieradata: Remove unused openstack endpoint keys [puppet] - 10https://gerrit.wikimedia.org/r/1307363 (owner: 10Majavah) [09:34:01] (03CR) 10Klausman: [C:03+1] admin_ng: Fix KServe view RBAC on install_kserve_resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306892 (owner: 10Bartosz Wójtowicz) [09:34:27] (03CR) 10Klausman: [C:03+1] admin_ng: raise llm namespace compute quota on ml-serve [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307723 (https://phabricator.wikimedia.org/T431017) (owner: 10Bartosz Wójtowicz) [09:36:03] (03CR) 10AikoChou: [C:03+1] ml-services: declare ephemeral-storage for qwen3-14b, move to ml-serve1013 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307732 (https://phabricator.wikimedia.org/T431017) (owner: 10Bartosz Wójtowicz) [09:36:50] (03CR) 10Marostegui: "PCC: https://puppet-compiler.wmflabs.org/output/1307730/7137/" [puppet] - 10https://gerrit.wikimedia.org/r/1307730 (https://phabricator.wikimedia.org/T404715) (owner: 10Marostegui) [09:37:05] (03PS1) 10Brouberol: dse-k8s-eqiad/mediawiki-dumps-legacy: avoid chowning the dumps data when starting toolbox pods [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307733 (https://phabricator.wikimedia.org/T431258) [09:38:11] (03PS1) 10Ozge: ml-services: Deploy new editing-suggestions image to ml-staging-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307735 (https://phabricator.wikimedia.org/T429675) [09:38:47] (03PS2) 10Brouberol: dse-k8s-eqiad/mediawiki-dumps-legacy: avoid chowning the dumps data when starting toolbox pods [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307733 (https://phabricator.wikimedia.org/T431258) [09:39:07] (03CR) 10Bartosz Wójtowicz: [C:03+2] admin_ng: raise llm namespace compute quota on ml-serve [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307723 (https://phabricator.wikimedia.org/T431017) (owner: 10Bartosz Wójtowicz) [09:39:22] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: declare ephemeral-storage for qwen3-14b, move to ml-serve1013 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307732 (https://phabricator.wikimedia.org/T431017) (owner: 10Bartosz Wójtowicz) [09:40:36] (03CR) 10Bartosz Wójtowicz: [C:03+1] ml-services: Deploy new editing-suggestions image to ml-staging-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307735 (https://phabricator.wikimedia.org/T429675) (owner: 10Ozge) [09:41:13] (03CR) 10Ozge: [C:03+2] ml-services: Deploy new editing-suggestions image to ml-staging-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307735 (https://phabricator.wikimedia.org/T429675) (owner: 10Ozge) [09:42:16] (03CR) 10Ozge: [V:03+2 C:03+2] ml-services: Deploy new editing-suggestions image to ml-staging-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307735 (https://phabricator.wikimedia.org/T429675) (owner: 10Ozge) [09:44:15] (03CR) 10Fabfur: [C:03+1] profile::benthos: add x-provenance handling for webrequest [puppet] - 10https://gerrit.wikimedia.org/r/1307169 (https://phabricator.wikimedia.org/T427068) (owner: 10Elukey) [09:47:57] (03Merged) 10jenkins-bot: admin_ng: raise llm namespace compute quota on ml-serve [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307723 (https://phabricator.wikimedia.org/T431017) (owner: 10Bartosz Wójtowicz) [09:48:18] (03Merged) 10jenkins-bot: ml-services: declare ephemeral-storage for qwen3-14b, move to ml-serve1013 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307732 (https://phabricator.wikimedia.org/T431017) (owner: 10Bartosz Wójtowicz) [09:48:46] (03CR) 10Hnowlan: [V:03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/8924/console" [puppet] - 10https://gerrit.wikimedia.org/r/1307092 (https://phabricator.wikimedia.org/T281095) (owner: 10Hnowlan) [09:48:49] (03CR) 10Bartosz Wójtowicz: [C:03+2] "Thank you @tklausmann@wikimedia.org! Please let me know once it's deployed" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307723 (https://phabricator.wikimedia.org/T431017) (owner: 10Bartosz Wójtowicz) [09:48:59] (03Merged) 10jenkins-bot: ml-services: Deploy new editing-suggestions image to ml-staging-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307735 (https://phabricator.wikimedia.org/T429675) (owner: 10Ozge) [09:49:07] (03CR) 10David Caro: [C:03+1] "🎉" [puppet] - 10https://gerrit.wikimedia.org/r/1307731 (owner: 10Majavah) [09:49:19] (03CR) 10Majavah: [C:03+2] hieradata: Remove unused hiera file [puppet] - 10https://gerrit.wikimedia.org/r/1307731 (owner: 10Majavah) [09:51:10] (03CR) 10David Caro: [C:03+1] "Answering to myself, no we are not as far as I can tell. There's some failed services around that probably should either not fail, or not " [puppet] - 10https://gerrit.wikimedia.org/r/1307725 (owner: 10Majavah) [09:51:54] (03CR) 10David Caro: [C:03+1] "> not that i am aware of." [puppet] - 10https://gerrit.wikimedia.org/r/1307725 (owner: 10Majavah) [09:52:03] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [09:53:55] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [software/spicerack] (debian) - 10https://gerrit.wikimedia.org/r/1307720 (https://phabricator.wikimedia.org/T429699) (owner: 10Elukey) [09:54:10] (03CR) 10Elukey: [C:03+2] debian: modify rules to allow unit tests and cov in auto-test [software/spicerack] (debian) - 10https://gerrit.wikimedia.org/r/1307720 (https://phabricator.wikimedia.org/T429699) (owner: 10Elukey) [09:54:20] !log ozge@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [09:56:11] (03PS1) 10Atsuko: icinga: ensure that elasticsearch alert output is stable [puppet] - 10https://gerrit.wikimedia.org/r/1307738 (https://phabricator.wikimedia.org/T431254) [09:58:54] (03PS1) 10Majavah: P:puppetserver::wmcs: Do not run facts cleanup on hosts with PuppetDB [puppet] - 10https://gerrit.wikimedia.org/r/1307739 (https://phabricator.wikimedia.org/T431261) [09:58:57] (03CR) 10Ladsgroup: [C:03+1] mariadb: Move s4 hosts to x4 [puppet] - 10https://gerrit.wikimedia.org/r/1307730 (https://phabricator.wikimedia.org/T404715) (owner: 10Marostegui) [09:59:42] (03PS1) 10Ozge: ml-services: Deploy new editing-suggestions image to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307740 (https://phabricator.wikimedia.org/T429675) [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260706T1000) [10:01:01] (03CR) 10Btullis: [C:03+1] dse-k8s-eqiad/mediawiki-dumps-legacy: avoid chowning the dumps data when starting toolbox pods [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307733 (https://phabricator.wikimedia.org/T431258) (owner: 10Brouberol) [10:01:09] (03CR) 10Brouberol: [C:03+2] dse-k8s-eqiad/mediawiki-dumps-legacy: avoid chowning the dumps data when starting toolbox pods [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307733 (https://phabricator.wikimedia.org/T431258) (owner: 10Brouberol) [10:01:47] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1 DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1307739 (https://phabricator.wikimedia.org/T431261) (owner: 10Majavah) [10:03:12] (03Merged) 10jenkins-bot: dse-k8s-eqiad/mediawiki-dumps-legacy: avoid chowning the dumps data when starting toolbox pods [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307733 (https://phabricator.wikimedia.org/T431258) (owner: 10Brouberol) [10:04:15] (03CR) 10Bartosz Wójtowicz: [C:03+1] "Thank you!!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307740 (https://phabricator.wikimedia.org/T429675) (owner: 10Ozge) [10:05:01] (03CR) 10Gehel: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1307738 (https://phabricator.wikimedia.org/T431254) (owner: 10Atsuko) [10:05:59] (03CR) 10Marostegui: [C:03+2] mariadb: Move s4 hosts to x4 [puppet] - 10https://gerrit.wikimedia.org/r/1307730 (https://phabricator.wikimedia.org/T404715) (owner: 10Marostegui) [10:14:39] (03CR) 10Ozge: [V:03+2 C:03+2] ml-services: Deploy new editing-suggestions image to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307740 (https://phabricator.wikimedia.org/T429675) (owner: 10Ozge) [10:16:06] (03CR) 10Btullis: [C:03+2] datahub: set SearchClientShim engine type explicitly [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307724 (https://phabricator.wikimedia.org/T402408) (owner: 10Btullis) [10:16:51] (03Merged) 10jenkins-bot: ml-services: Deploy new editing-suggestions image to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307740 (https://phabricator.wikimedia.org/T429675) (owner: 10Ozge) [10:17:19] !log uploaded spicerack_13.0.0 to apt.wikimedia.org bookworm-wikimedia,trixie-wikimedia [10:17:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:18:43] (03Merged) 10jenkins-bot: datahub: set SearchClientShim engine type explicitly [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307724 (https://phabricator.wikimedia.org/T402408) (owner: 10Btullis) [10:19:29] (03CR) 10Atsuko: [C:03+2] icinga: ensure that elasticsearch alert output is stable [puppet] - 10https://gerrit.wikimedia.org/r/1307738 (https://phabricator.wikimedia.org/T431254) (owner: 10Atsuko) [10:19:50] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [10:20:11] (03CR) 10Hnowlan: [V:03+1 C:03+2] icinga: clean up disabled librenms components [puppet] - 10https://gerrit.wikimedia.org/r/1307092 (https://phabricator.wikimedia.org/T281095) (owner: 10Hnowlan) [10:20:24] !log ozge@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [10:21:11] atsukoito: I think our puppet changes are overlapping, mine is safe to merge [10:21:22] hnowlan: mine too [10:21:24] !log elukey@cumin2002 START - Cookbook sre.hosts.reimage for host sretest1005.eqiad.wmnet with OS trixie [10:21:28] merging [10:21:40] go for it [10:22:37] done [10:23:31] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply [10:23:38] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply [10:23:41] (03PS1) 10Majavah: P:monitoring: Remove absented resources [puppet] - 10https://gerrit.wikimedia.org/r/1307745 [10:24:17] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply [10:24:22] (03PS1) 10Elukey: Remove ipmi-password-reset.py [cookbooks] - 10https://gerrit.wikimedia.org/r/1307746 (https://phabricator.wikimedia.org/T429699) [10:24:23] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply [10:24:37] !log spicerack 13.0.0 deployed on cumin2002 [10:24:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:24:46] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply [10:24:49] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply [10:27:59] (03PS2) 10Majavah: P:monitoring: Remove absented resources [puppet] - 10https://gerrit.wikimedia.org/r/1307745 [10:28:24] (03CR) 10Hashar: [C:03+1] "An alternative to disable CDN caching would be to have the Apache middleware to emit Cache-Control headers to supplement what Zuul is not " [puppet] - 10https://gerrit.wikimedia.org/r/1306353 (https://phabricator.wikimedia.org/T430462) (owner: 10Dzahn) [10:31:08] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply [10:31:19] !log Setup x4 codfw topology T404715 [10:31:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:31:22] T404715: Setup x4 section - https://phabricator.wikimedia.org/T404715 [10:31:44] (03PS1) 10Majavah: P:monitoring: Remove unused check_fresh_files_in_dir check [puppet] - 10https://gerrit.wikimedia.org/r/1307747 [10:31:55] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply [10:32:33] (03PS1) 10Majavah: openstack: apply_security_groups: Do not use Icinga monitoring [puppet] - 10https://gerrit.wikimedia.org/r/1307748 [10:33:19] (03PS1) 10Bartosz Wójtowicz: ml-services: run experimental qwen3-14b on a single GPU on ml-serve1012 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307749 (https://phabricator.wikimedia.org/T431268) [10:38:12] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1098 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:12] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1074 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:12] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1068 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:12] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1093 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:12] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1124 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:13] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1091 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:13] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1089 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:14] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1100 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:14] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1094 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:15] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1076 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:15] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1077 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:16] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1070 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:16] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1096 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:17] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1080 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:17] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1114 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:18] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1082 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:18] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1103 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:19] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1113 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:19] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1112 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:20] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1071 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:20] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1107 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:21] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1120 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:21] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1119 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:22] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1109 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:22] RECOVERY - OpenSearch unassigned shard check - 9400 on cirrussearch1125 is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [10:38:59] !log elukey@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1005.eqiad.wmnet with reason: host reimage [10:39:55] (03PS4) 10Kosta Harlan: WikimediaAntiAbuse: Register wmgUse config and load extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307717 (https://phabricator.wikimedia.org/T431023) [10:39:55] (03PS1) 10Kosta Harlan: Enable WikimediaAntiAbuse on Beta Cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307750 (https://phabricator.wikimedia.org/T431023) [10:39:57] (03PS1) 10Kosta Harlan: WikimediaAntiAbuse: Enable everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307751 (https://phabricator.wikimedia.org/T431023) [10:40:21] (03CR) 10Kosta Harlan: extension-list: Add WikimediaAntiAbuse [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307716 (https://phabricator.wikimedia.org/T431023) (owner: 10Kosta Harlan) [10:40:42] (03CR) 10Kosta Harlan: "@jforrester@wikimedia.org can you point me to the example of where this was proved?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307716 (https://phabricator.wikimedia.org/T431023) (owner: 10Kosta Harlan) [10:40:48] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.5 point update - https://phabricator.wikimedia.org/T427072#12089695 (10MoritzMuehlenhoff) [10:40:57] (03CR) 10Kosta Harlan: WikimediaAntiAbuse: Register wmgUse config and load extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307717 (https://phabricator.wikimedia.org/T431023) (owner: 10Kosta Harlan) [10:41:02] (03CR) 10AikoChou: [C:03+1] ml-services: run experimental qwen3-14b on a single GPU on ml-serve1012 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307749 (https://phabricator.wikimedia.org/T431268) (owner: 10Bartosz Wójtowicz) [10:41:12] (03CR) 10Kosta Harlan: [C:04-2] "Needs to wait for the extension to be on the train" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307751 (https://phabricator.wikimedia.org/T431023) (owner: 10Kosta Harlan) [10:41:22] (03PS1) 10Fabfur: ip_reputation_vendors: add basic safeguard to spur downloader [puppet] - 10https://gerrit.wikimedia.org/r/1307752 [10:42:20] (03PS1) 10Majavah: haproxy: Properly absent Icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/1307753 [10:42:21] (03PS1) 10Majavah: P:wmcs::cloudlb: Disable HAProxy Icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/1307754 [10:42:35] (03CR) 10Hnowlan: restbase: move nrpe check to prom blackbox check (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1306661 (https://phabricator.wikimedia.org/T407141) (owner: 10Hnowlan) [10:43:18] (03CR) 10Fabfur: [C:03+2] cache::haproxy: add correlation id feature [puppet] - 10https://gerrit.wikimedia.org/r/1305635 (https://phabricator.wikimedia.org/T426379) (owner: 10Fabfur) [10:43:27] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: run experimental qwen3-14b on a single GPU on ml-serve1012 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307749 (https://phabricator.wikimedia.org/T431268) (owner: 10Bartosz Wójtowicz) [10:44:17] !log elukey@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1005.eqiad.wmnet with reason: host reimage [10:44:23] (03CR) 10Hnowlan: [C:03+1] "lgtm, one nit" [puppet] - 10https://gerrit.wikimedia.org/r/1305769 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [10:45:39] (03Merged) 10jenkins-bot: ml-services: run experimental qwen3-14b on a single GPU on ml-serve1012 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307749 (https://phabricator.wikimedia.org/T431268) (owner: 10Bartosz Wójtowicz) [10:45:42] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 7): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/8926/co" [puppet] - 10https://gerrit.wikimedia.org/r/1307753 (owner: 10Majavah) [10:46:20] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [10:47:28] (03CR) 10Muehlenhoff: [C:03+2] Switch the URL downloaders back to the Trixie instances [dns] - 10https://gerrit.wikimedia.org/r/1307431 (https://phabricator.wikimedia.org/T427282) (owner: 10Muehlenhoff) [10:47:45] !log jmm@dns1004 START - running authdns-update [10:47:50] !log jmm@dns1004 START - running authdns-update [10:50:00] !log jmm@dns1004 END - running authdns-update [10:53:55] (03CR) 10Elukey: [C:03+2] Remove ipmi-password-reset.py [cookbooks] - 10https://gerrit.wikimedia.org/r/1307746 (https://phabricator.wikimedia.org/T429699) (owner: 10Elukey) [10:54:14] (03CR) 10Btullis: [C:03+2] datahub-next: drop dead noCodeDataMigration values [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307726 (https://phabricator.wikimedia.org/T402408) (owner: 10Btullis) [10:54:26] (03CR) 10Btullis: [C:03+2] datahub-next: pin OpenSearch shim engine type [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307727 (https://phabricator.wikimedia.org/T402408) (owner: 10Btullis) [10:56:36] (03Merged) 10jenkins-bot: datahub-next: drop dead noCodeDataMigration values [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307726 (https://phabricator.wikimedia.org/T402408) (owner: 10Btullis) [10:56:41] (03Merged) 10jenkins-bot: datahub-next: pin OpenSearch shim engine type [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307727 (https://phabricator.wikimedia.org/T402408) (owner: 10Btullis) [10:57:52] (03CR) 10Bartosz Wójtowicz: [C:03+2] "Thank you for reviews!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306892 (owner: 10Bartosz Wójtowicz) [10:59:05] (03PS1) 10Bartosz Wójtowicz: ml-services: run llm-qwen3-14b on a single GPU on ml-serve1012 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307757 (https://phabricator.wikimedia.org/T431268) [11:04:18] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply [11:04:23] !log elukey@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1005.eqiad.wmnet with OS trixie [11:04:46] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply [11:06:20] (03PS1) 10Marostegui: mariadb: Move s4 hosts to x4 [puppet] - 10https://gerrit.wikimedia.org/r/1307758 (https://phabricator.wikimedia.org/T404715) [11:06:35] (03CR) 10Kevin Bazira: "Thank you for working on this. Should the qwen3-14b deployment in the experimental ns be removed to free up resources for this qwen3-14b d" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307757 (https://phabricator.wikimedia.org/T431268) (owner: 10Bartosz Wójtowicz) [11:07:23] !log failover Ganeti master in codfw to ganeti2032 T430909 [11:07:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:07:27] T430909: codfw: rack B3 maintenance - https://phabricator.wikimedia.org/T430909 [11:09:07] (03CR) 10Marostegui: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1307758 (https://phabricator.wikimedia.org/T404715) (owner: 10Marostegui) [11:09:10] PROBLEM - ganeti-wconfd running on ganeti2048 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 110 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [11:09:54] (03CR) 10Bartosz Wójtowicz: "That is a good point! I think our LiftWing studio is currently using the experimental deployment as endpoint. @isarantopoulos@wikimedia.or" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307757 (https://phabricator.wikimedia.org/T431268) (owner: 10Bartosz Wójtowicz) [11:10:23] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, July 06 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1303490 (owner: 10Pushpaktiwari) [11:11:44] (03CR) 10Marostegui: "PCC: https://puppet-compiler.wmflabs.org/output/1307758/7138/" [puppet] - 10https://gerrit.wikimedia.org/r/1307758 (https://phabricator.wikimedia.org/T404715) (owner: 10Marostegui) [11:13:59] jouncebot: nowandnext [11:13:59] No deployments scheduled for the next 1 hour(s) and 46 minute(s) [11:13:59] In 1 hour(s) and 46 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260706T1300) [11:14:45] (03CR) 10Dreamy Jazz: [C:03+1] Enable WikimediaAntiAbuse on Beta Cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307750 (https://phabricator.wikimedia.org/T431023) (owner: 10Kosta Harlan) [11:15:31] (03CR) 10Dreamy Jazz: [C:03+1] WikimediaAntiAbuse: Register wmgUse config and load extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307717 (https://phabricator.wikimedia.org/T431023) (owner: 10Kosta Harlan) [11:31:42] !log installing nano security updates [11:31:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:33:04] (03CR) 10Ayounsi: [C:03+1] "That's great, thanks!" [alerts] - 10https://gerrit.wikimedia.org/r/1307432 (https://phabricator.wikimedia.org/T353323) (owner: 10Hnowlan) [11:34:32] 10ops-eqiad, 06Data-Persistence, 06DBA, 06DC-Ops, and 3 others: es7 primary (es1039.eqiad.wmnet) crashed - https://phabricator.wikimedia.org/T430764#12089863 (10VRiley-WMF) @Marostegui Of course, I will take a look at this and update it. [11:37:39] (03CR) 10Hnowlan: [C:03+2] Improve InterfaceSpeedError messages [alerts] - 10https://gerrit.wikimedia.org/r/1307432 (https://phabricator.wikimedia.org/T353323) (owner: 10Hnowlan) [11:37:44] (03CR) 10Kevin Bazira: [C:03+2] ml-services: deploy cope-b-a4b isvc image that emits vLLM metrics [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307708 (https://phabricator.wikimedia.org/T431136) (owner: 10Kevin Bazira) [11:39:28] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware: decommission mirror1001 - https://phabricator.wikimedia.org/T431088#12089895 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF [11:39:54] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12089902 (10VRiley-WMF) a:03VRiley-WMF [11:39:56] (03Merged) 10jenkins-bot: ml-services: deploy cope-b-a4b isvc image that emits vLLM metrics [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307708 (https://phabricator.wikimedia.org/T431136) (owner: 10Kevin Bazira) [11:40:20] 10ops-eqiad, 06Data-Persistence, 06DBA, 06DC-Ops, and 3 others: es7 primary (es1039.eqiad.wmnet) crashed - https://phabricator.wikimedia.org/T430764#12089906 (10VRiley-WMF) a:03VRiley-WMF [11:40:34] (03Merged) 10jenkins-bot: Improve InterfaceSpeedError messages [alerts] - 10https://gerrit.wikimedia.org/r/1307432 (https://phabricator.wikimedia.org/T353323) (owner: 10Hnowlan) [11:40:44] (03PS1) 10Majavah: team-wmcs: Port Memcached availability alert [alerts] - 10https://gerrit.wikimedia.org/r/1307762 (https://phabricator.wikimedia.org/T328502) [11:41:41] (03CR) 10David Caro: [C:03+1] "LGTM, just to make sure, this one we will get an alert already right? (if not we might want to add it before removing it from here)" [puppet] - 10https://gerrit.wikimedia.org/r/1307748 (owner: 10Majavah) [11:43:11] (03CR) 10David Caro: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1307739 (https://phabricator.wikimedia.org/T431261) (owner: 10Majavah) [11:43:27] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12089931 (10jcrespo) I forgot to mention, I consider data here lost/corrupted, the host was downtimed, feel free to do any operation with the server without asking. [11:43:36] (03CR) 10Majavah: [C:03+2] "yes, we already have Prometheus alerts that will fire if any systemd unit on the cloudcontrols fail" [puppet] - 10https://gerrit.wikimedia.org/r/1307748 (owner: 10Majavah) [11:46:45] (03CR) 10Majavah: [V:03+1 C:03+2] P:puppetserver::wmcs: Do not run facts cleanup on hosts with PuppetDB [puppet] - 10https://gerrit.wikimedia.org/r/1307739 (https://phabricator.wikimedia.org/T431261) (owner: 10Majavah) [11:47:38] (03PS1) 10Majavah: memcached: Allow disabling Icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/1307763 (https://phabricator.wikimedia.org/T384305) [11:47:44] (03PS1) 10Majavah: Disable Memcached Icinga checks for cloudcontrols/cloudwebs [puppet] - 10https://gerrit.wikimedia.org/r/1307764 (https://phabricator.wikimedia.org/T384305) [11:47:55] (03PS3) 10Krinkle: robots.php: Change Beta Cluster override from prepend to replace [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1265672 [11:49:09] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [11:53:59] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (DIFF 17): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/8927/consol" [puppet] - 10https://gerrit.wikimedia.org/r/1307763 (https://phabricator.wikimedia.org/T384305) (owner: 10Majavah) [11:56:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-int - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [11:57:33] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.5 point update - https://phabricator.wikimedia.org/T427072#12089987 (10MoritzMuehlenhoff) [11:57:44] !log installing curl security updates [11:57:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:00:33] (03PS2) 10Tiziano Fogli: lvs: enable nrpe2nodexp wrapper on check_rp_filter_disabled check [puppet] - 10https://gerrit.wikimedia.org/r/1200362 (https://phabricator.wikimedia.org/T407330) [12:01:15] RESOLVED: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-int - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [12:01:19] (03CR) 10Hnowlan: [V:03+1] "PCC SUCCESS (): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/8928/console" [puppet] - 10https://gerrit.wikimedia.org/r/1200362 (https://phabricator.wikimedia.org/T407330) (owner: 10Tiziano Fogli) [12:02:39] (03CR) 10Hnowlan: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/8929/co" [puppet] - 10https://gerrit.wikimedia.org/r/1200362 (https://phabricator.wikimedia.org/T407330) (owner: 10Tiziano Fogli) [12:04:03] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1265672 (owner: 10Krinkle) [12:05:01] (03Merged) 10jenkins-bot: robots.php: Change Beta Cluster override from prepend to replace [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1265672 (owner: 10Krinkle) [12:05:22] !log krinkle@deploy1003 Started scap sync-world: Backport for [[gerrit:1265672|robots.php: Change Beta Cluster override from prepend to replace]] [12:08:48] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2048.codfw.wmnet [12:09:12] !log krinkle@deploy1003 krinkle: Backport for [[gerrit:1265672|robots.php: Change Beta Cluster override from prepend to replace]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:09:38] 07sre-alert-triage, 06Data-Platform-SRE, 06Machine-Learning-Team (Q1 FY2026-27): Alert in need of triage: SmartNotHealthy (instance ml-serve1001:9100) - https://phabricator.wikimedia.org/T414969#12090062 (10isarantopoulos) [12:10:12] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2048.codfw.wmnet [12:17:17] (03PS1) 10Majavah: P:wmcs::proxy::static: Include hostname in access log [puppet] - 10https://gerrit.wikimedia.org/r/1307777 (https://phabricator.wikimedia.org/T431284) [12:19:50] (03PS1) 10Fabfur: cache::haproxy: add correlation id feature everywhere [puppet] - 10https://gerrit.wikimedia.org/r/1307778 (https://phabricator.wikimedia.org/T426379) [12:21:10] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1307779 (owner: 10L10n-bot) [12:21:18] (03CR) 10Btullis: [C:03+1] service: register the phabricator service [puppet] - 10https://gerrit.wikimedia.org/r/1305606 (https://phabricator.wikimedia.org/T430024) (owner: 10Brouberol) [12:21:34] (03CR) 10Btullis: [C:03+1] "Looks good to me." [puppet] - 10https://gerrit.wikimedia.org/r/1305607 (https://phabricator.wikimedia.org/T430024) (owner: 10Brouberol) [12:24:13] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.5 point update - https://phabricator.wikimedia.org/T427072#12090115 (10MoritzMuehlenhoff) [12:24:17] !log krinkle@deploy1003 krinkle: Continuing with deployment [12:29:25] (03PS1) 10Majavah: P:wmcs::novaproxy: Install HAProxy service [puppet] - 10https://gerrit.wikimedia.org/r/1307782 (https://phabricator.wikimedia.org/T429930) [12:29:28] (03PS1) 10Majavah: acme_chief: Add healthz endpoint to acme-challenge vhost [puppet] - 10https://gerrit.wikimedia.org/r/1307783 (https://phabricator.wikimedia.org/T429930) [12:29:31] (03PS1) 10Majavah: P:wmcs::novaproxy: Handle HTTP redirects with HAProxy [puppet] - 10https://gerrit.wikimedia.org/r/1307784 (https://phabricator.wikimedia.org/T429930) [12:30:11] (03Abandoned) 10Brouberol: service: register the phabricator service [puppet] - 10https://gerrit.wikimedia.org/r/1305606 (https://phabricator.wikimedia.org/T430024) (owner: 10Brouberol) [12:30:18] (03Abandoned) 10Brouberol: service_proxy: register phabricator services [puppet] - 10https://gerrit.wikimedia.org/r/1305607 (https://phabricator.wikimedia.org/T430024) (owner: 10Brouberol) [12:30:33] !log krinkle@deploy1003 Finished scap sync-world: Backport for [[gerrit:1265672|robots.php: Change Beta Cluster override from prepend to replace]] (duration: 25m 11s) [12:33:09] (03PS1) 10Brouberol: dse-k8s-eqiad/test-kitchen: allow both instances to egress to phabricator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307788 (https://phabricator.wikimedia.org/T430024) [12:33:58] 10SRE-tools, 06Infrastructure-Foundations: Diffscan: investigate IPv6 support and explore other scanning tooling - https://phabricator.wikimedia.org/T265329#12090166 (10ayounsi) I created a quick Netbox [[ https://netbox.wikimedia.org/extras/export-templates/35/ | export template ]] that lists all the known IP... [12:42:06] (03CR) 10Cathal Mooney: [C:03+2] dse-k8s: add new POD IPv4 block 10.68.0.0/17 to prefix lists [homer/public] - 10https://gerrit.wikimedia.org/r/1307396 (https://phabricator.wikimedia.org/T430658) (owner: 10Cathal Mooney) [12:42:26] (03CR) 10Ssingh: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1307778 (https://phabricator.wikimedia.org/T426379) (owner: 10Fabfur) [12:43:33] (03Merged) 10jenkins-bot: dse-k8s: add new POD IPv4 block 10.68.0.0/17 to prefix lists [homer/public] - 10https://gerrit.wikimedia.org/r/1307396 (https://phabricator.wikimedia.org/T430658) (owner: 10Cathal Mooney) [12:45:16] RECOVERY - Postfix SMTP on crm2001 is OK: OK - Certificate crm2001.codfw.wmnet will expire on Mon 03 Aug 2026 12:10:00 PM GMT +0000. https://wikitech.wikimedia.org/wiki/Mail%23Troubleshooting [12:47:09] (03CR) 10Ssingh: [C:03+1] "Looks good, let's verify it quickly once you merge that this is working as intended. Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1307778 (https://phabricator.wikimedia.org/T426379) (owner: 10Fabfur) [12:47:10] (03CR) 10Santiago Faci: [C:03+2] dse-k8s-eqiad/test-kitchen: allow both instances to egress to phabricator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307788 (https://phabricator.wikimedia.org/T430024) (owner: 10Brouberol) [12:49:56] (03Merged) 10jenkins-bot: dse-k8s-eqiad/test-kitchen: allow both instances to egress to phabricator [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307788 (https://phabricator.wikimedia.org/T430024) (owner: 10Brouberol) [12:51:51] FIRING: Transit, peering or transport OUT traffic above 90% capacity - cr1-codfw:xe-1/1/1:0 (Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294)) #page - https://w.wiki/Gbyf - https://grafana.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?var-site=codfw+prometheus%2Fops&var-device=cr1-codfw:9804&var-interface=xe-1%2F1%2F1%3A0 - https://alerts.wikimedia.org/?q=alertname%3DTransitPeeringTransportOutSa [12:52:08] !ack [12:52:08] 8148 (ACKED) TransitPeeringTransportOutSaturation network sre (cr1-codfw:9804 Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294) {#12252_12295-1} xe-1/1/1:0 gnmi codfw) [12:52:43] here, thanks sukhe [12:56:07] (03CR) 10Fabfur: [C:03+2] cache::haproxy: add correlation id feature everywhere [puppet] - 10https://gerrit.wikimedia.org/r/1307778 (https://phabricator.wikimedia.org/T426379) (owner: 10Fabfur) [12:56:51] RESOLVED: Transit, peering or transport OUT traffic above 90% capacity - cr1-codfw:xe-1/1/1:0 (Transport: cr4-ulsfo:xe-0/1/1 (Lumen, ... [12:56:51] 442550294)) #page - https://w.wiki/Gbyf - https://grafana.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?var-site=codfw+prometheus%2Fops&var-device=cr1-codfw:9804&var-interface=xe-1%2F1%2F1%3A0 - https://alerts.wikimedia.org/?q=alertname%3DTransitPeeringTransportOutSaturation [12:58:02] (03PS1) 10Cathal Mooney: HE Transport: enable OSPF on eqsin to eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1307795 (https://phabricator.wikimedia.org/T424839) [12:58:39] (03CR) 10Ilias Sarantopoulos: "Yes, feel free to go ahead we can change the endpoint afterwards in https://gitlab.wikimedia.org/repos/machine-learning/liftwing-studio/-/" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307757 (https://phabricator.wikimedia.org/T431268) (owner: 10Bartosz Wójtowicz) [12:59:54] (03CR) 10CDanis: [C:03+1] hiddenparma: add known fingerprints file (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1307013 (owner: 10Giuseppe Lavagetto) [13:00:05] urbanecm and TheresNoTime: UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260706T1300). Please do the needful. [13:00:05] No Gerrit patches in the queue for this window AFAICS. [13:05:17] (03PS6) 10Blake: bgp: Remove :0 from instance and remote_instance names. [alerts] - 10https://gerrit.wikimedia.org/r/1307787 (https://phabricator.wikimedia.org/T430290) [13:07:43] (03CR) 10Ayounsi: [C:03+1] HE Transport: enable OSPF on eqsin to eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1307795 (https://phabricator.wikimedia.org/T424839) (owner: 10Cathal Mooney) [13:08:03] (03CR) 10Majavah: "fwiw there's `| stripPort` that'll do the same thing but in a more readable way than manual regexes" [alerts] - 10https://gerrit.wikimedia.org/r/1307787 (https://phabricator.wikimedia.org/T430290) (owner: 10Blake) [13:08:55] (03CR) 10Hnowlan: [V:03+1] "PCC SUCCESS (): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/8930/console" [puppet] - 10https://gerrit.wikimedia.org/r/1307159 (https://phabricator.wikimedia.org/T407117) (owner: 10Hnowlan) [13:08:55] (03CR) 10Blake: "Oh, brilliant, thank you! I'll make the change." [alerts] - 10https://gerrit.wikimedia.org/r/1307787 (https://phabricator.wikimedia.org/T430290) (owner: 10Blake) [13:10:48] (03CR) 10Hnowlan: [V:03+1] "PCC SUCCESS (DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/8931/console" [puppet] - 10https://gerrit.wikimedia.org/r/1307159 (https://phabricator.wikimedia.org/T407117) (owner: 10Hnowlan) [13:12:12] (03PS1) 10Santiago Faci: test-kitchen Chart: Set phabricator discovery URL as the default one [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307797 (https://phabricator.wikimedia.org/T428984) [13:12:27] (03PS7) 10Blake: bgp: Remove :0 from instance and remote_instance names. [alerts] - 10https://gerrit.wikimedia.org/r/1307787 (https://phabricator.wikimedia.org/T430290) [13:12:42] (03PS8) 10Blake: bgp: Remove port from instance and remote_instance names. [alerts] - 10https://gerrit.wikimedia.org/r/1307787 (https://phabricator.wikimedia.org/T430290) [13:12:57] (03CR) 10Bking: [C:03+1] cirrussearch: adding max_clause_count parameter [puppet] - 10https://gerrit.wikimedia.org/r/1307470 (https://phabricator.wikimedia.org/T431086) (owner: 10Atsuko) [13:20:05] (03CR) 10Cathal Mooney: [C:03+2] HE Transport: enable OSPF on eqsin to eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1307795 (https://phabricator.wikimedia.org/T424839) (owner: 10Cathal Mooney) [13:22:02] (03PS6) 10Klausman: role::ml_k8s::codfw::master: enable IPIP encapsulation [puppet] - 10https://gerrit.wikimedia.org/r/1306877 (https://phabricator.wikimedia.org/T420438) [13:23:11] (03Merged) 10jenkins-bot: HE Transport: enable OSPF on eqsin to eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1307795 (https://phabricator.wikimedia.org/T424839) (owner: 10Cathal Mooney) [13:23:52] (03PS1) 10Btullis: datahub: explicitly disable metadata service auth when off [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307799 (https://phabricator.wikimedia.org/T402408) [13:27:08] (03CR) 10Klausman: [C:03+2] alertmanager: Add Slack alerts to public slack channel for ML team [puppet] - 10https://gerrit.wikimedia.org/r/1296529 (owner: 10Ilias Sarantopoulos) [13:27:14] (03CR) 10Btullis: [C:03+1] "Looks good to me." [puppet] - 10https://gerrit.wikimedia.org/r/1306877 (https://phabricator.wikimedia.org/T420438) (owner: 10Klausman) [13:29:09] (03CR) 10Klausman: [C:03+2] role::ml_k8s::codfw::master: enable IPIP encapsulation [puppet] - 10https://gerrit.wikimedia.org/r/1306877 (https://phabricator.wikimedia.org/T420438) (owner: 10Klausman) [13:32:01] (03CR) 10Btullis: [C:03+2] datahub: explicitly disable metadata service auth when off [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307799 (https://phabricator.wikimedia.org/T402408) (owner: 10Btullis) [13:32:21] (03CR) 10Ladsgroup: [C:03+1] mariadb: Move s4 hosts to x4 [puppet] - 10https://gerrit.wikimedia.org/r/1307758 (https://phabricator.wikimedia.org/T404715) (owner: 10Marostegui) [13:33:34] (03PS2) 10Majavah: P:wmcs::novaproxy: Handle HTTP redirects with HAProxy [puppet] - 10https://gerrit.wikimedia.org/r/1307784 (https://phabricator.wikimedia.org/T429930) [13:34:09] (03Merged) 10jenkins-bot: datahub: explicitly disable metadata service auth when off [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307799 (https://phabricator.wikimedia.org/T402408) (owner: 10Btullis) [13:35:03] !log klausman@cumin2002 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-serve-master-codfw [13:35:07] !log klausman@cumin2002 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve-ctrl2001.codfw.wmnet [13:35:08] !log klausman@cumin2002 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve-ctrl2001.codfw.wmnet [13:38:31] !log klausman@cumin2002 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve-ctrl2001.codfw.wmnet [13:38:33] !log klausman@cumin2002 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve-ctrl2001.codfw.wmnet [13:38:39] !log klausman@cumin2002 START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve-ctrl2002.codfw.wmnet [13:38:40] !log klausman@cumin2002 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve-ctrl2002.codfw.wmnet [13:42:11] !log klausman@cumin2002 START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve-ctrl2002.codfw.wmnet [13:42:13] !log klausman@cumin2002 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve-ctrl2002.codfw.wmnet [13:42:13] !log klausman@cumin2002 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-serve-master-codfw [13:42:15] (03PS3) 10Giuseppe Lavagetto: hiddenparma: add known fingerprints file [puppet] - 10https://gerrit.wikimedia.org/r/1307013 [13:42:58] (03CR) 10Giuseppe Lavagetto: [C:03+2] hiddenparma: add known fingerprints file (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1307013 (owner: 10Giuseppe Lavagetto) [13:43:19] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12090505 (10VRiley-WMF) Understood, thank you! [13:45:31] (03PS1) 10Cathal Mooney: WMCS LDAP Groups: change approver to Belinda Liviero [puppet] - 10https://gerrit.wikimedia.org/r/1307802 (https://phabricator.wikimedia.org/T431010) [13:45:45] !log cwilliams@cumin1003 START - Cookbook sre.mysql.major-upgrade [13:45:45] !log cwilliams@cumin1003 dbmaint on s4@codfw T429893 [13:45:49] T429893: Migrate s4 section to Debian Trixie - https://phabricator.wikimedia.org/T429893 [13:45:50] !log cwilliams@cumin1003 END (ERROR) - Cookbook sre.mysql.major-upgrade (exit_code=97) [13:46:35] (03CR) 10Bartosz Wójtowicz: "Perfect! I'll first merge this patch, next change the endpoint for liftwing studio and at the end remove the experimental deployment :)" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307757 (https://phabricator.wikimedia.org/T431268) (owner: 10Bartosz Wójtowicz) [13:51:20] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to cloudcontrol servers for tlepage - https://phabricator.wikimedia.org/T431010#12090532 (10cmooney) @BLiviero-WMF I had a quick chat with Mark and he agreed you are best placed to approve requests for the WMCS groups from now on (see change... [13:51:28] !log cwilliams@cumin1003 START - Cookbook sre.mysql.multiinstance_reboot for db-test[2001-2002].codfw.wmnet [13:55:36] 06SRE: Problems with SRE Team Vacations Calendar sync - https://phabricator.wikimedia.org/T431303#12090538 (10Peachey88) [13:56:00] (03PS16) 10Arnaudb: trafficserver: add a map for gitlab instances as a backend [puppet] - 10https://gerrit.wikimedia.org/r/1290731 (https://phabricator.wikimedia.org/T425441) [13:57:37] (03CR) 10Mark Bergsma: [C:03+1] WMCS LDAP Groups: change approver to Belinda Liviero [puppet] - 10https://gerrit.wikimedia.org/r/1307802 (https://phabricator.wikimedia.org/T431010) (owner: 10Cathal Mooney) [13:58:54] (03CR) 10Muehlenhoff: WMCS LDAP Groups: change approver to Belinda Liviero (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1307802 (https://phabricator.wikimedia.org/T431010) (owner: 10Cathal Mooney) [13:59:39] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db-test[2001-2002].codfw.wmnet [13:59:57] (03CR) 10Cathal Mooney: [C:03+2] WMCS LDAP Groups: change approver to Belinda Liviero [puppet] - 10https://gerrit.wikimedia.org/r/1307802 (https://phabricator.wikimedia.org/T431010) (owner: 10Cathal Mooney) [14:00:41] (03CR) 10Cathal Mooney: [C:03+2] WMCS LDAP Groups: change approver to Belinda Liviero (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1307802 (https://phabricator.wikimedia.org/T431010) (owner: 10Cathal Mooney) [14:04:02] (03CR) 10Kevin Bazira: [C:03+1] ml-services: run llm-qwen3-14b on a single GPU on ml-serve1012 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307757 (https://phabricator.wikimedia.org/T431268) (owner: 10Bartosz Wójtowicz) [14:07:31] !log cwilliams@cumin1003 START - Cookbook sre.mysql.multiinstance_reboot for db2230.codfw.wmnet [14:09:49] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: run llm-qwen3-14b on a single GPU on ml-serve1012 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307757 (https://phabricator.wikimedia.org/T431268) (owner: 10Bartosz Wójtowicz) [14:12:00] (03Merged) 10jenkins-bot: ml-services: run llm-qwen3-14b on a single GPU on ml-serve1012 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307757 (https://phabricator.wikimedia.org/T431268) (owner: 10Bartosz Wójtowicz) [14:12:22] (03PS1) 10Krinkle: Re-enable wgTrackMediaRequestProvenance on pilot wikis (group1) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307812 (https://phabricator.wikimedia.org/T414338) [14:13:38] !log cwilliams@cumin1003 END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for db2230.codfw.wmnet [14:13:45] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [14:14:07] !log Setup x4 eqiad topology T404715 [14:14:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:14:09] T404715: Setup x4 section - https://phabricator.wikimedia.org/T404715 [14:18:58] (03CR) 10Marostegui: [C:03+2] mariadb: Move s4 hosts to x4 [puppet] - 10https://gerrit.wikimedia.org/r/1307758 (https://phabricator.wikimedia.org/T404715) (owner: 10Marostegui) [14:20:23] (03CR) 10Scott French: mediawiki/php: don't pin libpcre for bookworm (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1307128 (https://phabricator.wikimedia.org/T423714) (owner: 10Kamila Součková) [14:29:53] (03CR) 10Klausman: [C:03+2] admin_ng: Fix KServe view RBAC on install_kserve_resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306892 (owner: 10Bartosz Wójtowicz) [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260706T1430) [14:31:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:31:23] (03CR) 10Brouberol: "Sorry, I'm only back from PTO today. Per module would be lovely!" [puppet] - 10https://gerrit.wikimedia.org/r/1305987 (https://phabricator.wikimedia.org/T372666) (owner: 10JHathaway) [14:32:33] !log upgrade all bookworm hosts to pywmflib 3.1 - T430552 [14:32:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:32:37] T430552: Deploy wmflib 3.0.0 to production - https://phabricator.wikimedia.org/T430552 [14:33:37] !log rolled out spicerack on all cumin nodes - T429699 [14:33:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:33:40] T429699: Add the management password from pwstore to the cumin hosts - https://phabricator.wikimedia.org/T429699 [14:36:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:36:48] (03CR) 10Klausman: [V:03+2 C:03+2] admin_ng: Fix KServe view RBAC on install_kserve_resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306892 (owner: 10Bartosz Wójtowicz) [14:37:40] 10SRE-swift-storage, 06Infrastructure-Foundations, 06MediaWiki-Media-Platform-Team, 10MediaWiki-Uploading, 07Wikimedia-production-error: MediaWiki\Upload\Exception\UploadChunkFileException: Error storing file in '{chunkPath}': backend-fail-internal; loca... - https://phabricator.wikimedia.org/T430986#12090755 [14:39:56] !log bking@localhost raised node concurrent recoveries from 4 to 7 on eqiad cirrussearch clusters T431311 [14:39:57] T431311: Migrate production OpenSearch clusters from 1.x-2.x - EQIAD - https://phabricator.wikimedia.org/T431311 [14:42:32] (03PS3) 10Atsuko: cirrussearch: adding max_clause_count parameter [puppet] - 10https://gerrit.wikimedia.org/r/1307470 (https://phabricator.wikimedia.org/T431086) [14:43:44] (03CR) 10Bking: [C:03+2] cirrussearch: adding max_clause_count parameter [puppet] - 10https://gerrit.wikimedia.org/r/1307470 (https://phabricator.wikimedia.org/T431086) (owner: 10Atsuko) [14:44:17] (03PS1) 10Atsuko: cirrussearch: set hieradata for OpenSearch 1->2 migration [puppet] - 10https://gerrit.wikimedia.org/r/1307814 (https://phabricator.wikimedia.org/T431311) [14:45:06] (03CR) 10Bking: [C:03+1] cirrussearch: set hieradata for OpenSearch 1->2 migration [puppet] - 10https://gerrit.wikimedia.org/r/1307814 (https://phabricator.wikimedia.org/T431311) (owner: 10Atsuko) [14:45:59] (03CR) 10Atsuko: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1307814 (https://phabricator.wikimedia.org/T431311) (owner: 10Atsuko) [14:46:04] (03CR) 10Bking: [C:03+1] "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1307814 (https://phabricator.wikimedia.org/T431311) (owner: 10Atsuko) [14:47:49] !log elukey@cumin1003 START - Cookbook sre.hosts.reimage for host aux-k8s-etcd1003.eqiad.wmnet with OS bookworm [14:49:01] !log bking@cumin2003 conftool action : set/pooled=false; selector: dnsdisc=${n},name=${DC} [14:49:02] !log bking@cumin2003 conftool action : set/pooled=false; selector: dnsdisc=${n},name=${DC} [14:49:03] !log bking@cumin2003 conftool action : set/pooled=false; selector: dnsdisc=${n},name=${DC} [14:49:19] (03CR) 10Filippo Giunchedi: [C:03+1] acme_chief: Add healthz endpoint to acme-challenge vhost [puppet] - 10https://gerrit.wikimedia.org/r/1307783 (https://phabricator.wikimedia.org/T429930) (owner: 10Majavah) [14:49:28] 06SRE, 10SRE-Access-Requests: Requesting access to cloudcontrol servers for tlepage - https://phabricator.wikimedia.org/T431010#12090827 (10BLiviero-WMF) Hi Cathal, thank you for reaching out, I approve the request [14:49:44] (03CR) 10Jforrester: "Reedy found T411516 which is the trial task." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307716 (https://phabricator.wikimedia.org/T431023) (owner: 10Kosta Harlan) [14:49:51] !log bking@cumin2003 conftool action : set/pooled=false; selector: dnsdisc=search,name=eqiad [14:49:52] !log bking@cumin2003 conftool action : set/pooled=false; selector: dnsdisc=search-omega,name=eqiad [14:49:52] !log bking@cumin2003 conftool action : set/pooled=false; selector: dnsdisc=search-psi,name=eqiad [14:50:16] (03CR) 10Filippo Giunchedi: [C:03+1] P:wmcs::proxy::static: Include hostname in access log [puppet] - 10https://gerrit.wikimedia.org/r/1307777 (https://phabricator.wikimedia.org/T431284) (owner: 10Majavah) [14:50:38] (03CR) 10Majavah: [C:03+2] P:wmcs::proxy::static: Include hostname in access log [puppet] - 10https://gerrit.wikimedia.org/r/1307777 (https://phabricator.wikimedia.org/T431284) (owner: 10Majavah) [14:52:05] (03CR) 10Filippo Giunchedi: [C:03+1] team-wmcs: Port Memcached availability alert [alerts] - 10https://gerrit.wikimedia.org/r/1307762 (https://phabricator.wikimedia.org/T328502) (owner: 10Majavah) [14:52:26] (03CR) 10Majavah: [C:03+2] team-wmcs: Port Memcached availability alert [alerts] - 10https://gerrit.wikimedia.org/r/1307762 (https://phabricator.wikimedia.org/T328502) (owner: 10Majavah) [14:53:41] (03CR) 10Filippo Giunchedi: [C:03+1] haproxy: Properly absent Icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/1307753 (owner: 10Majavah) [14:54:12] 06SRE, 06Data-Engineering, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 07Sustainability (Incident Followup): The webrequest_sampled_live data pipeline and its query tools have become mission-critical and require re-engineering for resilience - https://phabricator.wikimedia.org/T431112#12090840 (10BTullis... [14:54:12] (03CR) 10Filippo Giunchedi: [C:03+1] P:monitoring: Remove unused check_fresh_files_in_dir check [puppet] - 10https://gerrit.wikimedia.org/r/1307747 (owner: 10Majavah) [14:54:23] (03Merged) 10jenkins-bot: team-wmcs: Port Memcached availability alert [alerts] - 10https://gerrit.wikimedia.org/r/1307762 (https://phabricator.wikimedia.org/T328502) (owner: 10Majavah) [14:54:30] (03CR) 10Majavah: [V:03+1 C:03+2] haproxy: Properly absent Icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/1307753 (owner: 10Majavah) [14:54:57] (03CR) 10Atsuko: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1307814 (https://phabricator.wikimedia.org/T431311) (owner: 10Atsuko) [14:55:22] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, July 06 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307812 (https://phabricator.wikimedia.org/T414338) (owner: 10Krinkle) [14:55:24] (03PS2) 10Majavah: P:monitoring: Remove unused check_fresh_files_in_dir check [puppet] - 10https://gerrit.wikimedia.org/r/1307747 [14:55:50] (03PS2) 10Atsuko: cirrussearch: set hieradata for OpenSearch 1->2 migration [puppet] - 10https://gerrit.wikimedia.org/r/1307814 (https://phabricator.wikimedia.org/T431311) [14:55:56] (03CR) 10Arnaudb: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1290731 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [14:56:35] (03CR) 10Filippo Giunchedi: "LGTM, though I'll let o11y vote" [puppet] - 10https://gerrit.wikimedia.org/r/1307745 (owner: 10Majavah) [14:56:37] (03CR) 10Atsuko: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1307814 (https://phabricator.wikimedia.org/T431311) (owner: 10Atsuko) [14:56:50] (03CR) 10Filippo Giunchedi: [C:03+1] dynamicproxy: Validate each proxy is linked to exactly one domain [puppet] - 10https://gerrit.wikimedia.org/r/1307713 (https://phabricator.wikimedia.org/T429960) (owner: 10Majavah) [14:56:51] (03PS1) 10CWilliams: sre.mysql.pool: --skip-safety-checks fails for parsercache hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1307815 (https://phabricator.wikimedia.org/T429878) [14:56:54] (03PS2) 10Bartosz Wójtowicz: admin_ng: Fix KServe view RBAC on install_kserve_resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306892 [14:57:27] (03CR) 10Filippo Giunchedi: [C:03+1] dynamicproxy: Do not allow editing a proxy with the create endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1307712 (https://phabricator.wikimedia.org/T429960) (owner: 10Majavah) [14:57:38] !log elukey@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-etcd1003.eqiad.wmnet with reason: host reimage [14:57:50] (03CR) 10Majavah: [C:03+2] P:monitoring: Remove unused check_fresh_files_in_dir check [puppet] - 10https://gerrit.wikimedia.org/r/1307747 (owner: 10Majavah) [14:58:13] (03CR) 10Atsuko: [C:03+2] cirrussearch: set hieradata for OpenSearch 1->2 migration [puppet] - 10https://gerrit.wikimedia.org/r/1307814 (https://phabricator.wikimedia.org/T431311) (owner: 10Atsuko) [15:00:24] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host cirrussearch1069.eqiad.wmnet with OS trixie [15:00:45] FIRING: [3x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [15:05:14] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-etcd1003.eqiad.wmnet with reason: host reimage [15:05:17] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host cirrussearch1113.eqiad.wmnet with OS trixie [15:05:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in codfw - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [15:07:16] (03Abandoned) 10Bartosz Wójtowicz: admin_ng: Fix KServe view RBAC on install_kserve_resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306892 (owner: 10Bartosz Wójtowicz) [15:08:06] (03CR) 10Btullis: [C:03+2] Enable unit tests in CI and fix pre-existing test failures [dumps] - 10https://gerrit.wikimedia.org/r/1307379 (owner: 10Kosta Harlan) [15:10:17] seems like we have puppet failure because of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1307747 [15:10:31] pinging taavi [15:10:40] atsukoito: see -sre [15:11:19] yup, found already, thanks! [15:11:30] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply [15:11:32] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host cirrussearch1080.eqiad.wmnet with OS trixie [15:11:57] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: apply [15:13:46] (03PS2) 10Majavah: dynamicproxy: Improve HTTP body parameter validation [puppet] - 10https://gerrit.wikimedia.org/r/1307711 [15:13:46] (03PS2) 10Majavah: dynamicproxy: Do not allow editing a proxy with the create endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1307712 (https://phabricator.wikimedia.org/T429960) [15:13:46] (03PS2) 10Majavah: dynamicproxy: Validate each proxy is linked to exactly one domain [puppet] - 10https://gerrit.wikimedia.org/r/1307713 (https://phabricator.wikimedia.org/T429960) [15:14:54] (03CR) 10Majavah: [C:03+2] dynamicproxy: Improve HTTP body parameter validation [puppet] - 10https://gerrit.wikimedia.org/r/1307711 (owner: 10Majavah) [15:15:04] (03CR) 10Majavah: [C:03+2] dynamicproxy: Do not allow editing a proxy with the create endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1307712 (https://phabricator.wikimedia.org/T429960) (owner: 10Majavah) [15:15:15] (03CR) 10Majavah: [C:03+2] dynamicproxy: Validate each proxy is linked to exactly one domain [puppet] - 10https://gerrit.wikimedia.org/r/1307713 (https://phabricator.wikimedia.org/T429960) (owner: 10Majavah) [15:16:07] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1069.eqiad.wmnet with reason: host reimage [15:18:28] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1113.eqiad.wmnet with reason: host reimage [15:18:53] (03PS2) 10Majavah: acme_chief: Add healthz endpoint to acme-challenge vhost [puppet] - 10https://gerrit.wikimedia.org/r/1307783 (https://phabricator.wikimedia.org/T429930) [15:18:53] (03PS2) 10Majavah: P:wmcs::novaproxy: Install HAProxy service [puppet] - 10https://gerrit.wikimedia.org/r/1307782 (https://phabricator.wikimedia.org/T429930) [15:18:53] (03PS3) 10Majavah: P:wmcs::novaproxy: Handle HTTP redirects with HAProxy [puppet] - 10https://gerrit.wikimedia.org/r/1307784 (https://phabricator.wikimedia.org/T429930) [15:19:21] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1069.eqiad.wmnet with reason: host reimage [15:19:56] (03CR) 10Majavah: [C:03+2] acme_chief: Add healthz endpoint to acme-challenge vhost [puppet] - 10https://gerrit.wikimedia.org/r/1307783 (https://phabricator.wikimedia.org/T429930) (owner: 10Majavah) [15:20:49] 06SRE, 06collaboration-services, 10Wikimedia-Mailing-lists: Terms of use for Mailman - https://phabricator.wikimedia.org/T431096#12090925 (10LSobanski) p:05Triage→03Medium [15:22:11] (03CR) 10Kosta Harlan: [C:03+1] Filter change_tag and change_tag_def dumps [dumps] - 10https://gerrit.wikimedia.org/r/1307262 (https://phabricator.wikimedia.org/T386456) (owner: 10Dreamy Jazz) [15:22:24] (03CR) 10Btullis: [C:03+2] Filter change_tag and change_tag_def dumps [dumps] - 10https://gerrit.wikimedia.org/r/1307262 (https://phabricator.wikimedia.org/T386456) (owner: 10Dreamy Jazz) [15:23:11] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1113.eqiad.wmnet with reason: host reimage [15:24:12] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.5 point update - https://phabricator.wikimedia.org/T427072#12090940 (10MoritzMuehlenhoff) [15:26:20] 06SRE, 06collaboration-services, 10Wikimedia-Mailing-lists: Terms of use for Mailman - https://phabricator.wikimedia.org/T431096#12090946 (10LSobanski) a:03LSobanski [15:27:06] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1080.eqiad.wmnet with reason: host reimage [15:30:05] jan_drewniak: How many deployers does it take to do Wikimedia Portals Update deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260706T1530). [15:30:22] (03CR) 10Kosta Harlan: "thanks!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307716 (https://phabricator.wikimedia.org/T431023) (owner: 10Kosta Harlan) [15:30:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in codfw - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [15:32:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.93% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:33:38] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1080.eqiad.wmnet with reason: host reimage [15:34:29] !log klausman@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [15:35:45] RESOLVED: [4x] WidespreadPuppetFailure: Puppet has failed in codfw - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [15:36:08] !log klausman@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [15:37:15] !log klausman@deploy1003 helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [15:37:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.52% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [15:37:36] 06SRE, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Do we need prometheus-ethtool-exporter? - https://phabricator.wikimedia.org/T371375#12090984 (10LSobanski) Is there a plan to also remove prometheus-ethtool-exporter from Trixie hosts provisioned before the change was merged? Collab gets lo... [15:38:42] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1069.eqiad.wmnet with OS trixie [15:38:57] !log klausman@deploy1003 helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [15:39:38] !log klausman@deploy1003 helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [15:40:29] !log klausman@deploy1003 helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [15:41:07] !log dancy@deploy1003 Installing scap version "4.272.0" for 158 host(s) [15:42:23] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1113.eqiad.wmnet with OS trixie [15:43:35] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host cirrussearch1103.eqiad.wmnet with OS trixie [15:44:45] 06SRE, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Do we need prometheus-ethtool-exporter? - https://phabricator.wikimedia.org/T371375#12091022 (10MoritzMuehlenhoff) >>! In T371375#12090984, @LSobanski wrote: > Is there a plan to also remove prometheus-ethtool-exporter from Trixie hosts pro... [15:45:18] !log dancy@deploy1003 Installation of scap version "4.272.0" completed for 158 hosts [15:46:29] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host cirrussearch1089.eqiad.wmnet with OS trixie [15:46:51] (03CR) 10SBassett: [C:03+1] Revert "varnish: Add CSP report-only header value" [puppet] - 10https://gerrit.wikimedia.org/r/1307474 (owner: 10Ssingh) [15:47:39] 10ops-ulsfo, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: ULSFO: OOB IPV6 down - https://phabricator.wikimedia.org/T430599#12091030 (10Papaul) ` Hello Papaul, Apologies for the late response. We are currently coordinating with our internal team for further investigation. We will keep yo... [15:50:51] FIRING: Transit, peering or transport OUT traffic above 90% capacity - cr1-codfw:xe-1/1/1:0 (Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294)) #page - https://w.wiki/Gbyf - https://grafana.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?var-site=codfw+prometheus%2Fops&var-device=cr1-codfw:9804&var-interface=xe-1%2F1%2F1%3A0 - https://alerts.wikimedia.org/?q=alertname%3DTransitPeeringTransportOutSa [15:51:04] 10ops-eqiad, 06Data-Persistence, 06DBA, 06DC-Ops, and 3 others: es7 primary (es1039.eqiad.wmnet) crashed - https://phabricator.wikimedia.org/T430764#12091040 (10VRiley-WMF) Hey @Marostegui BIOS and iDRAC have been updated and it seems like it boots just fine now. Should it be okay to close this out? [15:51:12] !ack [15:51:13] 8149 (ACKED) TransitPeeringTransportOutSaturation network sre (cr1-codfw:9804 Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294) {#12252_12295-1} xe-1/1/1:0 gnmi codfw) [15:54:41] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1080.eqiad.wmnet with OS trixie [15:55:28] (03CR) 10Hnowlan: [C:03+1] opensearch: add config required to authenticate curator requests [puppet] - 10https://gerrit.wikimedia.org/r/1306969 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [15:55:44] !ack [15:55:44] All incidents are already acked. [15:56:04] 06SRE, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Do we need prometheus-ethtool-exporter? - https://phabricator.wikimedia.org/T371375#12091066 (10LSobanski) You are right, my assumption was invalid and this was in fact a Bookworm host. [15:56:51] (03CR) 10Ssingh: [C:03+1] cache::haproxy: Enable jwt for upload clusters [puppet] - 10https://gerrit.wikimedia.org/r/1305923 (https://phabricator.wikimedia.org/T400238) (owner: 10BCornwall) [15:58:53] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host cirrussearch1096.eqiad.wmnet with OS trixie [16:00:25] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1103.eqiad.wmnet with reason: host reimage [16:00:30] 10ops-eqiad, 06Data-Persistence, 06DBA, 06DC-Ops, and 3 others: es7 primary (es1039.eqiad.wmnet) crashed - https://phabricator.wikimedia.org/T430764#12091077 (10Marostegui) Can you assign it back to me becasue there're a few things required from my side. Thanks! [16:00:51] RESOLVED: Transit, peering or transport OUT traffic above 90% capacity - cr1-codfw:xe-1/1/1:0 (Transport: cr4-ulsfo:xe-0/1/1 (Lumen, ... [16:00:51] 442550294)) #page - https://w.wiki/Gbyf - https://grafana.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?var-site=codfw+prometheus%2Fops&var-device=cr1-codfw:9804&var-interface=xe-1%2F1%2F1%3A0 - https://alerts.wikimedia.org/?q=alertname%3DTransitPeeringTransportOutSaturation [16:01:37] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1089.eqiad.wmnet with reason: host reimage [16:02:12] !log elukey@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-etcd1003.eqiad.wmnet with OS bookworm [16:05:15] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1103.eqiad.wmnet with reason: host reimage [16:08:40] 06SRE, 06Data-Engineering, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 07Sustainability (Incident Followup): The webrequest_sampled_live data pipeline and its query tools have become mission-critical and require re-engineering for resilience - https://phabricator.wikimedia.org/T431112#12091106 (10elukey... [16:09:05] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1089.eqiad.wmnet with reason: host reimage [16:09:22] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:14:01] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1096.eqiad.wmnet with reason: host reimage [16:14:22] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:17:12] 06SRE, 06Data-Engineering, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 07Sustainability (Incident Followup): The webrequest_sampled_live data pipeline and its query tools have become mission-critical and require re-engineering for resilience - https://phabricator.wikimedia.org/T431112#12091128 (10CDanis... [16:18:35] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12091133 (10VRiley-WMF) I have opened up a ticket for this unit. It's good it failed now, as the service ends on this unit later this month. Dell SR ticket number 228686846 [16:19:15] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1096.eqiad.wmnet with reason: host reimage [16:24:53] (03CR) 10Andrew Bogott: [C:03+2] "self-merging because this is increasingly urgent for PAWS" [puppet] - 10https://gerrit.wikimedia.org/r/1306398 (https://phabricator.wikimedia.org/T429578) (owner: 10Andrew Bogott) [16:28:01] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1103.eqiad.wmnet with OS trixie [16:30:38] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1089.eqiad.wmnet with OS trixie [16:33:46] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host cirrussearch1084.eqiad.wmnet with OS trixie [16:34:42] (03CR) 10Marostegui: [C:03+1] sre.mysql.pool: --skip-safety-checks fails for parsercache hosts [cookbooks] - 10https://gerrit.wikimedia.org/r/1307815 (https://phabricator.wikimedia.org/T429878) (owner: 10CWilliams) [16:35:42] (03PS1) 10Andrew Bogott: profile::wmcs::nfs::daily_file_cleanup: fully qualify path to 'find' [puppet] - 10https://gerrit.wikimedia.org/r/1307823 (https://phabricator.wikimedia.org/T429578) [16:35:56] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host cirrussearch1070.eqiad.wmnet with OS trixie [16:36:02] (03CR) 10CI reject: [V:04-1] profile::wmcs::nfs::daily_file_cleanup: fully qualify path to 'find' [puppet] - 10https://gerrit.wikimedia.org/r/1307823 (https://phabricator.wikimedia.org/T429578) (owner: 10Andrew Bogott) [16:37:57] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1096.eqiad.wmnet with OS trixie [16:38:50] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host cirrussearch1071.eqiad.wmnet with OS trixie [16:39:00] (03PS2) 10Andrew Bogott: profile::wmcs::nfs::daily_file_cleanup: fully qualify path to 'find' [puppet] - 10https://gerrit.wikimedia.org/r/1307823 (https://phabricator.wikimedia.org/T429578) [16:43:03] (03CR) 10Andrew Bogott: [C:03+2] profile::wmcs::nfs::daily_file_cleanup: fully qualify path to 'find' [puppet] - 10https://gerrit.wikimedia.org/r/1307823 (https://phabricator.wikimedia.org/T429578) (owner: 10Andrew Bogott) [16:49:42] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1084.eqiad.wmnet with reason: host reimage [16:51:29] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1070.eqiad.wmnet with reason: host reimage [16:51:39] (03PS1) 10BCornwall: cache_misc: apply traffic classification [puppet] - 10https://gerrit.wikimedia.org/r/1307824 [16:52:00] (03CR) 10CI reject: [V:04-1] cache_misc: apply traffic classification [puppet] - 10https://gerrit.wikimedia.org/r/1307824 (owner: 10BCornwall) [16:53:27] (03CR) 10David Caro: "> The CSV content has PII. Will it be OK to store this on Prometheus?" [puppet] - 10https://gerrit.wikimedia.org/r/1294864 (https://phabricator.wikimedia.org/T423549) (owner: 10Komla Sapaty) [16:53:39] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1084.eqiad.wmnet with reason: host reimage [16:54:06] (03PS2) 10BCornwall: cache_misc: apply traffic classification [puppet] - 10https://gerrit.wikimedia.org/r/1307824 [16:54:24] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1071.eqiad.wmnet with reason: host reimage [16:54:36] (03PS3) 10BCornwall: cache_misc: apply traffic classification [puppet] - 10https://gerrit.wikimedia.org/r/1307824 [16:55:13] (03CR) 10CI reject: [V:04-1] cache_misc: apply traffic classification [puppet] - 10https://gerrit.wikimedia.org/r/1307824 (owner: 10BCornwall) [16:57:50] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1070.eqiad.wmnet with reason: host reimage [16:58:29] (03CR) 10BCornwall: [C:03+1] wikimedia.org: move dumps-nfs to dumps-lb [dns] - 10https://gerrit.wikimedia.org/r/1307339 (https://phabricator.wikimedia.org/T411248) (owner: 10Filippo Giunchedi) [16:59:34] (03CR) 10BCornwall: [C:03+2] Revert "varnish: Add CSP report-only header value" [puppet] - 10https://gerrit.wikimedia.org/r/1307474 (owner: 10Ssingh) [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260706T1700) [17:00:05] ryankemper: Time to do the Wikidata Query Service weekly deploy deploy. Don't look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260706T1700). [17:01:57] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1071.eqiad.wmnet with reason: host reimage [17:01:58] 10ops-eqiad, 06Data-Persistence, 06DBA, 06DC-Ops, and 3 others: es7 primary (es1039.eqiad.wmnet) crashed - https://phabricator.wikimedia.org/T430764#12091237 (10Marostegui) a:05VRiley-WMF→03Marostegui [17:02:54] (03CR) 10Majavah: [C:04-2] "this would break anycast-healthchecker checks" [puppet] - 10https://gerrit.wikimedia.org/r/1307754 (owner: 10Majavah) [17:06:02] (03PS1) 10BCornwall: Revert^2 "varnish: Add CSP report-only header value" [puppet] - 10https://gerrit.wikimedia.org/r/1307826 [17:12:00] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware: Decommission backup1004, backup1005, backup1006 & backup1007 - https://phabricator.wikimedia.org/T431097#12091265 (10VRiley-WMF) a:05jcrespo→03VRiley-WMF [17:12:41] (03CR) 10BCornwall: ip_reputation_vendors: add basic safeguard to spur downloader (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1307752 (owner: 10Fabfur) [17:13:40] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware: Decommission backup1004, backup1005, backup1006 & backup1007 - https://phabricator.wikimedia.org/T431097#12091270 (10VRiley-WMF) 05Open→03Resolved [17:16:24] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1084.eqiad.wmnet with OS trixie [17:18:53] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1070.eqiad.wmnet with OS trixie [17:23:34] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1071.eqiad.wmnet with OS trixie [17:28:25] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:33:35] (03CR) 10Ssingh: [C:04-1] "Patch looks good but -1 so we wait for @sbassett@wikimedia.org's approval" [puppet] - 10https://gerrit.wikimedia.org/r/1307826 (owner: 10BCornwall) [17:39:32] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host cirrussearch1072.eqiad.wmnet with OS trixie [17:42:38] bking@cumin2003 reimage (PID 3545276) is awaiting input [17:42:58] RECOVERY - ElasticSearch unassigned shard check - 9443 on search.svc.eqiad.wmnet is OK: OK - All good https://wikitech.wikimedia.org/wiki/Search%23Administration [17:44:28] (03PS6) 10Jforrester: tables-catalog: Document the new x1 wikifunctions_usage* tables [puppet] - 10https://gerrit.wikimedia.org/r/1305102 (https://phabricator.wikimedia.org/T428667) [17:44:31] (03CR) 10Ladsgroup: [C:03+2] tables-catalog: Document the new x1 wikifunctions_usage* tables [puppet] - 10https://gerrit.wikimedia.org/r/1305102 (https://phabricator.wikimedia.org/T428667) (owner: 10Jforrester) [17:44:34] (03CR) 10Ladsgroup: [V:03+2 C:03+2] tables-catalog: Document the new x1 wikifunctions_usage* tables [puppet] - 10https://gerrit.wikimedia.org/r/1305102 (https://phabricator.wikimedia.org/T428667) (owner: 10Jforrester) [17:55:12] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1072.eqiad.wmnet with reason: host reimage [17:57:25] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host cirrussearch1112.eqiad.wmnet with OS trixie [17:59:23] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1072.eqiad.wmnet with reason: host reimage [18:06:30] (03CR) 10Dzahn: [C:03+2] cache: turn off caching for zuul.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1306353 (https://phabricator.wikimedia.org/T430462) (owner: 10Dzahn) [18:10:07] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1112.eqiad.wmnet with reason: host reimage [18:14:06] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1112.eqiad.wmnet with reason: host reimage [18:16:38] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, July 06 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307211 (https://phabricator.wikimedia.org/T390858) (owner: 10Ebernhardson) [18:21:05] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1072.eqiad.wmnet with OS trixie [18:23:26] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host cirrussearch1114.eqiad.wmnet with OS trixie [18:25:10] FIRING: BFDdown: BFD session down between cr1-codfw and fe80::ee38:73ff:fe75:38c4 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [18:30:10] RESOLVED: BFDdown: BFD session down between cr1-codfw and fe80::ee38:73ff:fe75:38c4 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [18:31:37] (03CR) 10Komla Sapaty: "Thank you, David!" [puppet] - 10https://gerrit.wikimedia.org/r/1294864 (https://phabricator.wikimedia.org/T423549) (owner: 10Komla Sapaty) [18:32:28] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1112.eqiad.wmnet with OS trixie [18:34:29] (03PS1) 10Danielyepezgarces: extwiki: Rename wgSitename to Güiquipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307836 (https://phabricator.wikimedia.org/T431334) [18:35:55] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1114.eqiad.wmnet with reason: host reimage [18:39:32] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1114.eqiad.wmnet with reason: host reimage [18:43:09] (03CR) 10Komla Sapaty: profile::toolforge::bastion: add SSH login activity export timer (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1294864 (https://phabricator.wikimedia.org/T423549) (owner: 10Komla Sapaty) [18:47:23] (03CR) 10Kosta Harlan: [C:03+1] "I scheduled this for the puppet window tomorrow https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260707T1600" [puppet] - 10https://gerrit.wikimedia.org/r/1307100 (https://phabricator.wikimedia.org/T386456) (owner: 10Dreamy Jazz) [18:53:06] (03CR) 10Dzahn: [C:03+2] zuul: move vim: modeline to top of apache conf [puppet] - 10https://gerrit.wikimedia.org/r/1307447 (owner: 10Hashar) [18:56:40] (03CR) 10Pppery: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307836 (https://phabricator.wikimedia.org/T431334) (owner: 10Danielyepezgarces) [18:58:48] (03CR) 10Dzahn: [C:03+1] "can you bring this up in infra-foundations?" [puppet] - 10https://gerrit.wikimedia.org/r/1296495 (https://phabricator.wikimedia.org/T420184) (owner: 10Arnaudb) [18:59:06] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1114.eqiad.wmnet with OS trixie [18:59:44] (03CR) 10Dzahn: [C:03+1] "deploy tomorrow?" [puppet] - 10https://gerrit.wikimedia.org/r/1305044 (https://phabricator.wikimedia.org/T429367) (owner: 10Aklapper) [19:03:06] 06SRE, 10LDAP-Access-Requests: Grant Access to fr-tech groups for laurabarluzzi - https://phabricator.wikimedia.org/T431338 (10Laurabarluzzi) 03NEW [19:07:02] (03PS2) 10Hashar: zuul: allow encoded slashes [puppet] - 10https://gerrit.wikimedia.org/r/1307442 (https://phabricator.wikimedia.org/T431003) [19:09:49] (03PS3) 10Dzahn: zuul: allow encoded slashes [puppet] - 10https://gerrit.wikimedia.org/r/1307442 (https://phabricator.wikimedia.org/T431003) (owner: 10Hashar) [19:10:11] (03CR) 10Dzahn: [C:03+2] zuul: allow encoded slashes [puppet] - 10https://gerrit.wikimedia.org/r/1307442 (https://phabricator.wikimedia.org/T431003) (owner: 10Hashar) [19:10:13] (03CR) 10Dzahn: [V:03+2 C:03+2] zuul: allow encoded slashes [puppet] - 10https://gerrit.wikimedia.org/r/1307442 (https://phabricator.wikimedia.org/T431003) (owner: 10Hashar) [19:10:56] (03CR) 10Dzahn: "for now we have turned off caching - so let's discuss this later I guess" [puppet] - 10https://gerrit.wikimedia.org/r/1307459 (https://phabricator.wikimedia.org/T430462) (owner: 10Hashar) [19:16:50] 06SRE, 10LDAP-Access-Requests: Grant Access to fr-tech groups for laurabarluzzi - https://phabricator.wikimedia.org/T431338#12091752 (10Dzahn) Hi, This requests asks for membership in LDAP groups to match groups @Damilare is in. Searching for existing tickets from the past where Damilare was added we can jus... [19:17:36] (03PS1) 10Cathal Mooney: HE Transport: enable ospf on esams<->eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1307839 (https://phabricator.wikimedia.org/T424839) [19:17:53] 06SRE, 10LDAP-Access-Requests: Grant Access to fr-tech groups for laurabarluzzi - https://phabricator.wikimedia.org/T431338#12091754 (10Dzahn) Please see the bottom of https://wikitech.wikimedia.org/wiki/SRE/LDAP/Groups/Request_access#Using_the_Wikimedia_Identity_Management_System on how to handle this in the... [19:21:08] (03CR) 10Ayounsi: [C:03+1] HE Transport: enable ospf on esams<->eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1307839 (https://phabricator.wikimedia.org/T424839) (owner: 10Cathal Mooney) [19:22:13] (03CR) 10Cathal Mooney: [C:03+2] HE Transport: enable ospf on esams<->eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1307839 (https://phabricator.wikimedia.org/T424839) (owner: 10Cathal Mooney) [19:23:43] (03Merged) 10jenkins-bot: HE Transport: enable ospf on esams<->eqiad link [homer/public] - 10https://gerrit.wikimedia.org/r/1307839 (https://phabricator.wikimedia.org/T424839) (owner: 10Cathal Mooney) [19:25:07] (03PS1) 10Brouberol: dse-k8s-eqiad: increase memory quota for mediawiki-dumps-legacy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307840 [19:25:17] (03PS1) 10Dzahn: varnish: remove ancient Noise rule from text-frontend VCL [puppet] - 10https://gerrit.wikimedia.org/r/1215329 [19:26:04] (03CR) 10Dzahn: [V:03+1 C:03+1] "alright! I managed to check for all the keywords in Turnilo now, for last 7 days, on webrequest sampled. None of them had any hits/results" [puppet] - 10https://gerrit.wikimedia.org/r/1215329 (owner: 10Dzahn) [19:26:26] (03PS2) 10Brouberol: dse-k8s-eqiad: increase memory quota for mediawiki-dumps-legacy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307840 [19:33:25] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, July 06 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307836 (https://phabricator.wikimedia.org/T431334) (owner: 10Danielyepezgarces) [19:34:55] 06SRE, 10LDAP-Access-Requests: Grant Access to fr-tech groups for laurabarluzzi - https://phabricator.wikimedia.org/T431338#12091824 (10Laurabarluzzi) @Dzahn thanks, I already have wmf access from that flow granted. I was pointed here from my colleagues since at some point I tried to create a phab task clickin... [19:35:47] (03CR) 10Brouberol: [C:03+2] dse-k8s-eqiad: increase memory quota for mediawiki-dumps-legacy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1307840 (owner: 10Brouberol) [19:38:03] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [19:38:48] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [19:46:30] 06SRE, 10LDAP-Access-Requests: Grant Access to fr-tech groups for laurabarluzzi - https://phabricator.wikimedia.org/T431338#12091890 (10Dzahn) It seems like you were trying to create a request for SSH access to fr-tech servers from a Phabricator template. But for some reason you are not allowed to use that te... [19:48:43] 06SRE, 10LDAP-Access-Requests: Grant Access to fr-tech groups for laurabarluzzi - https://phabricator.wikimedia.org/T431338#12091908 (10Dzahn) So to summarize: - seems like fr-tech shell access is handled internally by fr-tech, not by SRE (is this correct?) - LDAP group requests are a different thing and sep... [19:51:26] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host cirrussearch1115.eqiad.wmnet with OS trixie [19:52:23] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host cirrussearch1077.eqiad.wmnet with OS trixie [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: #bothumor My software never has bugs. It just develops random features. Rise for UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260706T2000). [20:00:05] sfaci, Krinkle, ebernhardson, and dyepezg: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:45] o/ [20:02:51] (03CR) 10Kosta Harlan: maintain-views: Make change_tag_def and change_tag partially public (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1307100 (https://phabricator.wikimedia.org/T386456) (owner: 10Dreamy Jazz) [20:02:57] o/ [20:03:33] sfaci: are you okay self-servicing? (If not, we can wait for a deployer. I'm happy to do mine on my own.) [20:04:02] I cannot deploy [20:04:02] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1115.eqiad.wmnet with reason: host reimage [20:04:13] I was waiting for a deployer [20:04:17] sfaci: OK, let me look at yours [20:04:39] sfaci: you have a way to test it via WikimediaDebug? [20:04:53] I can see the change itself, yes [20:05:13] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1303490 (owner: 10Pushpaktiwari) [20:05:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307812 (https://phabricator.wikimedia.org/T414338) (owner: 10Krinkle) [20:05:50] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host cirrussearch1079.eqiad.wmnet with OS trixie [20:05:57] Thank you for deploying it! [20:06:26] (03Merged) 10jenkins-bot: T429269: Send logged-in experiment events to ins-502b [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1303490 (owner: 10Pushpaktiwari) [20:06:33] (03Merged) 10jenkins-bot: Re-enable wgTrackMediaRequestProvenance on pilot wikis (group1) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1307812 (https://phabricator.wikimedia.org/T414338) (owner: 10Krinkle) [20:06:54] !log krinkle@deploy1003 Started scap sync-world: Backport for [[gerrit:1303490|T429269: Send logged-in experiment events to ins-502b]], [[gerrit:1307812|Re-enable wgTrackMediaRequestProvenance on pilot wikis (group1) (T414338)]] [20:06:58] T429269: Send logged-in experiment events to ins-502b - https://phabricator.wikimedia.org/T429269 [20:06:59] T414338: FY25-26 WE5.4.12: Identify the provenance of image requests - https://phabricator.wikimedia.org/T414338 [20:07:41] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1077.eqiad.wmnet with reason: host reimage [20:07:45] FIRING: CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-search is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow [20:08:49] !log krinkle@deploy1003 krinkle, pushpaktiwari: Backport for [[gerrit:1303490|T429269: Send logged-in experiment events to ins-502b]], [[gerrit:1307812|Re-enable wgTrackMediaRequestProvenance on pilot wikis (group1) (T414338)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:09:21] FIRING: [6x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [20:09:34] sfaci: staged, ready for testing [20:09:49] I'm seeing already the change [20:09:50] It's done [20:09:59] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1115.eqiad.wmnet with reason: host reimage [20:10:47] !log krinkle@deploy1003 krinkle, pushpaktiwari: Continuing with deployment [20:10:57] mine looks fine as well [20:11:15] stand by for prod verification once deployment is done [20:11:29] 👍 [20:13:10] (03PS1) 10Dzahn: ci: disable jenkins on role::ci machines [puppet] - 10https://gerrit.wikimedia.org/r/1307850 (https://phabricator.wikimedia.org/T418109) [20:14:12] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1077.eqiad.wmnet with reason: host reimage [20:15:08] !log krinkle@deploy1003 Finished scap sync-world: Backport for [[gerrit:1303490|T429269: Send logged-in experiment events to ins-502b]], [[gerrit:1307812|Re-enable wgTrackMediaRequestProvenance on pilot wikis (group1) (T414338)]] (duration: 08m 14s) [20:15:12] T429269: Send logged-in experiment events to ins-502b - https://phabricator.wikimedia.org/T429269 [20:15:13] T414338: FY25-26 WE5.4.12: Identify the provenance of image requests - https://phabricator.wikimedia.org/T414338 [20:16:45] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [20:16:45] CirrusSearch consumer-search@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [20:16:56] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [20:17:02] CirrusSearch consumer-search@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [20:17:20] (03PS1) 10Dzahn: Revert^2 "integration: switch integration-agent-docker VMs to Java 21" [puppet] - 10https://gerrit.wikimedia.org/r/1307851 [20:20:14] sfaci: ready for prod verificiation [20:20:42] Looks good [20:20:47] OK :) [20:20:54] Thank you very much! [20:21:30] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1079.eqiad.wmnet with reason: host reimage [20:24:21] FIRING: [12x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [20:25:31] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1079.eqiad.wmnet with reason: host reimage [20:25:43] 06SRE, 10LDAP-Access-Requests: Grant Access to fr-tech groups for laurabarluzzi - https://phabricator.wikimedia.org/T431338#12092085 (10Dwisehaupt) I can clear up some of the communication as I think there is some combination of things going on. Laura does have fr-tech ssh access and that has already been set... [20:26:44] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1115.eqiad.wmnet with OS trixie [20:29:36] 06SRE, 10LDAP-Access-Requests: Grant Access to fr-tech groups for laurabarluzzi - https://phabricator.wikimedia.org/T431338#12092104 (10Dzahn) Gotcha! Thanks for the clarifications. Since this is an SSH access request for SRE then let's please use the template and tags for that. The correct template is: ht... [20:30:05] 06SRE, 10SRE-Access-Requests: Grant Access to fr-tech groups for laurabarluzzi - https://phabricator.wikimedia.org/T431338#12092105 (10Dzahn) [20:31:26] 06SRE, 10SRE-Access-Requests: Grant Access to fr-tech groups for laurabarluzzi - https://phabricator.wikimedia.org/T431338#12092116 (10Dzahn) > missing inclusion in a set of groups like WMF-NDA, Trusted-Contributors, acl*security, etc. Is there a general onboarding ticket for making these things happen? Ther... [20:33:31] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1077.eqiad.wmnet with OS trixie [20:34:22] (03PS2) 10Scott French: hieradata: Mark configcluster hosts as manual depool [puppet] - 10https://gerrit.wikimedia.org/r/1307852 (https://phabricator.wikimedia.org/T430930) [20:36:45] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [20:36:50] CirrusSearch consumer-search@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [20:37:02] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [20:37:08] CirrusSearch consumer-search@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [20:39:21] FIRING: [18x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [20:43:39] FIRING: [2x] CirrusSearchNodeIndexingNotIncreasing: Elasticsearch instance cirrussearch1072-production-search-eqiad is not indexing - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Indexing_hung_and_not_making_progress - https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?orgId=1&from=now-3d&to=now&viewPanel=57 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchNodeIndexingNotIncreasing [20:44:32] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1079.eqiad.wmnet with OS trixie [20:49:21] FIRING: [18x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [20:53:39] FIRING: [2x] CirrusSearchNodeIndexingNotIncreasing: Elasticsearch instance cirrussearch1072-production-search-eqiad is not indexing - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Indexing_hung_and_not_making_progress - https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?orgId=1&from=now-3d&to=now&viewPanel=57 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchNodeIndexingNotIncreasing [20:54:21] FIRING: [24x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [20:54:45] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [20:54:50] CirrusSearch consumer-search@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [20:55:02] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [20:55:08] CirrusSearch consumer-search@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [21:03:33] is anyone deploying right now? [21:03:39] RESOLVED: [2x] CirrusSearchNodeIndexingNotIncreasing: Elasticsearch instance cirrussearch1072-production-search-eqiad is not indexing - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Indexing_hung_and_not_making_progress - https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?orgId=1&from=now-3d&to=now&viewPanel=57 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchNodeIndexingNotIncreasing [21:03:43] I would like to deploy a security patch [21:04:21] FIRING: [30x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [21:11:15] preparing to run scap [21:18:07] !log Deployed security fix for T428006 [21:18:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:19:21] FIRING: [30x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [21:28:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:29:40] (03PS4) 10Dreamy Jazz: maintain-views: Make change_tag_def and change_tag partially public [puppet] - 10https://gerrit.wikimedia.org/r/1307100 (https://phabricator.wikimedia.org/T386456) [21:29:45] (03CR) 10Dreamy Jazz: maintain-views: Make change_tag_def and change_tag partially public (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1307100 (https://phabricator.wikimedia.org/T386456) (owner: 10Dreamy Jazz) [21:30:49] (03PS1) 10Andrew Bogott: scan-and-shrink.py: add a very rough scoring system for user dirs [puppet] - 10https://gerrit.wikimedia.org/r/1307855 (https://phabricator.wikimedia.org/T431340) [21:33:01] (03CR) 10CI reject: [V:04-1] scan-and-shrink.py: add a very rough scoring system for user dirs [puppet] - 10https://gerrit.wikimedia.org/r/1307855 (https://phabricator.wikimedia.org/T431340) (owner: 10Andrew Bogott) [21:34:21] FIRING: [30x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [21:35:25] 06SRE, 10SRE-Access-Requests: Grant Access to fr-tech groups for laurabarluzzi - https://phabricator.wikimedia.org/T431338#12092312 (10Aklapper) Re: Access to that Phabricator form: Sorry for that confusion (the error message is not great). This can be solved in two ways: * Either linking to form 1 which is su... [21:39:45] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [21:39:45] CirrusSearch consumer-search@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [21:40:02] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [21:40:08] CirrusSearch consumer-search@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [21:49:21] FIRING: [28x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [21:59:21] FIRING: [30x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [22:04:21] FIRING: [28x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [22:14:21] FIRING: [30x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [22:14:45] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [22:14:50] CirrusSearch consumer-search@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [22:14:53] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [22:14:59] CirrusSearch consumer-search@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [22:18:39] FIRING: CirrusSearchNodeIndexingNotIncreasing: Elasticsearch instance cirrussearch1115-production-search-eqiad is not indexing - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Indexing_hung_and_not_making_progress - https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?orgId=1&from=now-3d&to=now&viewPanel=57 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchNodeIndexingNotIncreasing [22:21:43] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!!" [puppet] - 10https://gerrit.wikimedia.org/r/1306969 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [22:22:48] !log bking@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cirrussearch[1079,1115].eqiad.wmnet with reason: reimage on hold until restore completes [22:24:21] FIRING: [30x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [22:24:53] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!!" [puppet] - 10https://gerrit.wikimedia.org/r/1306661 (https://phabricator.wikimedia.org/T407141) (owner: 10Hnowlan) [22:28:39] FIRING: CirrusSearchNodeIndexingNotIncreasing: Elasticsearch instance cirrussearch1114-production-search-eqiad is not indexing - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Indexing_hung_and_not_making_progress - https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?orgId=1&from=now-3d&to=now&viewPanel=57 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchNodeIndexingNotIncreasing [22:29:36] !log bking@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cirrussearch1114.eqiad.wmnet with reason: reimage on hold until restore completes [22:33:39] RESOLVED: CirrusSearchNodeIndexingNotIncreasing: Elasticsearch instance cirrussearch1072-production-search-eqiad is not indexing - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Indexing_hung_and_not_making_progress - https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?orgId=1&from=now-3d&to=now&viewPanel=57 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchNodeIndexingNotIncreasing [22:34:21] FIRING: [30x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [22:39:21] FIRING: [30x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [22:40:10] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, I just a left a comment." [puppet] - 10https://gerrit.wikimedia.org/r/1307087 (https://phabricator.wikimedia.org/T430149) (owner: 10Hnowlan) [22:41:15] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thanks!!" [puppet] - 10https://gerrit.wikimedia.org/r/1306001 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [22:44:21] FIRING: [26x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [22:47:45] RESOLVED: CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-search is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow [22:49:21] FIRING: [33x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [22:49:44] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [22:49:45] CirrusSearch consumer-search@eqiad is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [22:49:53] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [22:49:59] CirrusSearch consumer-search@eqiad is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=eqiad&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [22:50:46] !log bking@cumin2003 START - Cookbook sre.hosts.reimage for host cirrussearch1078.eqiad.wmnet with OS trixie [22:54:21] FIRING: [33x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [23:00:04] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260706T2300) [23:04:21] FIRING: [27x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [23:06:21] !log bking@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on cirrussearch1078.eqiad.wmnet with reason: host reimage [23:07:48] o/ [23:07:53] doing a few deploys now [23:09:14] (03PS1) 10Jdlrobson: Drop orphaned configuration for Vector skin rollout [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1306982 (https://phabricator.wikimedia.org/T358273) [23:09:23] (03CR) 10Jdlrobson: Remove wgMinervaEnableSiteNotice config flag [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1305921 (https://phabricator.wikimedia.org/T417638) (owner: 10Bernard Wang) [23:10:14] is phab down >.> [23:10:21] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cirrussearch1078.eqiad.wmnet with reason: host reimage [23:10:39] oop back up I think but isup.me was reporting it as down in my defense) [23:10:50] (03PS1) 10Jdlrobson: Drop unused VectorNightMode config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1306453 (https://phabricator.wikimedia.org/T393977) [23:11:56] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1306453 (https://phabricator.wikimedia.org/T393977) (owner: 10Jdlrobson) [23:11:56] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1305921 (https://phabricator.wikimedia.org/T417638) (owner: 10Bernard Wang) [23:11:57] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1306982 (https://phabricator.wikimedia.org/T358273) (owner: 10Jdlrobson) [23:11:57] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [skins/Vector] (wmf/1.47.0-wmf.9) - 10https://gerrit.wikimedia.org/r/1306983 (https://phabricator.wikimedia.org/T358273) (owner: 10Jdlrobson) [23:12:36] (03CR) 10Cwhite: [C:03+1] restbase: move nrpe check to prom blackbox check [puppet] - 10https://gerrit.wikimedia.org/r/1306661 (https://phabricator.wikimedia.org/T407141) (owner: 10Hnowlan) [23:12:58] (03Merged) 10jenkins-bot: Drop unused VectorNightMode config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1306453 (https://phabricator.wikimedia.org/T393977) (owner: 10Jdlrobson) [23:13:06] (03Merged) 10jenkins-bot: Drop orphaned configuration for Vector skin rollout [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1306982 (https://phabricator.wikimedia.org/T358273) (owner: 10Jdlrobson) [23:13:10] (03Merged) 10jenkins-bot: Remove wgMinervaEnableSiteNotice config flag [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1305921 (https://phabricator.wikimedia.org/T417638) (owner: 10Bernard Wang) [23:14:06] (03CR) 10Scott French: [C:03+1] "Interesting find!" [puppet] - 10https://gerrit.wikimedia.org/r/1306905 (https://phabricator.wikimedia.org/T428909) (owner: 10Clément Goubert) [23:14:21] FIRING: [21x] SLOBudgetBurn: Search update lag is below 95% target in eqiad - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [23:18:52] (03PS4) 10Andrew Bogott: scan-and-shrink.py: add a very rough scoring system for user dirs [puppet] - 10https://gerrit.wikimedia.org/r/1307855 (https://phabricator.wikimedia.org/T431340) [23:20:20] (03Merged) 10jenkins-bot: Remove unused user skin preference config [skins/Vector] (wmf/1.47.0-wmf.9) - 10https://gerrit.wikimedia.org/r/1306983 (https://phabricator.wikimedia.org/T358273) (owner: 10Jdlrobson) [23:20:53] !log jdlrobson@deploy1003 Started scap sync-world: Backport for [[gerrit:1306983|Remove unused user skin preference config (T358273)]], [[gerrit:1306982|Drop orphaned configuration for Vector skin rollout (T358273)]], [[gerrit:1305921|Remove wgMinervaEnableSiteNotice config flag (T417638)]], [[gerrit:1306453|Drop unused VectorNightMode config (T393977)]] [23:21:02] T358273: Remove Vector's unused broken configuration flags for user skin preference - https://phabricator.wikimedia.org/T358273 [23:21:02] T417638: Remove $wgMinervaEnableSiteNotice config flag from MinervaNeue - https://phabricator.wikimedia.org/T417638 [23:21:03] T393977: Drop drop drop drop night mode exclusions - https://phabricator.wikimedia.org/T393977 [23:22:48] !log jdlrobson@deploy1003 jdlrobson, bwang: Backport for [[gerrit:1306983|Remove unused user skin preference config (T358273)]], [[gerrit:1306982|Drop orphaned configuration for Vector skin rollout (T358273)]], [[gerrit:1305921|Remove wgMinervaEnableSiteNotice config flag (T417638)]], [[gerrit:1306453|Drop unused VectorNightMode config (T393977)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug) [23:22:48] . Changes can now be verified there. [23:26:11] !log jdlrobson@deploy1003 jdlrobson, bwang: Continuing with deployment [23:28:43] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cirrussearch1078.eqiad.wmnet with OS trixie [23:30:32] !log jdlrobson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1306983|Remove unused user skin preference config (T358273)]], [[gerrit:1306982|Drop orphaned configuration for Vector skin rollout (T358273)]], [[gerrit:1305921|Remove wgMinervaEnableSiteNotice config flag (T417638)]], [[gerrit:1306453|Drop unused VectorNightMode config (T393977)]] (duration: 09m 39s) [23:30:39] T358273: Remove Vector's unused broken configuration flags for user skin preference - https://phabricator.wikimedia.org/T358273 [23:30:39] T417638: Remove $wgMinervaEnableSiteNotice config flag from MinervaNeue - https://phabricator.wikimedia.org/T417638 [23:30:40] T393977: Drop drop drop drop night mode exclusions - https://phabricator.wikimedia.org/T393977 [23:32:35] all done here. [23:41:58] (03PS7) 10Cwhite: logstash: add parameters needed for security plugin [puppet] - 10https://gerrit.wikimedia.org/r/1305769 (https://phabricator.wikimedia.org/T350516) [23:42:12] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1307870 [23:42:12] (03CR) 10Cwhite: logstash: add parameters needed for security plugin (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1305769 (https://phabricator.wikimedia.org/T350516) (owner: 10Cwhite) [23:42:12] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1307870 (owner: 10TrainBranchBot) [23:46:38] 10ops-codfw, 06SRE, 06Data-Persistence, 06DC-Ops: FY2526 Q3:rack/setup/install restbase2039 - https://phabricator.wikimedia.org/T416538#12092636 (10Eevans) Oh, will `partman/custom/cassandrahosts-3ssd-jbod.cfg` not work for this host? [23:47:40] 10ops-codfw, 06SRE, 06Data-Persistence, 06DC-Ops: FY2526 Q3:rack/setup/install restbase2039 - https://phabricator.wikimedia.org/T416538#12092638 (10Eevans) [23:49:46] (03CR) 10Eevans: [C:03+1] cassandra: remove obsolete expiry check [puppet] - 10https://gerrit.wikimedia.org/r/1307104 (https://phabricator.wikimedia.org/T407117) (owner: 10Hnowlan) [23:50:11] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1307870 (owner: 10TrainBranchBot) [23:56:41] (03PS1) 10Jasmine: sre.k8s.renumber-node: Increase timeout on deploy hosts puppet run to 600, from default 300 [cookbooks] - 10https://gerrit.wikimedia.org/r/1307872 (https://phabricator.wikimedia.org/T430226)