[00:01:45] FIRING: WidespreadPuppetFailure: Puppet has failed in esams - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [00:06:45] FIRING: [2x] WidespreadPuppetFailure: Puppet has failed in esams - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [00:11:45] FIRING: [3x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [00:31:58] 06SRE, 10SRE-swift-storage, 10MediaWiki-extensions-Score, 06Reader Experience Team: Add cache key information to metadata json - https://phabricator.wikimedia.org/T257093#12193826 (10Krinkle) [01:07:53] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [01:12:26] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1322255 [01:12:26] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1322255 (owner: 10TrainBranchBot) [01:13:49] FIRING: PuppetFailure: Puppet has failed on cumin1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:13:49] FIRING: PuppetFailure: Puppet has failed on netflow1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:18:48] FIRING: [2x] PuppetFailure: Puppet has failed on netflow1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:23:48] FIRING: [3x] PuppetFailure: Puppet has failed on netflow1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:24:32] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1322255 (owner: 10TrainBranchBot) [01:28:49] FIRING: [5x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:33:49] FIRING: [6x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:54:43] FIRING: KubernetesCalicoDown: ml-serve1015.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s-mlserve&var-instance=ml-serve1015.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [02:00:49] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:04:43] FIRING: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node ml-serve1015 has a BGP session which is not in the 'established' state. [02:07:46] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 06m 56s) [02:07:53] FIRING: [4x] JobUnavailable: Reduced availability for job cfssl in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:09:43] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:14:43] FIRING: [4x] JobUnavailable: Reduced availability for job cfssl in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:36:56] !log Extended volume on prometheus2007 tor the disk space alert as per https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_host_running_out_of_space [02:37:55] !log Extended volume on prometheus2007 tor the disk space alert as per https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_host_running_out_of_space [02:37:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:42:32] !log Extended volume on prometheus2008 for the disk space alert as per https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_host_running_out_of_space [02:42:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:32:35] RECOVERY - Host pki1002 is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms [03:33:24] FIRING: [44x] ProbeDown: Service pki1002:443 has failed probes (http_PKI_aux_front_proxy_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#pki1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:34:43] FIRING: [2x] JobUnavailable: Reduced availability for job cfssl in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [03:37:25] FIRING: [24x] SystemdUnitFailed: cfssl-ocsprefresh-Wikimedia_Internal_Root_CA.service on pki1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:38:24] RESOLVED: [44x] ProbeDown: Service pki1002:443 has failed probes (http_PKI_aux_front_proxy_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#pki1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:38:48] FIRING: [6x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:41:45] FIRING: [3x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [03:46:45] FIRING: [3x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [03:48:39] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: pki1002 became unresponsive causing several hosts to alert on failed puppet runs. - https://phabricator.wikimedia.org/T434268 (10andrea.denisse) 03NEW [03:52:55] RESOLVED: [24x] SystemdUnitFailed: cfssl-ocsprefresh-Wikimedia_Internal_Root_CA.service on pki1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:53:49] FIRING: [5x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:53:53] RESOLVED: PuppetFailure: Puppet has failed on cumin1003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [03:58:48] FIRING: [4x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [04:01:45] RESOLVED: [2x] WidespreadPuppetFailure: Puppet has failed in esams - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [04:03:48] RESOLVED: [4x] PuppetFailure: Puppet has failed on netflow1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [05:07:54] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [05:21:40] (03PS1) 10Samwilson: Thumbor: Add the wikimedia_thumbor.filter.format filter [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322507 (https://phabricator.wikimedia.org/T430528) [05:42:52] (03PS1) 10Marostegui: mariadb: Decommission db1178 [puppet] - 10https://gerrit.wikimedia.org/r/1322520 (https://phabricator.wikimedia.org/T433471) [05:45:56] !log marostegui@cumin1003 START - Cookbook sre.mysql.decommission [05:46:25] !log marostegui@cumin1003 Removing db1178 from zarcillo T433471 [05:46:28] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) [05:46:29] T433471: decommission db1178.eqiad.wmnet - https://phabricator.wikimedia.org/T433471 [05:48:35] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 1 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [05:49:40] !log marostegui@cumin1003 START - Cookbook sre.hosts.decommission for hosts db1178.eqiad.wmnet [05:52:32] (03CR) 10Marostegui: "I think we should probably add a big note saying that this script actually doesn't run the decommission itself but the meta parts for data" [cookbooks] - 10https://gerrit.wikimedia.org/r/1291952 (https://phabricator.wikimedia.org/T426613) (owner: 10Federico Ceratto) [05:54:43] FIRING: KubernetesCalicoDown: ml-serve1015.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s-mlserve&var-instance=ml-serve1015.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [05:55:36] !log marostegui@cumin1003 START - Cookbook sre.dns.netbox [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260807T0600) [06:01:40] marostegui@cumin1003 decommission (PID 1375948) is awaiting input [06:04:43] FIRING: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node ml-serve1015 has a BGP session which is not in the 'established' state. [06:06:38] !log marostegui@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1178.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" [06:08:37] !log marostegui@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1178.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" [06:08:37] !log marostegui@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [06:08:39] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1178.eqiad.wmnet [06:09:34] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware, 13Patch-For-Review: decommission db1178.eqiad.wmnet - https://phabricator.wikimedia.org/T433471#12194006 (10Marostegui) a:05Marostegui→03None [06:09:41] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware, 13Patch-For-Review: decommission db1178.eqiad.wmnet - https://phabricator.wikimedia.org/T433471#12194010 (10Marostegui) Ready for DC-Ops! [06:09:43] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:35:57] !log updated istio to 1.29.4 on wikikube eqiad - T427401 [06:36:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:36:02] T427401: Update istio to 1.29 - https://phabricator.wikimedia.org/T427401 [06:42:34] (03PS1) 10JMeybohm: istio/main: Make istio 1.29 the default for wikikube [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322553 (https://phabricator.wikimedia.org/T427401) [06:49:44] (03CR) 10Slyngshede: varnish: Configure text for thumbnails (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1306896 (https://phabricator.wikimedia.org/T427465) (owner: 10Slyngshede) [06:56:36] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Degraded RAID on an-worker1191 - https://phabricator.wikimedia.org/T431828#12194038 (10brouberol) Sorry, I have to admit Ben is really the right person to talk to here. However, he's going to be back on tuesday, and I will flag th... [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260807T0700) [07:04:49] (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1321889 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [07:12:34] (03PS4) 10JMeybohm: admin_ng: pin to be updated cluster components to current versions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311394 (https://phabricator.wikimedia.org/T427400) (owner: 10Jelto) [07:13:57] (03PS8) 10Jelto: Update calico-crds to calico v3.30.7 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306165 (https://phabricator.wikimedia.org/T427400) [07:25:54] (03PS1) 10Brouberol: idm: sort airflow rules to help with next review [puppet] - 10https://gerrit.wikimedia.org/r/1322570 (https://phabricator.wikimedia.org/T406746) [07:25:57] (03PS1) 10Brouberol: idm: add airflow-{analytics,analytics-product,platform-eng,wmde}-ops groups [puppet] - 10https://gerrit.wikimedia.org/r/1322571 (https://phabricator.wikimedia.org/T406746) [07:26:00] (03PS1) 10Brouberol: admin: ensure all airflow-*-ops groups are marked as registered in bitu [puppet] - 10https://gerrit.wikimedia.org/r/1322572 (https://phabricator.wikimedia.org/T406746) [07:33:20] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: Lower ephemeral-storage requests for revscoring services. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1321967 (https://phabricator.wikimedia.org/T431089) (owner: 10Bartosz Wójtowicz) [07:34:43] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:36:23] (03Merged) 10jenkins-bot: ml-services: Lower ephemeral-storage requests for revscoring services. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1321967 (https://phabricator.wikimedia.org/T431089) (owner: 10Bartosz Wójtowicz) [07:37:28] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [07:38:40] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [07:41:19] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [07:45:04] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [07:45:50] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [07:47:02] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [07:47:34] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [07:48:19] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [07:48:29] 10ops-codfw, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 06Machine-Learning-Team (Q1 FY2026-27): hw troubleshooting: memory errors on ml-serve2004.codfw.wmnet - https://phabricator.wikimedia.org/T433478#12194108 (10Gehel) [07:48:32] 10ops-codfw, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 06Machine-Learning-Team (Q1 FY2026-27): hw troubleshooting: Memory errors for ml-serve2002.codfw.wmnet - https://phabricator.wikimedia.org/T433476#12194110 (10Gehel) [07:48:37] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 06Machine-Learning-Team (Q1 FY2026-27): hw troubleshooting: failing disk in ml-serve1001.eqiad.wmnet - https://phabricator.wikimedia.org/T432105#12194112 (10Gehel) [07:49:08] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [07:50:31] (03CR) 10JMeybohm: Update calico to v3.30.7 (036 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306307 (https://phabricator.wikimedia.org/T427400) (owner: 10Jelto) [07:50:33] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [07:50:37] (03PS5) 10Jelto: Update calico to v3.30.7 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306307 (https://phabricator.wikimedia.org/T427400) [07:51:12] (03CR) 10Slyngshede: [C:03+1] idm: sort airflow rules to help with next review [puppet] - 10https://gerrit.wikimedia.org/r/1322570 (https://phabricator.wikimedia.org/T406746) (owner: 10Brouberol) [07:51:33] (03CR) 10JMeybohm: Update calico to v3.30.7 (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306307 (https://phabricator.wikimedia.org/T427400) (owner: 10Jelto) [07:51:46] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [07:52:20] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [07:53:05] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [07:53:07] (03CR) 10Blake: [C:03+1] admin_ng: pin to be updated cluster components to current versions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311394 (https://phabricator.wikimedia.org/T427400) (owner: 10Jelto) [07:53:13] (03CR) 10Slyngshede: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1320817 (https://phabricator.wikimedia.org/T273950) (owner: 10Majavah) [07:54:05] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [07:54:24] (03CR) 10Slyngshede: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1322572 (https://phabricator.wikimedia.org/T406746) (owner: 10Brouberol) [07:54:43] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [07:55:31] (03PS2) 10Brouberol: idm: sort airflow rules to help with next review [puppet] - 10https://gerrit.wikimedia.org/r/1322570 (https://phabricator.wikimedia.org/T421952) [07:55:32] (03CR) 10JMeybohm: "Acknowledged" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306307 (https://phabricator.wikimedia.org/T427400) (owner: 10Jelto) [07:55:33] (03PS2) 10Brouberol: idm: add airflow-{analytics,analytics-product,platform-eng,wmde}-ops groups [puppet] - 10https://gerrit.wikimedia.org/r/1322571 (https://phabricator.wikimedia.org/T421952) [07:55:35] (03PS6) 10JMeybohm: Update calico to v3.30.7 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306307 (https://phabricator.wikimedia.org/T427400) (owner: 10Jelto) [07:55:36] (03PS2) 10Brouberol: admin: ensure all airflow-*-ops groups are marked as registered in bitu [puppet] - 10https://gerrit.wikimedia.org/r/1322572 (https://phabricator.wikimedia.org/T421952) [07:57:34] (03CR) 10JMeybohm: [C:03+2] Update calico to v3.30.7 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306307 (https://phabricator.wikimedia.org/T427400) (owner: 10Jelto) [07:57:39] (03CR) 10JMeybohm: [C:03+2] Update calico-crds to calico v3.30.7 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306165 (https://phabricator.wikimedia.org/T427400) (owner: 10Jelto) [07:57:44] (03CR) 10JMeybohm: [C:03+2] admin_ng: pin to be updated cluster components to current versions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311394 (https://phabricator.wikimedia.org/T427400) (owner: 10Jelto) [08:00:08] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [08:00:12] !log klausman@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on ml-serve1015.eqiad.wmnet with reason: Downtime to get full picture of current BIOS settings beyond what Redfish shows [08:05:59] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [08:06:54] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [08:07:52] (03CR) 10JMeybohm: [C:03+2] Update to v3.30.7 [debs/calico] (v3.30) - 10https://gerrit.wikimedia.org/r/1305139 (https://phabricator.wikimedia.org/T427400) (owner: 10Jelto) [08:07:56] (03CR) 10JMeybohm: [V:03+2 C:03+2] Update to v3.30.7 [debs/calico] (v3.30) - 10https://gerrit.wikimedia.org/r/1305139 (https://phabricator.wikimedia.org/T427400) (owner: 10Jelto) [08:08:17] (03Merged) 10jenkins-bot: admin_ng: pin to be updated cluster components to current versions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1311394 (https://phabricator.wikimedia.org/T427400) (owner: 10Jelto) [08:09:41] (03Merged) 10jenkins-bot: Update calico-crds to calico v3.30.7 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306165 (https://phabricator.wikimedia.org/T427400) (owner: 10Jelto) [08:09:44] (03Merged) 10jenkins-bot: Update calico to v3.30.7 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1306307 (https://phabricator.wikimedia.org/T427400) (owner: 10Jelto) [08:10:41] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [08:13:43] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [08:14:52] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [08:16:23] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [08:18:05] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [08:18:35] (03PS1) 10Mpostoronca: Register the mediawiki.wikimedia_antiabuse.content_policy_score stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1321579 (https://phabricator.wikimedia.org/T432848) [08:19:59] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [08:31:32] (03CR) 10JMeybohm: [C:03+2] wikikube: Update staging codfw to calico 3.30 [puppet] - 10https://gerrit.wikimedia.org/r/1321889 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [08:31:35] (03CR) 10JMeybohm: [C:03+2] calico: Add support for v3.30, drop v3.23 [puppet] - 10https://gerrit.wikimedia.org/r/1321888 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [08:34:13] (03CR) 10Slyngshede: [C:03+1] "Looks good, also fixes: T431077" [puppet] - 10https://gerrit.wikimedia.org/r/1322571 (https://phabricator.wikimedia.org/T421952) (owner: 10Brouberol) [08:36:14] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for Chlod Alejandro - https://phabricator.wikimedia.org/T433791#12194269 (10jcrespo) >>! In T433791#12193056, @Chlod wrote: > ^ Done! I've also added an SSH key for my backup Yubikey. If additional permission is needed for that... [08:36:49] (03PS3) 10Brouberol: idm: add airflow-{analytics,analytics-product,platform-eng,wmde}-ops groups [puppet] - 10https://gerrit.wikimedia.org/r/1322571 (https://phabricator.wikimedia.org/T421952) [08:37:52] (03CR) 10Brouberol: [C:03+2] idm: sort airflow rules to help with next review [puppet] - 10https://gerrit.wikimedia.org/r/1322570 (https://phabricator.wikimedia.org/T421952) (owner: 10Brouberol) [08:37:56] (03CR) 10Brouberol: [C:03+2] idm: add airflow-{analytics,analytics-product,platform-eng,wmde}-ops groups [puppet] - 10https://gerrit.wikimedia.org/r/1322571 (https://phabricator.wikimedia.org/T421952) (owner: 10Brouberol) [08:38:00] (03CR) 10Brouberol: [C:03+2] admin: ensure all airflow-*-ops groups are marked as registered in bitu [puppet] - 10https://gerrit.wikimedia.org/r/1322572 (https://phabricator.wikimedia.org/T421952) (owner: 10Brouberol) [08:39:47] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for Chlod Alejandro - https://phabricator.wikimedia.org/T433791#12194277 (10Chlod) [08:39:58] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for Chlod Alejandro - https://phabricator.wikimedia.org/T433791#12194278 (10Chlod) Added here :) [08:45:29] 06SRE, 10SRE-swift-storage, 10MediaWiki-extensions-Score, 06Reader Experience Team: Add cache key information to metadata json - https://phabricator.wikimedia.org/T257093#12194325 (10MatthewVernon) @HFan-WMF note that whatever ends up happening about that patch, it won't address the problem of storing 300G... [08:47:32] (03PS1) 10JMeybohm: wikikube-staging: Update calico to v3.30 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322672 (https://phabricator.wikimedia.org/T427400) [08:52:07] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12194361 (10jcrespo) Thank you @VRiley-WMF I am booting the server- The server should be able to be up and not crashing without thermal paste or with it "crusty", even if a bit over the norm,... [08:52:30] 06SRE, 06Infrastructure-Foundations, 10WMDE Analytics, 13Patch-For-Review: Make airflow-wmde-ops managed in Bitu - https://phabricator.wikimedia.org/T431077#12194362 (10brouberol) [08:56:20] 06SRE, 06Infrastructure-Foundations, 10WMDE Analytics, 13Patch-For-Review: Make airflow-wmde-ops managed in Bitu - https://phabricator.wikimedia.org/T431077#12194384 (10brouberol) Fixed by https://phabricator.wikimedia.org/T434242 [08:56:38] 06SRE, 06Infrastructure-Foundations, 10WMDE Analytics, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: Make airflow-wmde-ops managed in Bitu - https://phabricator.wikimedia.org/T431077#12194387 (10brouberol) [08:56:40] 06SRE, 06Infrastructure-Foundations, 10WMDE Analytics, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: Make airflow-wmde-ops managed in Bitu - https://phabricator.wikimedia.org/T431077#12194388 (10brouberol) 05In progress→03Resolved [09:09:43] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [09:12:14] 10ops-magru: Too low optic power on - cr2-magru:xe-0/1/0 - https://phabricator.wikimedia.org/T434275 (10ayounsi) 03NEW p:05Triage→03High [09:15:46] !log started stress testing db1245 dbs T431115 [09:15:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:15:50] T431115: db1245 crashed - https://phabricator.wikimedia.org/T431115 [09:19:19] (03PS3) 10Ayounsi: WIP: Add TransportLinksInUsageNoRedundancy alert [alerts] - 10https://gerrit.wikimedia.org/r/1321906 [09:19:29] 10ops-magru: Too low optic power on - cr2-magru:xe-0/1/0 - https://phabricator.wikimedia.org/T434275#12194548 (10cmooney) > Maybe we can use the opportunity of the upcoming x-connect for T430371 to investigate the issue, clean the fiber, etc. And if no improvement follow up with EdgeUno? Yeah I think that makes... [09:23:28] (03CR) 10CI reject: [V:04-1] WIP: Add TransportLinksInUsageNoRedundancy alert [alerts] - 10https://gerrit.wikimedia.org/r/1321906 (owner: 10Ayounsi) [09:29:51] (03CR) 10Klausman: [C:03+2] "One small suggestion, otherwise LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1321545 (owner: 10Dpogorzelski) [09:33:06] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops: db1245 crashed - https://phabricator.wikimedia.org/T431115#12194627 (10jcrespo) I did some unrealistic scenario, which is maximizing all cpu cores while also running a mysql benchmark, and temperature stabilized around 83 degrees Celsius, with no cra... [09:43:05] (03PS1) 10Jcrespo: installserver: Set db1245 as full uefi reimage, remove db1265/85 [puppet] - 10https://gerrit.wikimedia.org/r/1322690 (https://phabricator.wikimedia.org/T431115) [09:44:55] (03CR) 10Blake: [C:03+2] httpbb: Release v0.0.6 [software/httpbb] - 10https://gerrit.wikimedia.org/r/1321516 (https://phabricator.wikimedia.org/T434052) (owner: 10Blake) [09:45:28] (03CR) 10Jcrespo: "Manuel, I am not sure if this is the right way- as I haven't handled UEFI plans for dbs, can you confirm carefully this is what's intended" [puppet] - 10https://gerrit.wikimedia.org/r/1322690 (https://phabricator.wikimedia.org/T431115) (owner: 10Jcrespo) [09:47:16] (03Merged) 10jenkins-bot: httpbb: Release v0.0.6 [software/httpbb] - 10https://gerrit.wikimedia.org/r/1321516 (https://phabricator.wikimedia.org/T434052) (owner: 10Blake) [09:49:43] (03PS2) 10Jcrespo: installserver: Set db1245 as full uefi reimage, remove db1265/85 [puppet] - 10https://gerrit.wikimedia.org/r/1322690 (https://phabricator.wikimedia.org/T431115) [09:49:52] (03PS3) 10Jcrespo: installserver: Set db1245 as full uefi reimage, remove db1265/85 [puppet] - 10https://gerrit.wikimedia.org/r/1322690 (https://phabricator.wikimedia.org/T431115) [09:49:54] (03CR) 10Marostegui: [C:03+1] "Copied votes on follow-up patch sets have been updated:" [puppet] - 10https://gerrit.wikimedia.org/r/1322690 (https://phabricator.wikimedia.org/T431115) (owner: 10Jcrespo) [09:53:05] (03CR) 10Jcrespo: [C:03+2] installserver: Set db1245 as full uefi reimage, remove db1265/85 [puppet] - 10https://gerrit.wikimedia.org/r/1322690 (https://phabricator.wikimedia.org/T431115) (owner: 10Jcrespo) [09:53:15] (03PS4) 10Jcrespo: installserver: Set db1245 as full uefi reimage, remove db1265/85 [puppet] - 10https://gerrit.wikimedia.org/r/1322690 (https://phabricator.wikimedia.org/T431115) [09:53:33] (03PS2) 10Marostegui: mariadb: Decommission db1178 [puppet] - 10https://gerrit.wikimedia.org/r/1322520 (https://phabricator.wikimedia.org/T433471) [09:53:33] (03PS1) 10Marostegui: mariadb: Productionize db1271 [puppet] - 10https://gerrit.wikimedia.org/r/1322695 (https://phabricator.wikimedia.org/T407942) [09:54:48] (03CR) 10Marostegui: [C:03+2] mariadb: Productionize db1271 [puppet] - 10https://gerrit.wikimedia.org/r/1322695 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [09:54:57] (03CR) 10Marostegui: [C:03+2] mariadb: Decommission db1178 [puppet] - 10https://gerrit.wikimedia.org/r/1322520 (https://phabricator.wikimedia.org/T433471) (owner: 10Marostegui) [09:55:04] (03CR) 10Jcrespo: [C:03+2] installserver: Set db1245 as full uefi reimage, remove db1265/85 [puppet] - 10https://gerrit.wikimedia.org/r/1322690 (https://phabricator.wikimedia.org/T431115) (owner: 10Jcrespo) [09:59:41] !log marostegui@cumin1003 START - Cookbook sre.mysql.clone of db1156.eqiad.wmnet onto db1271.eqiad.wmnet [09:59:45] !log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1156: Depool db1156.eqiad.wmnet to then clone it to db1271.eqiad.wmnet - marostegui@cumin1003 [10:00:23] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:01:02] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1156: Depool db1156.eqiad.wmnet to then clone it to db1271.eqiad.wmnet - marostegui@cumin1003 [10:03:44] PROBLEM - MariaDB Replica IO: s2 on db1155 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db1156.eqiad.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db1156.eqiad.wmnet (111 Connection refused) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [10:05:00] ^ me [10:06:36] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 21 hosts with reason: cloning [10:09:43] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:13:45] 10ops-eqiad, 06SRE, 10Data-Persistence-Backup, 06DC-Ops, 13Patch-For-Review: db1245 crashed - https://phabricator.wikimedia.org/T431115#12194718 (10jcrespo) Sadly, it crashed again at 9:51 UTC, and it didn't seem it had anything to do with temperature, as it did after the stress test finalized: {F9747892... [10:18:26] !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'. [10:19:47] !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [10:20:11] !log jayme@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [10:20:47] !log jayme@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [10:21:07] !log jayme@deploy1003 helmfile [codfw] START helmfile.d/admin 'apply'. [10:22:01] !log jayme@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'apply'. [10:22:37] !log jayme@deploy1003 helmfile [eqiad] START helmfile.d/admin 'apply'. [10:23:11] !log jayme@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'apply'. [10:41:50] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [10:43:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 13.71% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:46:52] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 226.07 ms [10:47:33] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q4:rack/setup/install sretest2010 Config J 1P test host - https://phabricator.wikimedia.org/T394357#12194858 (10MatthewVernon) @RobH from my point of view, the JBOD disk-swap behaviour seems good, so if @elukey is happy that we're going to be able to reliab... [10:48:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.14% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:57:47] 06SRE, 10SRE-swift-storage, 10MediaWiki-extensions-Score, 06Reader Experience Team: Add cache key information to metadata json - https://phabricator.wikimedia.org/T257093#12194872 (10MatthewVernon) >>! In T257093#12193179, @TheDJ wrote: > There is no rush, it’s just that we have 315GB of storage of which a... [11:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260807T0700) [11:00:05] jelto, arnoldokoth, mutante, and arnaudb: #bothumor I � Unicode. All rise for GitLab version upgrades deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260807T1100). [11:00:53] 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: Netbox report: alert on cables in 'planned' state for more than two months - https://phabricator.wikimedia.org/T432329#12194879 (10ayounsi) 05Open→03Resolved a:03ayounsi All done! [11:02:44] RECOVERY - MariaDB Replica IO: s2 on db1155 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [11:12:52] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1156: Pool db1156.eqiad.wmnet in after cloning [11:15:45] (03PS1) 10Marostegui: installserver: Do not format db1271 [puppet] - 10https://gerrit.wikimedia.org/r/1322717 [11:19:55] (03PS1) 10Marostegui: instances.yaml: Add db1271 [puppet] - 10https://gerrit.wikimedia.org/r/1322718 (https://phabricator.wikimedia.org/T407942) [11:23:37] (03CR) 10Marostegui: [C:03+2] instances.yaml: Add db1271 [puppet] - 10https://gerrit.wikimedia.org/r/1322718 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [11:23:41] (03CR) 10Marostegui: [C:03+2] installserver: Do not format db1271 [puppet] - 10https://gerrit.wikimedia.org/r/1322717 (owner: 10Marostegui) [11:30:07] !log marostegui@cumin1003 dbctl commit (dc=all): 'Adding db1271 to dbctl', diff saved to https://phabricator.wikimedia.org/P95945 and previous config saved to /var/cache/conftool/dbconfig/20260807-113006-marostegui.json [11:31:13] 10SRE-tools, 10homer, 06Infrastructure-Foundations: Homer: optimize API calls to Netbox - https://phabricator.wikimedia.org/T271864#12194979 (10ayounsi) 05Open→03Resolved a:03ayounsi Going to boldly close that one as we have improved things significantly by using GraphQL. [11:33:41] (03PS1) 10Marostegui: db1271: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1322730 (https://phabricator.wikimedia.org/T407942) [11:34:43] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:34:55] (03PS5) 10Ayounsi: Decom cookbook: use Homer (except for VCs) and clean BGP [cookbooks] - 10https://gerrit.wikimedia.org/r/1275821 (https://phabricator.wikimedia.org/T416313) [11:34:59] (03CR) 10Marostegui: [C:03+2] db1271: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1322730 (https://phabricator.wikimedia.org/T407942) (owner: 10Marostegui) [11:35:01] !log sudo -i reprepro -C main include bookworm-wikimedia ${HOME}/httpbb/bookworm/httpbb_${VERSION?}-1_amd64.changes #T434052 [11:35:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:35:05] T434052: Release httpbb 0.0.6 - https://phabricator.wikimedia.org/T434052 [11:38:28] !log sudo -i reprepro -C main include trixie-wikimedia ${HOME}/httpbb/trixie/httpbb_${VERSION?}-1+deb13u1_amd64.changes #T434052 [11:38:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:50:42] (03CR) 10JMeybohm: [C:03+2] wikikube-staging: Update calico to v3.30 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322672 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [11:58:47] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-private-users for aramirezwmf - https://phabricator.wikimedia.org/T433999#12195050 (10jcrespo) 05In progress→03Open [11:59:08] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-private-users for aramirezwmf - https://phabricator.wikimedia.org/T433999#12195051 (10jcrespo) p:05Triage→03Medium [12:00:03] (03Merged) 10jenkins-bot: wikikube-staging: Update calico to v3.30 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322672 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [12:02:13] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1156: Pool db1156.eqiad.wmnet in after cloning [12:05:46] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for Chlod Alejandro - https://phabricator.wikimedia.org/T433791#12195064 (10jcrespo) [12:05:56] !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'sync'. [12:06:05] !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [12:06:31] !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'sync'. [12:09:19] !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [12:19:04] (03PS2) 10Jcrespo: admin: Add chlod to production access and deployment rights [puppet] - 10https://gerrit.wikimedia.org/r/1322077 (https://phabricator.wikimedia.org/T433791) (owner: 10Chlod Alejandro) [12:19:40] !log updated calico to v3.30.7 on staging-codfw - T427400 [12:19:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:19:44] T427400: Update calico to 3.30 - https://phabricator.wikimedia.org/T427400 [12:22:27] (03CR) 10Jcrespo: [C:04-1] "Blocked until NDA approval by legal." [puppet] - 10https://gerrit.wikimedia.org/r/1322077 (https://phabricator.wikimedia.org/T433791) (owner: 10Chlod Alejandro) [12:24:22] (03PS1) 10Jforrester: usage: Don't try to record Function usage for a target that isn't a ZID [extensions/WikiLambda] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1322752 (https://phabricator.wikimedia.org/T434194) [12:24:40] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for Chlod Alejandro - https://phabricator.wikimedia.org/T433791#12195091 (10jcrespo) As you can see in the amended PR, you will be provided deployment rights, which includes some spiderpig rights. After access is approval you... [12:26:05] (03PS3) 10Jcrespo: admin: Add chlod to production access and deployment rights [puppet] - 10https://gerrit.wikimedia.org/r/1322077 (https://phabricator.wikimedia.org/T433791) (owner: 10Chlod Alejandro) [12:27:25] (03CR) 10CI reject: [V:04-1] usage: Don't try to record Function usage for a target that isn't a ZID [extensions/WikiLambda] (wmf/1.47.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1322752 (https://phabricator.wikimedia.org/T434194) (owner: 10Jforrester) [12:27:28] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for Chlod Alejandro - https://phabricator.wikimedia.org/T433791#12195094 (10jcrespo) p:05Triage→03High [12:27:33] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for Chlod Alejandro - https://phabricator.wikimedia.org/T433791#12195095 (10jcrespo) a:03jcrespo [12:29:39] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for BGerdemann_(WMF) - https://phabricator.wikimedia.org/T433313#12195098 (10jcrespo) p:05Triage→03High [12:38:10] (03PS1) 10JMeybohm: calico: Disable usage reporting [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322755 (https://phabricator.wikimedia.org/T427400) [12:44:24] (03CR) 10Jcrespo: admin: Add chlod to production access and deployment rights [puppet] - 10https://gerrit.wikimedia.org/r/1322077 (https://phabricator.wikimedia.org/T433791) (owner: 10Chlod Alejandro) [12:48:39] PROBLEM - Druid historical on an-druid1006 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [13:02:26] !log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1271: Pool db1271.eqiad.wmnet in after cloning [13:02:54] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:04:12] (03PS1) 10Ayounsi: RANCID: ignore Fan Speed in SR-Linux [puppet] - 10https://gerrit.wikimedia.org/r/1322761 [13:04:45] (03PS2) 10Ayounsi: RANCID: ignore Fan Speed in SR-Linux [puppet] - 10https://gerrit.wikimedia.org/r/1322761 [13:04:47] (03CR) 10CI reject: [V:04-1] RANCID: ignore Fan Speed in SR-Linux [puppet] - 10https://gerrit.wikimedia.org/r/1322761 (owner: 10Ayounsi) [13:05:17] (03PS3) 10Ayounsi: RANCID: ignore Fan Speed in SR-Linux [puppet] - 10https://gerrit.wikimedia.org/r/1322761 [13:07:39] RECOVERY - Druid historical on an-druid1006 is OK: PROCS OK: 1 process with command name java, args org.apache.druid.cli.Main server historical https://wikitech.wikimedia.org/wiki/Analytics/Systems/Druid [13:09:43] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [13:19:33] (03CR) 10JMeybohm: [C:03+2] calico: Disable usage reporting [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322755 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [13:29:51] (03Merged) 10jenkins-bot: calico: Disable usage reporting [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322755 (https://phabricator.wikimedia.org/T427400) (owner: 10JMeybohm) [13:50:30] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1271: Pool db1271.eqiad.wmnet in after cloning [13:50:32] !log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1156.eqiad.wmnet onto db1271.eqiad.wmnet [14:00:38] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:15:04] 06SRE, 10SRE-Access-Requests: Grant access to analytics-privatedata-users for EAlbizzati-WMF - https://phabricator.wikimedia.org/T434034#12195399 (10EAlbizzati-WMF) Thanks y'all. it's working. [14:17:17] !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'sync'. [14:19:06] !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [14:26:46] (03CR) 10Blake: [C:03+2] Update to v1.39.0 [debs/envoyproxy] (v1.39) - 10https://gerrit.wikimedia.org/r/1321526 (https://phabricator.wikimedia.org/T421418) (owner: 10Blake) [14:35:23] RESOLVED: CertAlmostExpired: gNMI TLS certificate for lsw1-d3-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:43:19] (03CR) 10Cathal Mooney: [C:03+1] RANCID: ignore Fan Speed in SR-Linux [puppet] - 10https://gerrit.wikimedia.org/r/1322761 (owner: 10Ayounsi) [15:07:10] (03PS3) 10Blake: mw-pretrain: Add a jobrunner values.yaml. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1321561 (https://phabricator.wikimedia.org/T427668) [15:07:33] (03CR) 10Blake: mw-pretrain: Add a jobrunner values.yaml. (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1321561 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [15:11:15] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [15:13:10] !log cdobbins@cumin1003 START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie [15:13:19] 10ops-eqsin, 06SRE, 06DC-Ops, 06Traffic: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#12195585 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1003 for host cp5022.eqsin.wmnet with OS trixie [15:14:52] (03CR) 10Scott French: [C:03+1] "Thanks, Blake!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1321561 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [15:16:20] (03PS1) 10Blake: kubernetes: Add a jobrunner deployment for mw-pretrain. [puppet] - 10https://gerrit.wikimedia.org/r/1322782 (https://phabricator.wikimedia.org/T427668) [15:20:11] (03CR) 10Scott French: [C:03+1] kubernetes: Add a jobrunner deployment for mw-pretrain. [puppet] - 10https://gerrit.wikimedia.org/r/1322782 (https://phabricator.wikimedia.org/T427668) (owner: 10Blake) [15:26:39] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 248.36 ms [15:33:19] 06SRE, 10SRE-swift-storage, 10MediaWiki-extensions-Score, 06Reader Experience Team: Add cache key information to metadata json - https://phabricator.wikimedia.org/T257093#12195665 (10Cparle) I don't think those pngs are in fact useless. I might have the sequencing wrong here, but this is how it looks to me... [15:34:43] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:42:49] 06SRE, 10Wikimedia-Mailing-lists: Find list owners for lists without them - https://phabricator.wikimedia.org/T281779#12195688 (10Aklapper) 05Open→03Declined Boldly declining for now per last two comments; please reopen if there is a point that I missed. Thank you! [15:47:24] (03PS2) 10Ahmon Dancy: scap: Add optional logstash credentials file [puppet] - 10https://gerrit.wikimedia.org/r/1321653 (https://phabricator.wikimedia.org/T434114) [15:47:32] (03CR) 10Ahmon Dancy: scap: Add optional logstash credentials file (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1321653 (https://phabricator.wikimedia.org/T434114) (owner: 10Ahmon Dancy) [16:00:00] (03PS1) 10Bking: base: don't run fstrim against EFI system partition [puppet] - 10https://gerrit.wikimedia.org/r/1322791 (https://phabricator.wikimedia.org/T434299) [16:00:17] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1322791 (https://phabricator.wikimedia.org/T434299) (owner: 10Bking) [16:00:37] (03CR) 10CI reject: [V:04-1] base: don't run fstrim against EFI system partition [puppet] - 10https://gerrit.wikimedia.org/r/1322791 (https://phabricator.wikimedia.org/T434299) (owner: 10Bking) [16:02:13] (03PS2) 10Bking: base: don't run fstrim against EFI system partition [puppet] - 10https://gerrit.wikimedia.org/r/1322791 (https://phabricator.wikimedia.org/T434299) [16:03:35] (03PS3) 10Bking: base: don't run fstrim against EFI system partition [puppet] - 10https://gerrit.wikimedia.org/r/1322791 (https://phabricator.wikimedia.org/T434299) [16:04:17] (03CR) 10Bking: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1322791 (https://phabricator.wikimedia.org/T434299) (owner: 10Bking) [16:06:07] !log cdobbins@cumin1003 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie [16:06:14] 10ops-eqsin, 06SRE, 06DC-Ops, 06Traffic: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#12195852 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1003 for host cp5022.eqsin.wmnet with OS trixie executed with errors: - cp5022 (**FAIL**) - Removed from... [16:07:54] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:08:56] 06SRE, 10LDAP-Access-Requests: Grant Access to NDA for jaleman-vdr-wmf - https://phabricator.wikimedia.org/T433417#12195856 (10JArguello-WMF) hi @jcrespo thanks for the reply, Jose is already a part of the WMF group if I am not mistaken. https://ldap.toolforge.org/user/jaleman-vdr-wmf Yes there is an end date... [16:14:43] FIRING: [3x] JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:15:51] raine we would like to do a security deploy for a critical issue [16:16:01] maryum: we are discussing this in the security channel [16:16:06] sorry! [16:16:08] riku is there [16:16:09] no worries! [16:16:24] I'm not in there so just let me know [16:16:30] oh I see. noted. [16:18:19] <_joe_> !ack [16:18:19] 8234 (ACKED) Manual (paged) by rsilvola (riku.silvola@wikimedia.org): Emergency deploy [16:26:34] (03PS1) 10Lerickson: WDQS V2: add http_proxy environment variable to the qlever container. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322799 (https://phabricator.wikimedia.org/T434321) [16:27:41] !log urbanecm@deploy1003 mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --verbose --scoreLessThan=0.7 --exceptDatasetChecksums=T434319-enwiki-models.txt # T434319 [16:27:45] T434319: Revalidate Add Link suggestions on enwiki - https://phabricator.wikimedia.org/T434319 [16:33:49] (03CR) 10Lerickson: [C:03+2] WDQS V2: add http_proxy environment variable to the qlever container. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322799 (https://phabricator.wikimedia.org/T434321) (owner: 10Lerickson) [16:36:45] (03Merged) 10jenkins-bot: WDQS V2: add http_proxy environment variable to the qlever container. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322799 (https://phabricator.wikimedia.org/T434321) (owner: 10Lerickson) [16:42:01] preparing to run scap for a security deploy [16:47:43] scap is running [16:48:46] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [16:50:06] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [16:50:29] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [16:50:51] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [16:52:31] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [16:54:06] !log Deployed security fix for T434278 [16:54:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:58:14] (03PS1) 10Ssingh: page_form.html: be explicit about not entering private data [software/klaxon] - 10https://gerrit.wikimedia.org/r/1322809 [17:04:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [17:09:43] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [17:13:06] !log ihurbain@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [17:13:46] !log ihurbain@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [17:13:47] !log ihurbain@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [17:14:28] !log ihurbain@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [17:25:25] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [17:38:26] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thanks!!" [software/klaxon] - 10https://gerrit.wikimedia.org/r/1078077 (owner: 10Reedy) [17:38:48] (03CR) 10Andrea Denisse: [C:03+1] "recheck" [software/klaxon] - 10https://gerrit.wikimedia.org/r/1078077 (owner: 10Reedy) [17:39:49] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [17:59:56] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [18:00:04] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [18:00:15] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [18:01:02] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [18:04:11] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [18:06:23] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [18:06:32] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [18:06:41] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [18:07:54] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [18:08:12] (03PS1) 10Cwhite: logs-api: allow GET POST for $index/_search endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1322837 (https://phabricator.wikimedia.org/T434114) [18:08:37] (03PS1) 10Scott French: zookeeper::server: Use default zookeeper CLASSPATH when using 3.4 [puppet] - 10https://gerrit.wikimedia.org/r/1322828 (https://phabricator.wikimedia.org/T424266) [18:08:47] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [18:09:14] (03CR) 10CI reject: [V:04-1] logs-api: allow GET POST for $index/_search endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1322837 (https://phabricator.wikimedia.org/T434114) (owner: 10Cwhite) [18:09:30] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [18:09:49] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [18:11:47] (03PS2) 10Cwhite: logs-api: allow GET POST for $index/_search endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1322837 (https://phabricator.wikimedia.org/T434114) [18:12:04] (03PS3) 10Cwhite: logs-api: allow GET POST for $index/_search endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1322837 (https://phabricator.wikimedia.org/T434114) [18:12:44] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [18:16:15] (03PS1) 10Atsuko: airflow3 support [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322840 (https://phabricator.wikimedia.org/T433388) [18:16:56] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [18:17:22] (03PS2) 10Atsuko: airflow3 support [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322840 (https://phabricator.wikimedia.org/T433388) [18:19:49] FIRING: [3x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [18:22:23] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [18:22:43] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [18:23:00] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [18:27:11] (03PS3) 10Atsuko: airflow3 support [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322840 (https://phabricator.wikimedia.org/T433388) [18:35:27] !log jhancock@cumin1003 START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [18:35:37] (03CR) 10RLazarus: [C:03+1] zookeeper::server: Use default zookeeper CLASSPATH when using 3.4 [puppet] - 10https://gerrit.wikimedia.org/r/1322828 (https://phabricator.wikimedia.org/T424266) (owner: 10Scott French) [18:41:41] !log jhancock@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [18:48:21] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12196363 (10Jhancock.wm) @jasmine_ is it alright if we change the status of this server to failed and down time it? I can't seem to... [18:52:39] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [18:52:45] FIRING: [5x] WidespreadPuppetFailure: Puppet has failed in codfw - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [18:54:49] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [18:54:59] 10ops-magru: Too low optic power on - cr2-magru:xe-0/1/0 - https://phabricator.wikimedia.org/T434275#12196371 (10RobH) I don't want to combine things, as the cross connect order is via an order form an email, not the portal. So having an unrelated troubleshooting step will overly complicate that particular orde... [18:57:45] FIRING: [7x] WidespreadPuppetFailure: Puppet has failed in codfw - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [19:05:48] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12196383 (10jasmine_) >>! In T406596#12196363, @Jhancock.wm wrote: > @jasmine_ is it alright if we change the status of this server... [19:19:26] (03PS1) 10Lerickson: Update Qlever to a new image with http_proxy support. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322872 (https://phabricator.wikimedia.org/T434321) [19:20:45] (03CR) 10Lerickson: [C:03+1] WDQSv2: Qlever index rebuild container (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1320946 (https://phabricator.wikimedia.org/T432627) (owner: 10Trueg) [19:21:27] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [19:22:51] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [19:23:11] (03PS2) 10Lerickson: WIP: Update Qlever to a new image with http_proxy support. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322872 (https://phabricator.wikimedia.org/T434321) [19:25:54] !log jhancock@cumin1003 START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [19:32:14] !log jhancock@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [19:32:20] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [19:32:26] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [19:32:33] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [19:33:23] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [19:35:52] (03PS1) 10Andrew Bogott: Revert "ceph.conf: support adjusting slow ops health messages" [puppet] - 10https://gerrit.wikimedia.org/r/1322877 [19:35:55] (03PS1) 10Andrew Bogott: Revert "Ceph: support setting bdev_enable_discard" [puppet] - 10https://gerrit.wikimedia.org/r/1322878 [19:36:28] (03CR) 10CI reject: [V:04-1] Revert "ceph.conf: support adjusting slow ops health messages" [puppet] - 10https://gerrit.wikimedia.org/r/1322877 (owner: 10Andrew Bogott) [19:36:38] (03CR) 10CI reject: [V:04-1] Revert "Ceph: support setting bdev_enable_discard" [puppet] - 10https://gerrit.wikimedia.org/r/1322878 (owner: 10Andrew Bogott) [19:36:57] (03PS2) 10Andrew Bogott: Revert "Ceph: support setting bdev_enable_discard" [puppet] - 10https://gerrit.wikimedia.org/r/1322878 (https://phabricator.wikimedia.org/T429387) [19:37:14] (03PS3) 10Andrew Bogott: Revert "Ceph: support setting bdev_enable_discard" [puppet] - 10https://gerrit.wikimedia.org/r/1322878 (https://phabricator.wikimedia.org/T429387) [19:37:53] (03CR) 10CI reject: [V:04-1] Revert "Ceph: support setting bdev_enable_discard" [puppet] - 10https://gerrit.wikimedia.org/r/1322878 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [19:43:14] (03PS4) 10Andrew Bogott: Revert "Ceph: support setting bdev_enable_discard" [puppet] - 10https://gerrit.wikimedia.org/r/1322878 (https://phabricator.wikimedia.org/T429387) [19:43:50] (03CR) 10CI reject: [V:04-1] Revert "Ceph: support setting bdev_enable_discard" [puppet] - 10https://gerrit.wikimedia.org/r/1322878 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [19:44:40] (03PS5) 10Andrew Bogott: Revert "Ceph: support setting bdev_enable_discard" [puppet] - 10https://gerrit.wikimedia.org/r/1322878 (https://phabricator.wikimedia.org/T429387) [19:45:14] (03CR) 10CI reject: [V:04-1] Revert "Ceph: support setting bdev_enable_discard" [puppet] - 10https://gerrit.wikimedia.org/r/1322878 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [19:45:22] !log jhancock@cumin1003 START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [19:46:36] (03PS6) 10Andrew Bogott: Revert "Ceph: support setting bdev_enable_discard" [puppet] - 10https://gerrit.wikimedia.org/r/1322878 (https://phabricator.wikimedia.org/T429387) [19:48:48] 06SRE, 10Pywikibot, 06Traffic, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error - https://phabricator.wikimedia.org/T421642#12196506 (10Matthias) It seems today is a bad day again for pywikibot trying to make only a single edit. [19:51:42] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1322878 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [19:52:00] !log jhancock@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [19:54:38] (03CR) 10Andrew Bogott: [C:03+2] Revert "Ceph: support setting bdev_enable_discard" [puppet] - 10https://gerrit.wikimedia.org/r/1322878 (https://phabricator.wikimedia.org/T429387) (owner: 10Andrew Bogott) [20:05:09] (03CR) 10Andrea Denisse: WIP: Add TransportLinksInUsageNoRedundancy alert (032 comments) [alerts] - 10https://gerrit.wikimedia.org/r/1321906 (owner: 10Ayounsi) [20:14:55] FIRING: JobUnavailable: Reduced availability for job liberica in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:20:21] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-int releases routed via main at eqiad: 22.57% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-int&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:20:57] FIRING: ProbeDown: Service chart-renderer:30443 has failed probes (http_chart-renderer_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#chart-renderer:30443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:24:46] FIRING: ProbeDown: Service chart-renderer:30443 has failed probes (http_chart-renderer_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#chart-renderer:30443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:25:57] RESOLVED: ProbeDown: Service chart-renderer:30443 has failed probes (http_chart-renderer_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#chart-renderer:30443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:28:35] 06SRE, 06tools-infrastructure-team: openjdk upgrade breaks puppetservers - https://phabricator.wikimedia.org/T434339 (10Andrew) 03NEW [20:28:54] 06SRE, 06tools-infrastructure-team: openjdk upgrade breaks puppetservers - https://phabricator.wikimedia.org/T434339#12196607 (10Andrew) p:05Triage→03Unbreak! [20:29:41] RESOLVED: ProbeDown: Service chart-renderer:30443 has failed probes (http_chart-renderer_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#chart-renderer:30443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:30:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-int releases routed via main at eqiad: 20.3% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-int&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:32:38] !log restarting puppetserver service on puppetserver* for T434339 [20:32:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:32:43] T434339: openjdk upgrade breaks puppetservers - https://phabricator.wikimedia.org/T434339 [20:40:25] 06SRE, 06tools-infrastructure-team: openjdk upgrade breaks puppetservers - https://phabricator.wikimedia.org/T434339#12196637 (10Andrew) 05Open→03Resolved a:03Andrew @taavi observed that my dependency-chasing was in vain and that we just needed to restart the puppetservers. @jhathaway upgraded the jd... [20:44:00] 06SRE, 06MediaWiki-Core-Platform-Team, 05Front-end Modernization: Introduce a Front-end Build Step for MediaWiki Skins and Extensions - https://phabricator.wikimedia.org/T279108#12196641 (10egardner) A lot has changed since 2021. At this point it is possible to ship Vue-based UIs via ResourceLoader, written... [20:46:21] (03CR) 10Bking: [C:03+1] airflow3 support [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322840 (https://phabricator.wikimedia.org/T433388) (owner: 10Atsuko) [20:54:36] !log jhancock@cumin1003 START - Cookbook sre.hosts.reimage for host wikikube-ctrl2006.codfw.wmnet with OS trixie [20:54:51] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Kubernetes, 13Patch-For-Review: wikikube-ctrl2006 implementation tracking - https://phabricator.wikimedia.org/T406596#12196674 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host wikikube-ctrl2006.co... [21:07:45] FIRING: [7x] WidespreadPuppetFailure: Puppet has failed in codfw - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [21:09:43] FIRING: [3x] CirrusSearchSaneitizerFixRateTooHigh: MediaWiki CirrusSearch Saneitizer is fixing an abnormally high number of documents in cloudelastic - https://wikitech.wikimedia.org/wiki/Search/CirrusStreamingUpdater#San(e)itizing - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchSaneitizerFixRateTooHigh [21:13:16] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322920 [21:27:45] FIRING: [5x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [21:32:45] RESOLVED: [5x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [21:47:13] jhancock@cumin1003 reimage (PID 1529717) is awaiting input [21:49:17] 10ops-codfw, 06DC-Ops, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: Noisy alerts: investigate fstrim.service failure on dse-k8s-wdqs2001 (new Supermicro host) - https://phabricator.wikimedia.org/T434299#12196765 (10bking) Correction: the problem is localized to `dse-k8s-wdqs2001`, a b... [21:49:23] 10ops-codfw, 06DC-Ops, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: Noisy alerts: investigate fstrim.service failure on dse-k8s-wdqs2001 (new Supermicro host) - https://phabricator.wikimedia.org/T434299#12196777 (10bking) [21:50:11] 10ops-codfw, 06DC-Ops, 06Data-Platform-SRE (2026-07-03 - 2026-07-31), 13Patch-For-Review: Noisy alerts: investigate fstrim.service failure on dse-k8s-wdqs2001 (new Supermicro host) - https://phabricator.wikimedia.org/T434299#12196780 (10bking) [22:20:03] (03PS1) 10Superpes15: [slwiki] Reverting temporary logo for Wikipedia 25 (Vector legacy + Vector 2022) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1322961 (https://phabricator.wikimedia.org/T414265) [22:20:57] (03PS2) 10Superpes15: [slwiki] Reverting temporary logo for Wikipedia 25 (Vector legacy + Vector 2022) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1322961 (https://phabricator.wikimedia.org/T414265) [22:25:45] 06SRE, 10SRE-swift-storage, 10MediaWiki-extensions-Score, 06Reader Experience Team: Add cache key information to metadata json - https://phabricator.wikimedia.org/T257093#12196817 (10Volker_E) >>! In T257093#12195665, @Cparle wrote: > I don't think those pngs are in fact useless. I might have the sequencin... [22:26:24] 06SRE, 10SRE-swift-storage, 10MediaWiki-extensions-Score, 06Reader Experience Team: Add cache key information to metadata json - https://phabricator.wikimedia.org/T257093#12196821 (10Volker_E) [22:27:21] (03PS1) 10JHathaway: nftables: add ipencap support [puppet] - 10https://gerrit.wikimedia.org/r/1322963 (https://phabricator.wikimedia.org/T433601) [22:30:14] (03CR) 10JHathaway: "I pushed a patch for adding IPIP support, 1322963. But, I think you are right that the MSS piece would be touch to fit into our current pu" [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) (owner: 10Cathal Mooney) [22:35:52] (03PS1) 10Aleksandar Mastilovic: Update the blunderbuss chart to the latest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1322970 (https://phabricator.wikimedia.org/T434344) [22:41:39] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [22:41:40] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [22:43:52] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [22:45:00] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [22:55:04] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-rollback - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [23:06:20] (03CR) 10MusikAnimal: "I won't -1 (the new [CodeMirrorVisualEditorHighlight](https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/CodeMirror/+/58e" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1322175 (https://phabricator.wikimedia.org/T432558) (owner: 10DLynch) [23:11:43] (03CR) 10Cwhite: [C:03+2] logs-api: allow GET POST for $index/_search endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1322837 (https://phabricator.wikimedia.org/T434114) (owner: 10Cwhite) [23:29:04] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [23:30:08] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [23:42:27] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1322997 [23:42:27] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1322997 (owner: 10TrainBranchBot) [23:52:13] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1322997 (owner: 10TrainBranchBot)