[00:05:28] PROBLEM - Check unit status of kube-reload-certs on deploy1003 is CRITICAL: CRITICAL: Status of the systemd unit kube-reload-certs https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [00:08:00] (03CR) 10Dzahn: [C:03+1] wikistats: update deploy script, fix xmldump and small fixes [puppet] - 10https://gerrit.wikimedia.org/r/1347785 (https://phabricator.wikimedia.org/T409014) (owner: 10Dzahn) [00:38:29] (03PS1) 10Dzahn: miscweb: attribution: bump image to 2026-10-02-003517 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347900 (https://phabricator.wikimedia.org/T439779) [00:39:00] !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp11[08,10,12,14].eqiad.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 () [00:40:32] (03CR) 10Dzahn: [C:03+2] miscweb: attribution: bump image to 2026-10-02-003517 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347900 (https://phabricator.wikimedia.org/T439779) (owner: 10Dzahn) [00:43:32] (03Merged) 10jenkins-bot: miscweb: attribution: bump image to 2026-10-02-003517 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347900 (https://phabricator.wikimedia.org/T439779) (owner: 10Dzahn) [00:52:31] !log dzahn@deploy1003 helmfile [staging] START helmfile.d/services/miscweb: apply [00:53:13] !log dzahn@deploy1003 helmfile [staging] DONE helmfile.d/services/miscweb: apply [00:53:29] !log dzahn@deploy1003 helmfile [codfw] START helmfile.d/services/miscweb: apply [00:54:07] !log dzahn@deploy1003 helmfile [codfw] DONE helmfile.d/services/miscweb: apply [00:54:14] !log dzahn@deploy1003 helmfile [eqiad] START helmfile.d/services/miscweb: apply [00:54:37] !log dzahn@deploy1003 helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [01:00:03] (03Abandoned) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1346853 (owner: 10TrainBranchBot) [01:08:58] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1347910 [01:08:58] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1347910 (owner: 10TrainBranchBot) [01:16:55] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:17:35] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1347910 (owner: 10TrainBranchBot) [01:22:03] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12387906 (10Dzahn) [01:22:05] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12387907 (10Dzahn) a:05EAlbizzati-WMF→03None [01:22:42] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for EAlbizzati-WMF - https://phabricator.wikimedia.org/T437741#12387909 (10Dzahn) 05Open→03In progress [01:46:50] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [02:00:50] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:08:26] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 36s) [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:34:02] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [02:39:27] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1238.eqiad.wmnet with reason: maint [02:48:22] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-c6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [03:28:44] 10SRE-SLO, 06SRE Observability, 10Test Kitchen: Support SLOs based on Test Kitchen instruments - https://phabricator.wikimedia.org/T438274#12388030 (10RLazarus) Thanks @mpopov and @JVanderhoop-WMF! Appreciate the correction in particular, that sounds really promising. This has come up in draft SLOs for a co... [03:34:13] FIRING: NetworkDeviceAlarmActive: Alarm active on ssw1-f1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=ssw1-f1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [04:03:42] 10SRE-SLO, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): page_change SLO windows - https://phabricator.wikimedia.org/T438054#12388089 (10RLazarus) Thanks @APizzata-WMF! Sorry to take so long to reply; I'm about to be out of office for a couple of weeks, so leaving some thoughts here before I go, b... [05:16:55] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:46:43] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar: Grant access to analytics-privatedata-users for Eniroomand-WMF - https://phabricator.wikimedia.org/T439644#12388109 (10ayounsi) Great! Please open a new ticket, the person in charge of access requests changes every 7 days (it'... [05:46:50] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [06:00:04] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261002T0600) [06:31:00] FIRING: PingLossPercent: Blackbox probe packet loss 4.375% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [06:34:02] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [06:48:22] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-c6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [06:49:45] FIRING: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [06:51:45] (03PS1) 10Muehlenhoff: Remove access for dimakoushna [puppet] - 10https://gerrit.wikimedia.org/r/1348294 (https://phabricator.wikimedia.org/T439916) [06:53:51] (03CR) 10Muehlenhoff: [C:03+2] Remove access for dimakoushna [puppet] - 10https://gerrit.wikimedia.org/r/1348294 (https://phabricator.wikimedia.org/T439916) (owner: 10Muehlenhoff) [06:55:45] !log jmm@cumin2003 DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Dima koushha out of all services on: 2448 hosts [07:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261002T0700) [07:02:45] (03PS2) 10Elukey: sre.hosts.provision: re-add a 10s sleep before checking ipmi [cookbooks] - 10https://gerrit.wikimedia.org/r/1347442 (https://phabricator.wikimedia.org/T426180) [07:02:45] (03PS2) 10Elukey: WIP: sre.hosts.provision: move DHCP configs to the DHCP cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1347466 [07:02:59] (03CR) 10Elukey: sre.hosts.provision: re-add a 10s sleep before checking ipmi (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1347442 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [07:15:55] (03CR) 10Elukey: [C:03+2] sre.hosts.provision: re-add a 10s sleep before checking ipmi [cookbooks] - 10https://gerrit.wikimedia.org/r/1347442 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [07:19:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [07:22:50] (03CR) 10Slyngshede: [C:03+2] Signout: Handle custom OIDC backend for signout [software/bitu] - 10https://gerrit.wikimedia.org/r/1347622 (owner: 10Slyngshede) [07:24:56] (03Merged) 10jenkins-bot: Signout: Handle custom OIDC backend for signout [software/bitu] - 10https://gerrit.wikimedia.org/r/1347622 (owner: 10Slyngshede) [07:25:09] (03Abandoned) 10Giuseppe Lavagetto: mediawiki: add mw.name helper [deployment-charts] - 10https://gerrit.wikimedia.org/r/1079443 (owner: 10Giuseppe Lavagetto) [07:25:15] (03Abandoned) 10Giuseppe Lavagetto: mediawiki: deduplicate network policies and configmaps [deployment-charts] - 10https://gerrit.wikimedia.org/r/1079444 (https://phabricator.wikimedia.org/T376795) (owner: 10Giuseppe Lavagetto) [07:25:22] (03Abandoned) 10Giuseppe Lavagetto: mw-script: deduplicate resources [deployment-charts] - 10https://gerrit.wikimedia.org/r/1079445 (https://phabricator.wikimedia.org/T376795) (owner: 10Giuseppe Lavagetto) [07:26:22] (03Abandoned) 10Giuseppe Lavagetto: mobileapps:update modules [deployment-charts] - 10https://gerrit.wikimedia.org/r/999376 (owner: 10Giuseppe Lavagetto) [07:26:34] (03Abandoned) 10Giuseppe Lavagetto: python-webapp: update module versions [deployment-charts] - 10https://gerrit.wikimedia.org/r/998988 (owner: 10Giuseppe Lavagetto) [07:26:48] (03Abandoned) 10Giuseppe Lavagetto: flink-app: update modules to recent versions [deployment-charts] - 10https://gerrit.wikimedia.org/r/998986 (owner: 10Giuseppe Lavagetto) [07:26:56] (03Abandoned) 10Giuseppe Lavagetto: spark-history: fix package.json, update modules [deployment-charts] - 10https://gerrit.wikimedia.org/r/998989 (owner: 10Giuseppe Lavagetto) [07:27:22] (03Abandoned) 10Giuseppe Lavagetto: modules: add job 1.0.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/969801 (owner: 10Giuseppe Lavagetto) [07:27:28] (03Abandoned) 10Giuseppe Lavagetto: modules: fix app.job [deployment-charts] - 10https://gerrit.wikimedia.org/r/969802 (owner: 10Giuseppe Lavagetto) [07:28:09] (03Abandoned) 10Giuseppe Lavagetto: modules: clean up old unused versions of mesh.configuration [deployment-charts] - 10https://gerrit.wikimedia.org/r/1048359 (owner: 10Giuseppe Lavagetto) [07:32:36] (03PS1) 10Slyngshede: P:idm Add setting to manage sign out requests correctly [puppet] - 10https://gerrit.wikimedia.org/r/1348341 [07:34:13] FIRING: NetworkDeviceAlarmActive: Alarm active on ssw1-f1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=ssw1-f1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [07:37:45] FIRING: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [07:47:55] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) [07:50:30] PROBLEM - Host wikikube-worker1006 is DOWN: PING CRITICAL - Packet loss = 100% [07:52:02] !log oblivian@cumin1004 END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) [07:52:32] (03CR) 10Ayounsi: [C:03+1] "Overall lgtm! one UI nit" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1347593 (https://phabricator.wikimedia.org/T439537) (owner: 10Cathal Mooney) [07:52:52] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) [07:53:18] !log oblivian@cumin1004 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) [07:58:37] !log oblivian@cumin1004 START - Cookbook sre.hosts.provision for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) [07:59:02] FIRING: KubernetesCalicoDown: wikikube-worker1006.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1006.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:02:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [08:04:14] 06SRE-OnFire, 06MediaWiki-Engineering, 06ServiceOps, 06SRE Observability, 07Sustainability (Incident Followup): Reduce the amount of messages sent through channel:Memcached during failures - https://phabricator.wikimedia.org/T390529#12388296 (10MLechvien-WMF) →14Duplicate dup:03T438375 [08:04:58] RECOVERY - Host wikikube-worker1006 is UP: PING OK - Packet loss = 0%, RTA = 0.39 ms [08:07:18] PROBLEM - Host wikikube-worker1006 is DOWN: PING CRITICAL - Packet loss = 100% [08:12:02] (03PS1) 10Gehel: cumin: use non-legacy facts for distribution alias [puppet] - 10https://gerrit.wikimedia.org/r/1348433 (https://phabricator.wikimedia.org/T372666) [08:13:30] RECOVERY - Host wikikube-worker1006 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [08:16:44] oblivian@cumin1004 provision (PID 2314936) is awaiting input [08:19:02] RESOLVED: KubernetesCalicoDown: wikikube-worker1006.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1006.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:26:08] (03CR) 10Elukey: [C:03+1] cumin: use non-legacy facts for distribution alias [puppet] - 10https://gerrit.wikimedia.org/r/1348433 (https://phabricator.wikimedia.org/T372666) (owner: 10Gehel) [08:26:48] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1348433 (https://phabricator.wikimedia.org/T372666) (owner: 10Gehel) [08:27:52] (03CR) 10Gehel: [C:03+2] cumin: use non-legacy facts for distribution alias [puppet] - 10https://gerrit.wikimedia.org/r/1348433 (https://phabricator.wikimedia.org/T372666) (owner: 10Gehel) [08:29:45] (03PS1) 10Brouberol: airlfow-dumps: inscrease the number of tasks selected by the scheduler in a single pass [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348456 [08:29:45] (03PS1) 10Brouberol: mediawiki-dumps-legacy: increase cpu limits quota by 25% [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348457 [08:34:33] (03PS3) 10Elukey: WIP: sre.hosts.provision: move DHCP configs to the DHCP cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1347466 [08:34:33] (03PS1) 10Elukey: sre.hosts.reimage: move ipmi/redfish to wmfroot [cookbooks] - 10https://gerrit.wikimedia.org/r/1348465 (https://phabricator.wikimedia.org/T426180) [08:36:34] (03CR) 10Btullis: [C:03+1] mediawiki-dumps-legacy: increase cpu limits quota by 25% [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348457 (owner: 10Brouberol) [08:36:48] (03CR) 10Btullis: [C:03+1] airlfow-dumps: inscrease the number of tasks selected by the scheduler in a single pass [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348456 (owner: 10Brouberol) [08:37:17] (03PS3) 10Muehlenhoff: Open a Phab task if Ganeti CA expires soon [alerts] - 10https://gerrit.wikimedia.org/r/1345966 (https://phabricator.wikimedia.org/T439440) [08:39:51] (03CR) 10Brouberol: [C:03+2] airlfow-dumps: inscrease the number of tasks selected by the scheduler in a single pass [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348456 (owner: 10Brouberol) [08:41:16] (03CR) 10Brouberol: [C:03+2] mediawiki-dumps-legacy: increase cpu limits quota by 25% [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348457 (owner: 10Brouberol) [08:42:08] (03Merged) 10jenkins-bot: airlfow-dumps: inscrease the number of tasks selected by the scheduler in a single pass [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348456 (owner: 10Brouberol) [08:43:24] (03CR) 10Brouberol: [V:03+2 C:03+2] mediawiki-dumps-legacy: increase cpu limits quota by 25% [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348457 (owner: 10Brouberol) [08:45:53] (03PS3) 10Btullis: kubernetes: Add an option to select the service account key type [puppet] - 10https://gerrit.wikimedia.org/r/1347693 (https://phabricator.wikimedia.org/T439865) [08:45:53] (03PS3) 10Btullis: dse-k8s-codfw: Sign service account tokens with an RSA key [puppet] - 10https://gerrit.wikimedia.org/r/1347694 (https://phabricator.wikimedia.org/T439865) [08:46:13] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1347693 (https://phabricator.wikimedia.org/T439865) (owner: 10Btullis) [08:47:08] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [08:47:39] (03CR) 10Btullis: "This should be a noop for all clusters, until activated for dse-k8s-codfw by the next patch in the stack." [puppet] - 10https://gerrit.wikimedia.org/r/1347693 (https://phabricator.wikimedia.org/T439865) (owner: 10Btullis) [08:48:16] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [08:48:47] (03CR) 10CI reject: [V:04-1] kubernetes: Add an option to select the service account key type [puppet] - 10https://gerrit.wikimedia.org/r/1347693 (https://phabricator.wikimedia.org/T439865) (owner: 10Btullis) [08:48:51] !log brouberol@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [08:49:14] (03PS1) 10Ayounsi: drain eqsin-codfw HE link [homer/public] - 10https://gerrit.wikimedia.org/r/1348470 (https://phabricator.wikimedia.org/T439947) [08:49:48] !log brouberol@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [08:50:43] (03PS4) 10Btullis: kubernetes: Add an option to select the service account key type [puppet] - 10https://gerrit.wikimedia.org/r/1347693 (https://phabricator.wikimedia.org/T439865) [08:50:44] (03PS4) 10Btullis: dse-k8s-codfw: Sign service account tokens with an RSA key [puppet] - 10https://gerrit.wikimedia.org/r/1347694 (https://phabricator.wikimedia.org/T439865) [08:51:21] (03PS1) 10Ayounsi: un-drain esams-codfw HE link [homer/public] - 10https://gerrit.wikimedia.org/r/1348471 [08:53:49] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1347693 (https://phabricator.wikimedia.org/T439865) (owner: 10Btullis) [08:54:11] (03PS1) 10Ayounsi: Un-drain magru-codfw link [homer/public] - 10https://gerrit.wikimedia.org/r/1348476 [08:54:25] (03CR) 10Ayounsi: [C:03+2] drain eqsin-codfw HE link [homer/public] - 10https://gerrit.wikimedia.org/r/1348470 (https://phabricator.wikimedia.org/T439947) (owner: 10Ayounsi) [08:55:24] (03Merged) 10jenkins-bot: drain eqsin-codfw HE link [homer/public] - 10https://gerrit.wikimedia.org/r/1348470 (https://phabricator.wikimedia.org/T439947) (owner: 10Ayounsi) [08:56:09] !log elukey@cumin1004 START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [08:56:53] (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1347694 (https://phabricator.wikimedia.org/T439865) (owner: 10Btullis) [08:57:38] PROBLEM - Host sretest2001 is DOWN: PING CRITICAL - Packet loss = 100% [09:00:20] RECOVERY - Host sretest2001 is UP: PING OK - Packet loss = 0%, RTA = 30.33 ms [09:00:36] (03PS4) 10Elukey: WIP: sre.hosts.provision: move DHCP configs to the DHCP cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1347466 [09:03:52] PROBLEM - Host sretest2001 is DOWN: PING CRITICAL - Packet loss = 100% [09:04:30] (03CR) 10Ayounsi: [C:03+1] sre.hosts.reimage: move ipmi/redfish to wmfroot [cookbooks] - 10https://gerrit.wikimedia.org/r/1348465 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [09:05:20] RECOVERY - Host sretest2001 is UP: PING OK - Packet loss = 0%, RTA = 30.35 ms [09:06:26] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [09:06:38] (03CR) 10Klausman: [C:03+1] dse-k8s-codfw: Sign service account tokens with an RSA key [puppet] - 10https://gerrit.wikimedia.org/r/1347694 (https://phabricator.wikimedia.org/T439865) (owner: 10Btullis) [09:06:48] (03CR) 10Klausman: [C:03+1] kubernetes: Add an option to select the service account key type [puppet] - 10https://gerrit.wikimedia.org/r/1347693 (https://phabricator.wikimedia.org/T439865) (owner: 10Btullis) [09:16:26] (03PS1) 10Slyngshede: Signout: Fix variable name [software/bitu] - 10https://gerrit.wikimedia.org/r/1348491 [09:16:55] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:26:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.333% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [09:29:18] (03PS5) 10Elukey: sre.hosts.provision: move DHCP configs to the DHCP cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1347466 [09:30:56] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [09:31:49] (03CR) 10Slyngshede: [C:03+2] Signout: Fix variable name [software/bitu] - 10https://gerrit.wikimedia.org/r/1348491 (owner: 10Slyngshede) [09:34:50] (03Merged) 10jenkins-bot: Signout: Fix variable name [software/bitu] - 10https://gerrit.wikimedia.org/r/1348491 (owner: 10Slyngshede) [09:36:21] btullis@cumin1004 netbox (PID 2336355) is awaiting input [09:41:57] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004" [09:44:26] PROBLEM - Host wikikube-worker1006 is DOWN: PING CRITICAL - Packet loss = 100% [09:45:00] btullis@cumin1004 netbox (PID 2336355) is awaiting input [09:45:36] (03PS1) 10Dpogorzelski: wikimedia.org: rename liftwing-studio to hummingbird [dns] - 10https://gerrit.wikimedia.org/r/1348507 (https://phabricator.wikimedia.org/T439416) [09:45:43] (03PS1) 10Dpogorzelski: liftwing-studio: serve on hummingbird.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1348508 (https://phabricator.wikimedia.org/T439416) [09:45:44] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-commissioning ml-cache2001 - btullis@cumin1004" [09:45:45] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [09:45:51] (03PS1) 10Dpogorzelski: liftwing-studio: serve on hummingbird.wikimedia.org [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348509 (https://phabricator.wikimedia.org/T439416) [09:46:50] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [09:49:02] FIRING: KubernetesCalicoDown: wikikube-worker1006.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1006.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [09:51:58] RECOVERY - Host wikikube-worker1006 is UP: PING OK - Packet loss = 0%, RTA = 0.40 ms [09:53:34] !log oblivian@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART (NO reboot will be performed automatically) [09:59:02] RESOLVED: KubernetesCalicoDown: wikikube-worker1006.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1006.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:01:05] (03PS3) 10Muehlenhoff: Add conftool-data for ldap-ro-test [puppet] - 10https://gerrit.wikimedia.org/r/1347474 (https://phabricator.wikimedia.org/T439730) [10:01:27] (03CR) 10Muehlenhoff: "Doh, that was typo. Fixed." [puppet] - 10https://gerrit.wikimedia.org/r/1347474 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [10:07:33] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Essential-Work: mc20[56-73] implementation tracking - https://phabricator.wikimedia.org/T436273#12388584 (10MLechvien-WMF) [10:07:38] 06SRE, 06ServiceOps, 10ServiceOps-Upgrades-Hardware, 07Essential-Work: mc20[56-73] implementation tracking - https://phabricator.wikimedia.org/T436273#12388586 (10MLechvien-WMF) p:05Medium→03High [10:09:17] (03PS1) 10Dpogorzelski: kserve-llm-inference: bump chart to 0.1.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348529 (https://phabricator.wikimedia.org/T439659) [10:09:45] (03CR) 10Dpogorzelski: [C:03+2] kserve-llm-inference: bump chart to 0.1.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348529 (https://phabricator.wikimedia.org/T439659) (owner: 10Dpogorzelski) [10:09:47] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] kserve-llm-inference: bump chart to 0.1.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348529 (https://phabricator.wikimedia.org/T439659) (owner: 10Dpogorzelski) [10:11:13] !log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [10:11:29] !log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [10:12:13] !log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [10:12:58] !log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [10:14:25] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [10:19:09] btullis@cumin1004 provision (PID 2344161) is awaiting input [10:22:22] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [10:23:04] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host ml-cache2001.codfw.wmnet with OS bookworm [10:27:33] (03PS1) 10Muehlenhoff: Remove cluster::unprivmanagement role from cuminunpriv1001 [puppet] - 10https://gerrit.wikimedia.org/r/1348538 (https://phabricator.wikimedia.org/T439522) [10:29:00] !log installing Linux 6.12.111 on Trixie hosts [10:29:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:31:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:33:38] 06SRE, 10hCaptcha, 06Product Safety and Integrity, 06ServiceOps, and 2 others: memcached errors seen in hCaptcha health checks - https://phabricator.wikimedia.org/T430340#12388742 (10MLechvien-WMF) [10:34:02] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [10:36:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:39:30] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage [10:39:34] (03PS1) 10Blake: mathoid: replace url-downloader cname with discovery name. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348555 (https://phabricator.wikimedia.org/T439713) [10:42:23] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2001.codfw.wmnet with reason: host reimage [10:43:16] (03CR) 10Arthur taylor: Enable mobile editing for all Wikidata editors (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1347373 (https://phabricator.wikimedia.org/T437054) (owner: 10Sadiya.mohammed13) [10:45:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:47:09] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [10:48:22] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-c6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:50:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:52:31] btullis@cumin1004 netbox (PID 2348020) is awaiting input [10:53:54] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.6 point update - https://phabricator.wikimedia.org/T434866#12388808 (10MoritzMuehlenhoff) [10:54:56] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" [10:54:59] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" [10:54:59] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [10:57:03] (03CR) 10Kamila Součková: [C:03+1] sidecars: remove pre-stop sleep, stop envoy last [deployment-charts] - 10https://gerrit.wikimedia.org/r/1346502 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:58:29] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" [11:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261002T0700) [11:00:05] jelto, arnoldokoth, mutante, and arnaudb: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for GitLab version upgrades deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261002T1100). [11:01:32] btullis@cumin1004 reimage (PID 2344512) is awaiting input [11:03:01] (03CR) 10Btullis: [C:03+1] "There's an option to slim down the script since it dates from the pre-airflow days when there were snapshot, dumpsdata, and clouddumps ser" [puppet] - 10https://gerrit.wikimedia.org/r/1346752 (https://phabricator.wikimedia.org/T437463) (owner: 10Xcollazo) [11:08:08] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" [11:08:08] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2001.codfw.wmnet with OS bookworm [11:26:44] (03CR) 10Elukey: [C:03+2] sre.hosts.reimage: move ipmi/redfish to wmfroot [cookbooks] - 10https://gerrit.wikimedia.org/r/1348465 (https://phabricator.wikimedia.org/T426180) (owner: 10Elukey) [11:30:10] (03CR) 10Elukey: [C:03+1] Open a Phab task if Ganeti CA expires soon [alerts] - 10https://gerrit.wikimedia.org/r/1345966 (https://phabricator.wikimedia.org/T439440) (owner: 10Muehlenhoff) [11:34:07] (03CR) 10Elukey: "Proposal to simplify the logic - I'd simply use a boolean flag called something like "use_next_api = False" or similar, and then use the n" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [11:34:13] FIRING: NetworkDeviceAlarmActive: Alarm active on ssw1-f1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=ssw1-f1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [11:36:30] (03CR) 10Elukey: "All right I see:" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1347450 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [11:37:33] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [11:40:07] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [11:40:23] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [11:45:02] btullis@cumin1004 provision (PID 2358999) is awaiting input [11:50:16] (03CR) 10Muehlenhoff: [C:03+2] Open a Phab task if Ganeti CA expires soon [alerts] - 10https://gerrit.wikimedia.org/r/1345966 (https://phabricator.wikimedia.org/T439440) (owner: 10Muehlenhoff) [11:51:28] (03CR) 10Muehlenhoff: [C:03+2] Remove cluster::unprivmanagement role from cuminunpriv1001 [puppet] - 10https://gerrit.wikimedia.org/r/1348538 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [11:55:06] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1348341 (owner: 10Slyngshede) [11:56:48] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [11:58:43] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [11:59:16] (03PS1) 10Mark Bergsma: Replace ssh key by 2 Yubikey FIDO2 backed keys for user mark [puppet] - 10https://gerrit.wikimedia.org/r/1348637 [12:00:00] PROBLEM - Host wikikube-worker1016 is DOWN: PING CRITICAL - Packet loss = 50%, RTA = 2978.50 ms [12:00:35] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [12:00:38] RECOVERY - Host wikikube-worker1016 is UP: PING OK - Packet loss = 0%, RTA = 0.39 ms [12:00:48] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host ml-cache2002.codfw.wmnet with OS bookworm [12:01:07] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [12:06:31] btullis@cumin1004 netbox (PID 2361836) is awaiting input [12:10:03] (03PS6) 10Elukey: sre.hosts.provision: move DHCP configs to the DHCP cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1347466 [12:10:55] (03PS7) 10Elukey: sre.hosts.provision: move DHCP configs to the DHCP cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1347466 [12:11:38] !log elukey@cumin1004 START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [12:11:42] !log elukey@cumin1004 END (ERROR) - Cookbook sre.hosts.provision (exit_code=97) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [12:13:53] PROBLEM - Host sretest2001 is DOWN: PING CRITICAL - Packet loss = 100% [12:14:24] (03PS8) 10Elukey: sre.hosts.provision: move DHCP configs to the DHCP cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1347466 [12:14:35] !log elukey@cumin1004 START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [12:14:37] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (push config only) [12:14:42] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (push config only) [12:15:50] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" [12:15:53] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Re-adding ml-cache2002 - btullis@cumin1004" [12:15:53] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:16:55] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage [12:17:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [12:18:21] RECOVERY - Host sretest2001 is UP: PING OK - Packet loss = 0%, RTA = 30.44 ms [12:19:48] !log elukey@cumin1004 START - Cookbook sre.hosts.dhcp for host sretest2001.codfw.wmnet (remove config only) [12:19:53] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest2001.codfw.wmnet (remove config only) [12:20:03] !log elukey@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [12:20:27] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2002.codfw.wmnet with reason: host reimage [12:21:15] (03CR) 10Elukey: "Tested on sretest2001, both provisioning and dhcp alone. Please give it a go!" [cookbooks] - 10https://gerrit.wikimedia.org/r/1347466 (owner: 10Elukey) [12:22:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [12:27:17] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [12:28:13] (03PS1) 10Ayounsi: spicerack config: make netbox URLs their own dict [puppet] - 10https://gerrit.wikimedia.org/r/1348660 (https://phabricator.wikimedia.org/T439408) [12:28:20] (03CR) 10Ayounsi: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1348660 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [12:31:08] btullis@cumin1004 provision (PID 2365717) is awaiting input [12:31:40] (03CR) 10Giuseppe Lavagetto: [C:03+1] "Verified in a video call" [puppet] - 10https://gerrit.wikimedia.org/r/1348637 (owner: 10Mark Bergsma) [12:36:36] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" [12:39:39] btullis@cumin1004 reimage (PID 2361790) is awaiting input [12:49:56] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [12:51:45] (03CR) 10Muehlenhoff: [C:03+1] "Looks great, one nit inline" [puppet] - 10https://gerrit.wikimedia.org/r/1319486 (https://phabricator.wikimedia.org/T433601) (owner: 10Cathal Mooney) [13:01:55] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:04:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [13:05:59] (03CR) 10Muehlenhoff: "Since this just cleans up a comment, could you please also do the same for charts/chromium-render/templates/_config.yaml, charts/mobileapp" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348555 (https://phabricator.wikimedia.org/T439713) (owner: 10Blake) [13:09:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [13:15:03] (03CR) 10Slyngshede: [C:03+1] "Verified via Slack" [puppet] - 10https://gerrit.wikimedia.org/r/1347704 (owner: 10Ssingh) [13:17:31] (03CR) 10Ssingh: [C:03+2] admin: add backup Yubikey for sukhe [puppet] - 10https://gerrit.wikimedia.org/r/1347704 (owner: 10Ssingh) [13:22:32] (03PS2) 10Hashar: cinderutils: honor mount_options when mountpoints exist [puppet] - 10https://gerrit.wikimedia.org/r/1343081 [13:22:32] (03PS2) 10Hashar: cinderutils: support appending mount options [puppet] - 10https://gerrit.wikimedia.org/r/1343082 [13:22:32] (03PS2) 10Hashar: ci: set noatime on castor /srv partition [puppet] - 10https://gerrit.wikimedia.org/r/1343083 [13:23:54] (03CR) 10CI reject: [V:04-1] cinderutils: support appending mount options [puppet] - 10https://gerrit.wikimedia.org/r/1343082 (owner: 10Hashar) [13:35:12] (03CR) 10Muehlenhoff: [C:03+1] "Looks good. I compared all templates against the default files from Trixie and all puppetisation changes are re-applied." [puppet] - 10https://gerrit.wikimedia.org/r/1345921 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [13:35:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [13:36:53] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [13:40:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [13:42:57] (03PS1) 10Hashar: Gemfile: bump rake and add ruby 3.3 gems [puppet] - 10https://gerrit.wikimedia.org/r/1348698 [13:44:15] (03CR) 10CI reject: [V:04-1] Gemfile: bump rake and add ruby 3.3 gems [puppet] - 10https://gerrit.wikimedia.org/r/1348698 (owner: 10Hashar) [13:44:54] (03PS2) 10Hashar: Gemfile: bump rake and add ruby 3.3 gems [puppet] - 10https://gerrit.wikimedia.org/r/1348698 [13:45:54] 06SRE, 10Infrastructure Security, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07SecTeam-Processed, and 2 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12389383 (10klausman) [13:46:50] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [13:48:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [13:49:52] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" [13:49:52] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2002.codfw.wmnet with OS bookworm [13:50:36] (03PS1) 10Jforrester: abstractwiki-rust: Bump version number; trixie-backports has upgraded rustc etc. from 1.94.1 to 1.95.0 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1348701 [13:51:53] (03CR) 10Jforrester: "Would be good to get this in before the weekly re-build." [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1348701 (owner: 10Jforrester) [13:53:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [13:58:28] (03CR) 10Muehlenhoff: [C:03+1] "Looks good!" [puppet] - 10https://gerrit.wikimedia.org/r/1346531 (https://phabricator.wikimedia.org/T439509) (owner: 10Slyngshede) [13:58:41] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [14:03:00] (03CR) 10JHathaway: [C:03+1] "looks good" [cookbooks] - 10https://gerrit.wikimedia.org/r/1346499 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [14:03:11] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [14:03:25] (03CR) 10Muehlenhoff: Add service catalogue entry for ldap-ro-test (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1347624 (https://phabricator.wikimedia.org/T439730) (owner: 10Muehlenhoff) [14:03:27] (03PS2) 10Muehlenhoff: Add service catalogue entry for ldap-ro-test [puppet] - 10https://gerrit.wikimedia.org/r/1347624 (https://phabricator.wikimedia.org/T439730) [14:04:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [14:08:45] (03PS1) 10Hashar: rubocop: raise target ruby: 2.3..2.7 [puppet] - 10https://gerrit.wikimedia.org/r/1348717 [14:09:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [14:10:42] !log btullis@cumin1004 START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. [14:12:53] (03PS2) 10Hashar: rubocop: raise target ruby: 2.3..2.7 [puppet] - 10https://gerrit.wikimedia.org/r/1348717 [14:13:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [14:14:18] (03CR) 10Ssingh: [C:03+1] wikimedia.org: rename liftwing-studio to hummingbird [dns] - 10https://gerrit.wikimedia.org/r/1348507 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [14:14:19] (03PS1) 10Muehlenhoff: Remove profile::base::cuminunpriv from installservers [puppet] - 10https://gerrit.wikimedia.org/r/1348724 (https://phabricator.wikimedia.org/T439522) [14:15:47] (03PS1) 10Elukey: sre.hosts.reimage: refactor DHCP configs as cookbook calls [cookbooks] - 10https://gerrit.wikimedia.org/r/1348725 [14:15:55] (03CR) 10Ssingh: "Needs a review from someone in serviceops, but looks good as far as renaming goes." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348509 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [14:16:23] (03CR) 10CI reject: [V:04-1] rubocop: raise target ruby: 2.3..2.7 [puppet] - 10https://gerrit.wikimedia.org/r/1348717 (owner: 10Hashar) [14:17:04] !log btullis@cumin1004 END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-analytics cluster: Roll restart of jvm daemons. [14:17:37] (03CR) 10Ssingh: [C:03+1] liftwing-studio: serve on hummingbird.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1348508 (https://phabricator.wikimedia.org/T439416) (owner: 10Dpogorzelski) [14:18:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [14:20:11] (03PS2) 10Scott French: Actually increment the replicated_events counter [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1310234 [14:20:11] (03CR) 10Scott French: "@glavagetto@wikimedia.org - This is not urgent in any way, but I wanted to get it on your radar in case we ever want to pursue it in the f" [software/etcd-mirror] - 10https://gerrit.wikimedia.org/r/1310234 (owner: 10Scott French) [14:23:09] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [14:23:26] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:25:10] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host an-conf1004.eqiad.wmnet [14:25:30] (03CR) 10Ssingh: [C:03+1] P:tofurkey enable Tofurkey for DRMRS [puppet] - 10https://gerrit.wikimedia.org/r/1345923 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [14:25:50] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ml-cache2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [14:25:58] (03CR) 10Ssingh: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345923 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [14:27:03] (03PS2) 10Elukey: sre.hosts.reimage: refactor DHCP configs as cookbook calls [cookbooks] - 10https://gerrit.wikimedia.org/r/1348725 [14:27:56] (03PS1) 10Blake: mcrouter: switch updateStrategy to OnDelete [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348640 (https://phabricator.wikimedia.org/T420223) [14:28:16] (03PS1) 10Blake: mcrouter: test connection reset/keepalive options [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348641 (https://phabricator.wikimedia.org/T420223) [14:28:24] (03PS2) 10Ssingh: P:tofurkey enable Tofurkey for DRMRS [puppet] - 10https://gerrit.wikimedia.org/r/1345923 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [14:29:38] (03CR) 10Andrew Bogott: "Does listening on a floating IP work here? I would expect haproxy to see the traffic come in on the private vip, 172.16.16.6, and be basic" [puppet] - 10https://gerrit.wikimedia.org/r/1346602 (https://phabricator.wikimedia.org/T420921) (owner: 10Majavah) [14:30:03] (03PS3) 10Elukey: sre.hosts.reimage: refactor DHCP configs as cookbook calls [cookbooks] - 10https://gerrit.wikimedia.org/r/1348725 [14:30:25] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1004.eqiad.wmnet [14:30:41] (03CR) 10Majavah: [V:03+1] "how are openstack floating IPs involved here?" [puppet] - 10https://gerrit.wikimedia.org/r/1346602 (https://phabricator.wikimedia.org/T420921) (owner: 10Majavah) [14:31:21] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1348724 (https://phabricator.wikimedia.org/T439522) (owner: 10Muehlenhoff) [14:31:47] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm [14:32:06] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ml-cache2003.codfw.wmnet with OS bookworm [14:32:08] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host an-conf1005.eqiad.wmnet [14:32:13] (03CR) 10Andrew Bogott: [C:03+1] "Because I am 100% thinking about the wrong proxy. Nevermind!" [puppet] - 10https://gerrit.wikimedia.org/r/1346602 (https://phabricator.wikimedia.org/T420921) (owner: 10Majavah) [14:32:15] (03PS2) 10Blake: various: replace url-downloader cname. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348555 (https://phabricator.wikimedia.org/T439713) [14:32:18] (03CR) 10Ssingh: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345923 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [14:32:19] (03CR) 10Eevans: [C:03+1] abstractwiki-rust: Bump version number; trixie-backports has upgraded rustc etc. from 1.94.1 to 1.95.0 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1348701 (owner: 10Jforrester) [14:32:34] (03CR) 10Blake: "Added!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348555 (https://phabricator.wikimedia.org/T439713) (owner: 10Blake) [14:33:12] (03CR) 10Majavah: [V:03+1 C:03+2] hieradata: cloudlb: Bind all codfw1dev services to specific IPs [puppet] - 10https://gerrit.wikimedia.org/r/1346602 (https://phabricator.wikimedia.org/T420921) (owner: 10Majavah) [14:34:02] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [14:34:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [14:34:26] (03CR) 10Andrew Bogott: [C:03+1] cloudlb: Update LABS_NETWORKS to CLOUD_NETWORKS [puppet] - 10https://gerrit.wikimedia.org/r/1346604 (owner: 10Majavah) [14:35:02] (03CR) 10Elukey: [C:03+1] "@blake@wikimedia.org Thanks for working on this!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348641 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake) [14:35:59] (03CR) 10Majavah: [C:03+2] cloudlb: Update LABS_NETWORKS to CLOUD_NETWORKS [puppet] - 10https://gerrit.wikimedia.org/r/1346604 (owner: 10Majavah) [14:36:26] (03CR) 10Ssingh: [C:03+1] "no code change, updated commit message to specify the correct hostname (cp6004.drmrs.wmnet instead of cp6004.magru.wmnet) which was causin" [puppet] - 10https://gerrit.wikimedia.org/r/1345923 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [14:37:26] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1005.eqiad.wmnet [14:37:53] (03CR) 10Muehlenhoff: [C:03+1] "Thanks!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348555 (https://phabricator.wikimedia.org/T439713) (owner: 10Blake) [14:37:57] (03CR) 10Andrew Bogott: [C:03+1] P:wcms::cloudvps_meta: Publish data about infra service IPs [puppet] - 10https://gerrit.wikimedia.org/r/1343545 (owner: 10Majavah) [14:38:30] (03CR) 10Majavah: [V:03+1 C:03+2] P:wcms::cloudvps_meta: Publish data about infra service IPs [puppet] - 10https://gerrit.wikimedia.org/r/1343545 (owner: 10Majavah) [14:39:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [14:39:35] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host an-conf1006.eqiad.wmnet [14:41:44] (03CR) 10Hashar: "`" [puppet] - 10https://gerrit.wikimedia.org/r/1348717 (owner: 10Hashar) [14:43:33] (03PS3) 10Hashar: rake_modules: run rubocop when .rubocop.yml changes [puppet] - 10https://gerrit.wikimedia.org/r/1348717 [14:44:18] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host ml-cache2003.codfw.wmnet with OS bookworm [14:44:20] (03PS1) 10Majavah: P:wmcs::cloudvps_meta: Fix variable copy-paste [puppet] - 10https://gerrit.wikimedia.org/r/1348746 [14:44:31] (03CR) 10Hashar: "Hi Jesse, no rush but those few changes would let me run some rspec directly 😊" [puppet] - 10https://gerrit.wikimedia.org/r/1348698 (owner: 10Hashar) [14:45:04] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-conf1006.eqiad.wmnet [14:45:44] (03CR) 10Hashar: "Spotted in patchset 1 which touched `.rubocop.yml` but did not cause a run of rubocop." [puppet] - 10https://gerrit.wikimedia.org/r/1348717 (owner: 10Hashar) [14:47:15] !log btullis@cumin1004 START - Cookbook sre.druid.reboot-workers for Druid analytics cluster: Reboot Druid nodes [14:48:13] (03CR) 10Majavah: [C:03+2] P:wmcs::cloudvps_meta: Fix variable copy-paste [puppet] - 10https://gerrit.wikimedia.org/r/1348746 (owner: 10Majavah) [14:48:23] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-c6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:56:50] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [14:56:54] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [15:00:45] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage [15:01:29] (03CR) 10JHathaway: [C:03+1] rake_modules: run rubocop when .rubocop.yml changes [puppet] - 10https://gerrit.wikimedia.org/r/1348717 (owner: 10Hashar) [15:01:59] (03CR) 10JHathaway: [C:03+1] Gemfile: bump rake and add ruby 3.3 gems [puppet] - 10https://gerrit.wikimedia.org/r/1348698 (owner: 10Hashar) [15:03:30] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-cache2003.codfw.wmnet with reason: host reimage [15:04:46] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [15:13:22] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.7 point update - https://phabricator.wikimedia.org/T437715#12389954 (10MoritzMuehlenhoff) [15:20:04] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" [15:21:31] (03PS1) 10Aqu: airflow: add CephFS PVCs dev-metrics-git-pipeline [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348762 (https://phabricator.wikimedia.org/T433696) [15:21:41] (03CR) 10CI reject: [V:04-1] airflow: add CephFS PVCs dev-metrics-git-pipeline [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348762 (https://phabricator.wikimedia.org/T433696) (owner: 10Aqu) [15:21:47] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - btullis@cumin1004" [15:21:47] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-cache2003.codfw.wmnet with OS bookworm [15:23:04] (03PS2) 10Aqu: airflow: add CephFS PVCs dev-metrics-git-pipeline [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348762 (https://phabricator.wikimedia.org/T433696) [15:24:33] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 10Observability-Logging: decommission kafka-logging100[1-3].eqiad.wmnet - https://phabricator.wikimedia.org/T439559#12390033 (10Jclark-ctr) [15:34:13] FIRING: NetworkDeviceAlarmActive: Alarm active on ssw1-f1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=ssw1-f1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [15:40:56] (03PS1) 10Btullis: dse-k8s-codfw: Add the renamed ml-cache machines to site and preseed [puppet] - 10https://gerrit.wikimedia.org/r/1348777 (https://phabricator.wikimedia.org/T430654) [15:42:20] 06SRE, 06Commons, 10MediaWiki-File-management, 06Traffic, and 3 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12390120 (10Krinkle) 05Open→03Resolved [15:42:42] (03PS2) 10Btullis: dse-k8s-codfw: Add the renamed ml-cache machines to site and preseed [puppet] - 10https://gerrit.wikimedia.org/r/1348777 (https://phabricator.wikimedia.org/T430654) [15:44:40] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm [15:45:44] !log bking@cumin2003 END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) [15:47:34] !log btullis@cumin1004 END (ERROR) - Cookbook sre.druid.reboot-workers (exit_code=97) for Druid analytics cluster: Reboot Druid nodes [15:48:06] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host an-test-client1002.eqiad.wmnet [15:51:27] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [15:52:15] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-test-client1002.eqiad.wmnet [15:52:56] (03CR) 10Bking: [C:03+1] dse-k8s-codfw: Add the renamed ml-cache machines to site and preseed [puppet] - 10https://gerrit.wikimedia.org/r/1348777 (https://phabricator.wikimedia.org/T430654) (owner: 10Btullis) [15:53:26] (03CR) 10Btullis: [C:03+2] dse-k8s-codfw: Add the renamed ml-cache machines to site and preseed [puppet] - 10https://gerrit.wikimedia.org/r/1348777 (https://phabricator.wikimedia.org/T430654) (owner: 10Btullis) [15:55:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:56:59] !log btullis@cumin1004 START - Cookbook sre.hosts.rename from ml-cache2001 to dse-k8s-worker2009 [15:57:22] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [16:00:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [16:01:12] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004" [16:01:19] btullis@cumin1004 rename (PID 2400946) is awaiting input [16:02:43] !log btullis@cumin1004 START - Cookbook sre.hosts.rename from ml-cache2002 to dse-k8s-worker2010 [16:02:50] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [16:02:54] !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply [16:02:56] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2001 to dse-k8s-worker2009 - btullis@cumin1004" [16:02:56] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [16:02:56] !log btullis@cumin1004 START - Cookbook sre.dns.wipe-cache dse-k8s-worker2009 on all recursors [16:03:00] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2009 on all recursors [16:03:00] !log btullis@cumin1004 START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2009 [16:03:07] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [16:03:12] !log btullis@cumin1004 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2009 [16:03:47] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2001 to dse-k8s-worker2009 [16:04:03] !log btullis@cumin1004 START - Cookbook sre.hosts.rename from ml-cache2003 to dse-k8s-worker2011 [16:05:15] !log btullis@cumin1004 END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [16:05:15] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [16:07:30] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host dse-k8s-worker2009.codfw.wmnet with OS bookworm [16:09:01] !log btullis@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004" [16:09:11] (03PS1) 10Hashar: rubocop: enable Lint/UnneededDisable [puppet] - 10https://gerrit.wikimedia.org/r/1348795 [16:09:11] (03PS1) 10Hashar: graphite: define class method with . instead of :: [puppet] - 10https://gerrit.wikimedia.org/r/1348796 [16:09:11] (03PS1) 10Hashar: rubocop: upgrade and target ruby 2.7 [puppet] - 10https://gerrit.wikimedia.org/r/1348797 [16:10:35] (03PS8) 10Ahmon Dancy: puppetserver::volatile: Do not include profile::puppetserver [puppet] - 10https://gerrit.wikimedia.org/r/1137013 (owner: 10Krinkle) [16:10:40] !log btullis@cumin1004 START - Cookbook sre.dns.netbox [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:11:43] (03CR) 10Ahmon Dancy: [V:03+1] "Jesse, are you okay with this change? It is working in beta cluster." [puppet] - 10https://gerrit.wikimedia.org/r/1137013 (owner: 10Krinkle) [16:12:04] btullis@cumin1004 rename (PID 2401546) is awaiting input [16:13:05] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [16:13:05] !log btullis@cumin1004 START - Cookbook sre.dns.wipe-cache dse-k8s-worker2010 on all recursors [16:13:08] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2010 on all recursors [16:13:09] !log btullis@cumin1004 START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2010 [16:16:13] btullis@cumin1004 rename (PID 2400946) is awaiting input [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:21:24] (03CR) 10Dzahn: [C:03+2] wikistats: update deploy script, fix xmldump and small fixes [puppet] - 10https://gerrit.wikimedia.org/r/1347785 (https://phabricator.wikimedia.org/T409014) (owner: 10Dzahn) [16:21:38] !log btullis@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming ml-cache2003 to dse-k8s-worker2011 - btullis@cumin1004" [16:21:38] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [16:21:39] !log btullis@cumin1004 START - Cookbook sre.dns.wipe-cache dse-k8s-worker2011 on all recursors [16:21:40] !log btullis@cumin1004 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2010 [16:21:42] !log btullis@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2011 on all recursors [16:21:42] !log btullis@cumin1004 START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2011 [16:21:53] (03PS1) 10Kevin Bazira: ml-services: update tts-section-generator service to speak "m" and "mm" as meters and millimeters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348809 (https://phabricator.wikimedia.org/T440001) [16:21:59] !log btullis@cumin1004 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2011 [16:22:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [16:22:15] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2002 to dse-k8s-worker2010 [16:22:34] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from ml-cache2003 to dse-k8s-worker2011 [16:24:20] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage [16:25:04] (03CR) 10Kevin Bazira: [C:03+2] ml-services: update tts-section-generator service to speak "m" and "mm" as meters and millimeters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348809 (https://phabricator.wikimedia.org/T440001) (owner: 10Kevin Bazira) [16:25:08] (03CR) 10Dzahn: [C:03+1] gerrit: remove a unit copy left by systemctl edit --full [puppet] - 10https://gerrit.wikimedia.org/r/1345040 (https://phabricator.wikimedia.org/T439209) (owner: 10Arnaudb) [16:27:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [16:27:32] (03Merged) 10jenkins-bot: ml-services: update tts-section-generator service to speak "m" and "mm" as meters and millimeters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348809 (https://phabricator.wikimedia.org/T440001) (owner: 10Kevin Bazira) [16:28:04] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2009.codfw.wmnet with reason: host reimage [16:28:43] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [16:29:32] (03CR) 10Dzahn: [C:03+1] "all 3 services, zuul, zuul-merger and git-daemon have systemd units and it's a valid point that having systemd monitoring already makes th" [puppet] - 10https://gerrit.wikimedia.org/r/1343955 (https://phabricator.wikimedia.org/T384939) (owner: 10Hnowlan) [16:31:28] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . [16:33:09] !log kevinbazira@deploy1003 helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [16:40:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [16:40:15] (03PS4) 10Dzahn: zuul: decom VM zuul1002, replaced by zuul1005, keep zuul2002 [puppet] - 10https://gerrit.wikimedia.org/r/1346164 (https://phabricator.wikimedia.org/T427353) [16:44:17] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2009.codfw.wmnet with OS bookworm [16:45:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [16:53:57] FIRING: ProbeDown: Service mw-api-ext-next:4455 has failed probes (http_mw-api-ext-next_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext-next:4455 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:58:57] RESOLVED: ProbeDown: Service mw-api-ext-next:4455 has failed probes (http_mw-api-ext-next_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext-next:4455 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:07:30] btullis@cumin1004 reimage (PID 2400035) is awaiting input [17:11:34] (03PS1) 10Ilias Sarantopoulos: liftwing-studio: add Qwen3.8 to list of models [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348836 [17:18:24] 10ops-eqiad, 06SRE, 06DC-Ops, 10ServiceOps-Upgrades-Hardware, 06ServiceOps (Next quarter): Q1:rack/setup/install conf101[0-2] - https://phabricator.wikimedia.org/T435426#12390669 (10Scott_French) [17:23:25] (03CR) 10Andrew Bogott: [C:03+2] openstack apis: Add additional uwsgi ini file to support logstash logging [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [17:23:30] (03CR) 10Andrew Bogott: [C:03+2] Revert "Puppetize keystone-uwsgi.ini" [puppet] - 10https://gerrit.wikimedia.org/r/1344333 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [17:23:35] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [17:25:13] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [17:26:45] (03CR) 10Scott French: "The test yesterday went very smoothly (thank you for all of your work!), but highlighted a couple of ways in which the manual-approval che" [puppet] - 10https://gerrit.wikimedia.org/r/1347761 (https://phabricator.wikimedia.org/T412941) (owner: 10Scott French) [17:32:04] (03PS1) 10Dzahn: site: remove decom'ed zuul1001 [puppet] - 10https://gerrit.wikimedia.org/r/1348847 (https://phabricator.wikimedia.org/T427353) [17:33:26] (03CR) 10Dzahn: [C:03+2] site: remove decom'ed zuul1001 [puppet] - 10https://gerrit.wikimedia.org/r/1348847 (https://phabricator.wikimedia.org/T427353) (owner: 10Dzahn) [17:35:01] !log bking@cumin2003 START - Cookbook sre.hosts.reboot-single for host flink-zk1001.eqiad.wmnet [17:36:14] (03PS1) 10Andrew Bogott: keystone-public-service.erb: correct variable def [puppet] - 10https://gerrit.wikimedia.org/r/1348851 (https://phabricator.wikimedia.org/T421911) [17:37:24] PROBLEM - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [17:38:03] (03CR) 10Andrew Bogott: [C:03+2] keystone-public-service.erb: correct variable def [puppet] - 10https://gerrit.wikimedia.org/r/1348851 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [17:39:06] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1001.eqiad.wmnet [17:39:18] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [17:39:36] (03PS1) 10Dzahn: switch zuul.discovery.wmnet from zuul1001 to zuul1004 [dns] - 10https://gerrit.wikimedia.org/r/1348852 (https://phabricator.wikimedia.org/T427353) [17:41:40] (03CR) 10Dzahn: [C:03+2] switch zuul.discovery.wmnet from zuul1001 to zuul1004 [dns] - 10https://gerrit.wikimedia.org/r/1348852 (https://phabricator.wikimedia.org/T427353) (owner: 10Dzahn) [17:41:45] !log dzahn@dns1004 START - running authdns-update [17:42:23] !log bking@cumin2003 START - Cookbook sre.hosts.reboot-single for host flink-zk1002.eqiad.wmnet [17:44:22] !log dzahn@dns1004 END - running authdns-update [17:45:29] !log dzahn@cumin1004 START - Cookbook sre.hosts.decommission for hosts zuul1001.eqiad.wmnet [17:46:18] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1002.eqiad.wmnet [17:46:50] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [17:47:01] !log bking@cumin2003 START - Cookbook sre.hosts.reboot-single for host flink-zk2001.codfw.wmnet [17:50:22] !log dzahn@cumin1004 START - Cookbook sre.dns.netbox [17:51:02] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2001.codfw.wmnet [17:55:06] (03PS1) 10Dzahn: zuul: stop the containers when the systemd units are stopped [puppet] - 10https://gerrit.wikimedia.org/r/1348861 (https://phabricator.wikimedia.org/T395938) [17:55:44] (03CR) 10Dzahn: [C:03+2] zuul: stop the containers when the systemd units are stopped [puppet] - 10https://gerrit.wikimedia.org/r/1348861 (https://phabricator.wikimedia.org/T395938) (owner: 10Dzahn) [17:55:46] dzahn@cumin1004 decommission (PID 2417915) is awaiting input [17:56:43] !log bking@cumin2003 START - Cookbook sre.hosts.reboot-single for host flink-zk1003.eqiad.wmnet [17:56:51] !log bking@cumin2003 START - Cookbook sre.hosts.reboot-single for host flink-zk2002.codfw.wmnet [17:58:10] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12390782 (10Dzahn) [18:00:41] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk1003.eqiad.wmnet [18:00:54] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2002.codfw.wmnet [18:01:12] !log dzahn@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" [18:01:37] !log dzahn@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: zuul1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - dzahn@cumin1004" [18:01:37] !log dzahn@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [18:01:38] !log dzahn@cumin1004 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts zuul1001.eqiad.wmnet [18:01:54] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12390795 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin1004 for hosts: `zuul1001.eqiad.wmnet` - zuul1001.eq... [18:03:16] !log dzahn@cumin1004 START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie [18:03:28] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12390797 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1004 for host zuul1004.eqiad.wmnet with OS tr... [18:13:39] (03CR) 10AKhatun: "Thanks for the changes! I had some comments. Let me know what you think. I can make the feature count lambda related changes after you if " [deployment-charts] - 10https://gerrit.wikimedia.org/r/1347666 (https://phabricator.wikimedia.org/T437000) (owner: 10Clément Goubert) [18:15:24] (03PS1) 10Hashar: ipmi: ruby: stop using then for multiline when statement [puppet] - 10https://gerrit.wikimedia.org/r/1348873 [18:15:24] (03PS1) 10Hashar: install_server: fix spec to show all examples [puppet] - 10https://gerrit.wikimedia.org/r/1348874 [18:15:24] (03PS1) 10Hashar: wmflib: fix rubocop ignore missing a department [puppet] - 10https://gerrit.wikimedia.org/r/1348875 [18:15:25] (03PS1) 10Hashar: rubocop: upgrade to 0.81.0 [puppet] - 10https://gerrit.wikimedia.org/r/1348876 [18:34:02] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [18:36:19] !log bking@cumin2003 START - Cookbook sre.hosts.reboot-single for host flink-zk2003.codfw.wmnet [18:37:24] RECOVERY - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:38:03] FIRING: ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:40:09] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flink-zk2003.codfw.wmnet [18:43:03] RESOLVED: ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:48:23] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-c6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [18:53:33] dzahn@cumin1004 reimage (PID 2419022) is awaiting input [18:53:34] (03PS2) 10Andrew Bogott: zookeeper: remove need to opt-in to enable logging on trixie and beyond [puppet] - 10https://gerrit.wikimedia.org/r/1345209 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [18:56:44] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345209 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [19:02:15] (03PS3) 10Andrew Bogott: zookeeper: remove need to opt-in to enable logging on trixie and beyond [puppet] - 10https://gerrit.wikimedia.org/r/1345209 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [19:03:23] (03CR) 10CI reject: [V:04-1] zookeeper: remove need to opt-in to enable logging on trixie and beyond [puppet] - 10https://gerrit.wikimedia.org/r/1345209 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [19:08:13] !log dzahn@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1004.eqiad.wmnet with OS trixie [19:08:27] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12390902 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1004 for host zuul1004.eqiad.wmnet with OS trixie... [19:08:45] (03PS4) 10Andrew Bogott: zookeeper: remove need to opt-in to enable logging on trixie and beyond [puppet] - 10https://gerrit.wikimedia.org/r/1345209 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [19:09:46] (03CR) 10Jdlrobson: [C:03+1] "FYI: Removed in https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1259251" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1193445 (https://phabricator.wikimedia.org/T406627) (owner: 10LorenMora) [19:09:56] (03CR) 10CI reject: [V:04-1] zookeeper: remove need to opt-in to enable logging on trixie and beyond [puppet] - 10https://gerrit.wikimedia.org/r/1345209 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [19:12:14] (03PS5) 10Andrew Bogott: zookeeper: remove need to opt-in to enable logging on trixie and beyond [puppet] - 10https://gerrit.wikimedia.org/r/1345209 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [19:14:12] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345209 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [19:25:19] (03PS6) 10Andrew Bogott: zookeeper: remove need to opt-in to enable logging on trixie and beyond [puppet] - 10https://gerrit.wikimedia.org/r/1345209 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [19:27:40] 10ops-eqiad, 06SRE, 06DC-Ops, 06Traffic: Q3:rack/setup/install lvs102[1-4] - https://phabricator.wikimedia.org/T440030 (10RobH) 03NEW [19:28:34] 10ops-eqiad, 06SRE, 06DC-Ops, 06Traffic: Q3:rack/setup/install lvs102[1-4] - https://phabricator.wikimedia.org/T440030#12390962 (10RobH) a:03ssingh @ssingh, I've made some assumptions regarding hostnames and networking. Please review and also provide racking deails. Please update the site.pp file with... [19:29:35] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1345209 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [19:33:10] (03CR) 10Ahmon Dancy: [V:03+1 C:03+1] "The changes LGTM! Just a typo in the commit message." [puppet] - 10https://gerrit.wikimedia.org/r/1347761 (https://phabricator.wikimedia.org/T412941) (owner: 10Scott French) [19:33:36] !log zuul1004 - reimage stuck in d-i on leftover software RAID/LVM from its previous life as a ganeti host; wiping old md/LVM metadata and partition tables from the installer shell same fix as on zuul1005 (as also mentioned by Valerie/dcops) - T427353 [19:33:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:33:39] T427353: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353 [19:34:02] FIRING: NetworkDeviceAlarmActive: Alarm active on ssw1-f1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=ssw1-f1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [19:34:31] 06SRE, 10SRE-Access-Requests: Remaining access pieces for Eva to use Jupyter - https://phabricator.wikimedia.org/T440031 (10eniroomand) 03NEW [19:37:23] !log bking@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] [19:37:49] !log bking@cumin2003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003'] [19:37:57] !log bking@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] [19:38:24] PROBLEM - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [19:39:05] !log dzahn@cumin1004 START - Cookbook sre.hosts.reimage for host zuul1004.eqiad.wmnet with OS trixie [19:39:25] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12390997 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1004 for host zuul1004.eqiad.wmnet with OS tr... [19:40:29] (03PS1) 10Hashar: rake: pass --ff-only when doing git pull [puppet] - 10https://gerrit.wikimedia.org/r/1348935 [19:46:10] !log bking@cumin2003 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['an-druid1003'] [19:47:53] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12391017 (10Dzahn) zuul1004 reimage got stuck in the installer again (leftover md RAID + LVM from the previous install, same as zuul1005).... [19:47:57] !log bking@cumin2003 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['an-druid1003'] [19:54:06] !log dzahn@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage [19:56:12] (03CR) 10Dzahn: [C:03+1] "thank you for amending!:) yes, this looks all good to me:)" [puppet] - 10https://gerrit.wikimedia.org/r/1345209 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [19:58:38] !log dzahn@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1004.eqiad.wmnet with reason: host reimage [19:58:57] (03CR) 10Dzahn: [C:03+1] "compiling on "C:zookeeper" as well, just in case" [puppet] - 10https://gerrit.wikimedia.org/r/1345209 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [19:59:08] (03PS2) 10Hashar: rubocop: upgrade to 0.82.0 [puppet] - 10https://gerrit.wikimedia.org/r/1348876 [19:59:08] (03PS1) 10Hashar: logstash: change Object#kind_of? to Object#is_a? [puppet] - 10https://gerrit.wikimedia.org/r/1348957 [19:59:09] (03PS1) 10Hashar: jupyterhub: change Object#kind_of? to Object#is_a? [puppet] - 10https://gerrit.wikimedia.org/r/1348958 [20:00:45] (03CR) 10Dzahn: [V:03+1 C:03+1] "https://puppet-compiler.wmflabs.org/output/1345209/9497/" [puppet] - 10https://gerrit.wikimedia.org/r/1345209 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [20:01:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:02:35] 10ops-eqiad, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): an-druid1003 Bootlooping: able to reseat/power drain? - https://phabricator.wikimedia.org/T440033 (10bking) 03NEW [20:03:05] 10ops-eqiad, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): an-druid1003 Bootlooping: able to reseat/power drain? - https://phabricator.wikimedia.org/T440033#12391076 (10bking) p:05Triage→03High a:05bking→03None [20:04:38] 06SRE, 06Traffic, 06Traffic-Icebox, 07Community-Wishlist, and 2 others: Support Encrypted Client Hello (ECH) on Wikimedia servers - https://phabricator.wikimedia.org/T205378#12391105 (10ssingh) We have captured the current state of the ECH deployment plan at https://meta.wikimedia.org/wiki/Talk:Community_W... [20:06:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:10:08] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host dse-k8s-worker2011.codfw.wmnet with OS bookworm [20:10:20] PROBLEM - MariaDB Replica Lag: s8 on db2198 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 603.43 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:10:22] PROBLEM - MariaDB Replica Lag: s1 on db2250 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 605.12 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:10:33] !log btullis@cumin1004 START - Cookbook sre.hosts.reimage for host dse-k8s-worker2010.codfw.wmnet with OS bookworm [20:10:50] PROBLEM - MariaDB Replica Lag: s1 on db1240 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 634.21 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:11:00] PROBLEM - MariaDB Replica Lag: s8 on db1285 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 644.22 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:17:25] !log dzahn@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1004.eqiad.wmnet with OS trixie [20:17:36] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12391466 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1004 for host zuul1004.eqiad.wmnet with OS trixie... [20:18:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [20:20:56] !log osleger@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [20:21:27] !log osleger@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [20:21:28] !log osleger@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [20:22:04] !log osleger@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [20:23:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [20:23:28] !log osleger@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [20:23:31] !log osleger@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [20:23:32] !log osleger@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [20:23:35] !log osleger@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [20:26:36] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage [20:26:57] !log btullis@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage [20:31:57] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2011.codfw.wmnet with reason: host reimage [20:37:47] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2010.codfw.wmnet with reason: host reimage [20:38:24] RECOVERY - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [20:40:53] !log jclark@cumin1004 START - Cookbook sre.hosts.reimage for host an-druid1003.eqiad.wmnet with OS bookworm [20:49:02] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2011.codfw.wmnet with OS bookworm [20:53:49] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2010.codfw.wmnet with OS bookworm [20:54:34] !log bking@an-druid100[4-7] `tune2fs -m0 /dev/mapper/vg0-srv` T440033 [20:54:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:54:37] T440033: an-druid1003 Bootlooping: able to reseat/power drain? - https://phabricator.wikimedia.org/T440033 [21:01:38] !log jclark@cumin1004 START - Cookbook sre.hosts.provision for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [21:10:20] !log jclark@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host an-druid1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [21:13:00] RECOVERY - MariaDB Replica Lag: s8 on db1285 is OK: OK slave_sql_lag Replication lag: 0.01 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:13:20] RECOVERY - MariaDB Replica Lag: s8 on db2198 is OK: OK slave_sql_lag Replication lag: 0.34 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:21:35] (03PS1) 10JHathaway: stdlib: replace deprecated function names [puppet] - 10https://gerrit.wikimedia.org/r/1348986 (https://phabricator.wikimedia.org/T438912) [21:21:52] (03CR) 10Lerickson: [C:03+1] wdqs: add wikidata prefixes config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343616 (https://phabricator.wikimedia.org/T438476) (owner: 10Gmodena) [21:22:05] (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1348986 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [21:24:20] PROBLEM - MariaDB Replica Lag: x3 on db2200 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 644.27 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:35:04] (03CR) 10JHathaway: "Thanks for the nudge @adancy@wikimedia.org, why does Cloud VPS need to use the resource like syntax? I looked at the code, but it wasn't c" [puppet] - 10https://gerrit.wikimedia.org/r/1137013 (owner: 10Krinkle) [21:37:28] (03CR) 10JHathaway: "ready for review!" [puppet] - 10https://gerrit.wikimedia.org/r/1348986 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [21:38:24] RECOVERY - MariaDB Replica Lag: s1 on db2250 is OK: OK slave_sql_lag Replication lag: 0.01 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:39:20] RECOVERY - MariaDB Replica Lag: x3 on db2200 is OK: OK slave_sql_lag Replication lag: 0.46 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:43:58] 06SRE, 06Traffic: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391683 (10Reedy) p:05Triage→03High [21:44:53] 06SRE, 06Traffic: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391686 (10Reedy) [21:45:48] 06SRE, 06Traffic: MW Tarball download issues - https://phabricator.wikimedia.org/T440062#12391691 (10Reedy) [21:46:50] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [21:47:19] 06SRE, 06Traffic: MW Tarball download issues - https://phabricator.wikimedia.org/T440062#12391697 (10taavi) I can reproduce when hitting both esams and drmrs, but not when hitting eqiad. In case traceroutes are helpful, see P96552. [21:47:50] RECOVERY - MariaDB Replica Lag: s1 on db1240 is OK: OK slave_sql_lag Replication lag: 0.14 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:48:30] 06SRE, 06Traffic: MW Tarball download issues - https://phabricator.wikimedia.org/T440062#12391698 (10Reedy) [21:49:33] 06SRE, 06Traffic: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391699 (10Reedy) [21:53:31] 06SRE, 06Traffic: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391704 (10taavi) Can reproduce from the esams bastion directly: `lang=shell-session,lines=5 taavi@bast3007 ~ $ wget "https://releases.wikimedia.org/mediawiki/1.43/mediawiki-core-1.43.10.tar.gz" --2026-10-02... [21:53:48] PROBLEM - MariaDB Replica Lag: x3 on db1216 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 611.44 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:54:22] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): an-druid1003 Bootlooping: able to reseat/power drain? - https://phabricator.wikimedia.org/T440033#12391706 (10bking) a:03Jclark-ctr [21:54:46] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-09-18 - 2026-10-09): an-druid1003 Bootlooping: able to reseat/power drain? - https://phabricator.wikimedia.org/T440033#12391710 (10bking) Assiging to @Jclark-ctr per conversation in #wikimedia-dcops IRC. [21:56:44] (03PS1) 10Dzahn: zuul: create the zuul group with a fixed GID 923 [puppet] - 10https://gerrit.wikimedia.org/r/1349036 (https://phabricator.wikimedia.org/T395938) [22:02:57] jclark@cumin1004 reimage (PID 2440885) is awaiting input [22:05:48] RECOVERY - MariaDB Replica Lag: x3 on db1216 is OK: OK slave_sql_lag Replication lag: 0.41 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:06:18] PROBLEM - MariaDB Replica Lag: s4 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 643.14 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:23:50] PROBLEM - MariaDB Replica Lag: s4 on db1265 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 601.16 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:32:42] (03CR) 10Ahmon Dancy: [V:03+1] "Good question! Looking at git history, class profile::puppetserver::wmcs was introduced with this pattern by @jbond in I07582a5bea8de4a937" [puppet] - 10https://gerrit.wikimedia.org/r/1137013 (owner: 10Krinkle) [22:34:02] FIRING: [2x] Temperature: UBB Inlet Temp issue on ml-serve1016:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=ml-serve1016 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [22:34:55] (03CR) 10Ahmon Dancy: [V:03+1] "correction: all of WMCS uses /usr/local/bin/puppet-enc." [puppet] - 10https://gerrit.wikimedia.org/r/1137013 (owner: 10Krinkle) [22:38:42] 06SRE, 06Traffic: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391791 (10Scott_French) Pulling a couple of data points together, this is reproducible internally when fetching directly from `releases.discovery.wmnet` (currently a CNAME to `releases1003.eqiad.wmnet`). Wh... [22:48:23] FIRING: [2x] CertAlmostExpired: gNMI TLS certificate for lsw1-c6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [22:58:59] (03PS1) 10Dzahn: releases: set envoy proxy upstream response timeout to 300.0 [puppet] - 10https://gerrit.wikimedia.org/r/1349116 (https://phabricator.wikimedia.org/T440062) [23:00:12] (03CR) 10Dzahn: [C:03+2] releases: set envoy proxy upstream response timeout to 300.0 [puppet] - 10https://gerrit.wikimedia.org/r/1349116 (https://phabricator.wikimedia.org/T440062) (owner: 10Dzahn) [23:10:27] 06SRE, 06Traffic, 13Patch-For-Review: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391824 (10Scott_French) To recap, @Dzahn noticed that envoy had a 65s timeout on the RouteAction associated with the local listener. Indeed, in a scenario where the overall response hit... [23:15:14] 06SRE, 06Traffic: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068 (10Reedy) 03NEW [23:15:16] 06SRE, 06Traffic, 13Patch-For-Review: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391839 (10Dzahn) I set `profile::tlsproxy::envoy::upstream_response_timeout: 300` in Hiera on the releases server role with the change linked above. This reloaded envoy proxy via puppe... [23:16:52] 06SRE, 06Traffic, 13Patch-For-Review: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391840 (10Dzahn) ` root@releases1003:/home/dzahn# curl -s http://127.0.0.1:9631/stats | grep -E 'upstream_rq_timeout|upstream_rq_completed' | grep -v ': 0$' cluster.local_port_80.exte... [23:17:38] 06SRE, 06Collaboration-Services, 06Traffic, 13Patch-For-Review: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391841 (10Dzahn) [23:19:14] 06SRE, 06Collaboration-Services, 06Traffic: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12391847 (10Dzahn) [23:19:30] 06SRE, 06Collaboration-Services, 06Traffic: Slow download speed from releases.wikimedia.org - https://phabricator.wikimedia.org/T440068#12391848 (10Reedy) [23:24:07] (03CR) 10Dzahn: [C:03+2] zuul: create the zuul group with a fixed GID 923 [puppet] - 10https://gerrit.wikimedia.org/r/1349036 (https://phabricator.wikimedia.org/T395938) (owner: 10Dzahn) [23:32:18] 06SRE, 06Collaboration-Services, 06Traffic: MW Tarball download issues from Europe? - https://phabricator.wikimedia.org/T440062#12391863 (10Dzahn) p:05High→03Medium [23:34:02] FIRING: NetworkDeviceAlarmActive: Alarm active on ssw1-f1-eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=ssw1-f1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [23:36:29] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12391866 (10Dzahn) Summary of today's work on the zuul hosts: **zuul1001 (VM) decommissioned** * Before decom: exported the Zuul proj... [23:39:04] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1349161 [23:39:04] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1349161 (owner: 10TrainBranchBot) [23:40:47] (03PS1) 10Dzahn: zuul: export the project encryption keys daily for backup [puppet] - 10https://gerrit.wikimedia.org/r/1349165 (https://phabricator.wikimedia.org/T436131) [23:41:18] RECOVERY - MariaDB Replica Lag: s4 on db2239 is OK: OK slave_sql_lag Replication lag: 0.12 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [23:44:29] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1349161 (owner: 10TrainBranchBot)