[00:00:06] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox: sync [00:00:16] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [00:00:19] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [00:00:25] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 2.520 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [00:00:35] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [00:03:01] !log jdlrobson@deploy1003 Started scap sync-world: Backport for [[gerrit:1348770|Drop MediaViewerMobileBeta (T436799)]] [00:03:05] T436799: Remove "beta" moniker from MMV beta/mobile/vue version - https://phabricator.wikimedia.org/T436799 [00:03:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2007 bytes in 8.917 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [00:04:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:06:11] RESOLVED: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:07:19] !log jdlrobson@deploy1003 jdlrobson: Backport for [[gerrit:1348770|Drop MediaViewerMobileBeta (T436799)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [00:07:25] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:08:23] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 0.111 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [00:08:30] !log jdlrobson@deploy1003 jdlrobson: Continuing with deployment [00:09:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:11:11] RESOLVED: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:13:01] (03PS1) 10Dwisehaupt: fr-tech: shift frdata services back to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1357016 (https://phabricator.wikimedia.org/T419556) [00:13:47] FIRING: [7x] GnmiInterfaceCountersDrop: asw1-b3-magru is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [00:13:51] !log jdlrobson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1348770|Drop MediaViewerMobileBeta (T436799)]] (duration: 10m 50s) [00:13:55] T436799: Remove "beta" moniker from MMV beta/mobile/vue version - https://phabricator.wikimedia.org/T436799 [00:14:25] (03CR) 10Dwisehaupt: [C:03+2] fr-tech: shift frdata services back to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1357016 (https://phabricator.wikimedia.org/T419556) (owner: 10Dwisehaupt) [00:14:36] !log dwisehaupt@dns1004 START - running authdns-update [00:17:06] !log dwisehaupt@dns1004 END - running authdns-update [00:18:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:21:42] done [00:23:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:25:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [00:26:11] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:40:09] (03PS1) 10Dzahn: gitlab: block 12 Oculus Proxy prefixes due to abuse [puppet] - 10https://gerrit.wikimedia.org/r/1357022 (https://phabricator.wikimedia.org/T441247) [00:43:27] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 3.035 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [00:45:28] (03PS2) 10Dzahn: gitlab: block 3 Oculus Proxy prefixes due to abuse [puppet] - 10https://gerrit.wikimedia.org/r/1357022 (https://phabricator.wikimedia.org/T441247) [00:46:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [00:46:38] (03CR) 10Dzahn: [C:03+2] gitlab: block 3 Oculus Proxy prefixes due to abuse [puppet] - 10https://gerrit.wikimedia.org/r/1357022 (https://phabricator.wikimedia.org/T441247) (owner: 10Dzahn) [00:53:31] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 6.684 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [01:06:01] (03PS1) 10Dzahn: gitlab: add 2 more Oculus prefixes to abusers [puppet] - 10https://gerrit.wikimedia.org/r/1357023 (https://phabricator.wikimedia.org/T441247) [01:07:16] (03PS2) 10Dzahn: gitlab: add 2 more Oculus prefixes to abusers [puppet] - 10https://gerrit.wikimedia.org/r/1357023 (https://phabricator.wikimedia.org/T441247) [01:07:21] (03CR) 10Dzahn: [C:03+2] gitlab: add 2 more Oculus prefixes to abusers [puppet] - 10https://gerrit.wikimedia.org/r/1357023 (https://phabricator.wikimedia.org/T441247) (owner: 10Dzahn) [01:08:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:09:02] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1357024 [01:09:02] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1357024 (owner: 10TrainBranchBot) [01:11:48] (03CR) 10Dzahn: [V:03+1 C:03+2] "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/1357023 (https://phabricator.wikimedia.org/T441247) (owner: 10Dzahn) [01:12:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [01:13:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:15:51] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1357024 (owner: 10TrainBranchBot) [01:22:25] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 0.211 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [01:23:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:24:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:25:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [01:27:31] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 6.593 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [01:31:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [01:32:27] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29309 bytes in 3.919 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [01:34:25] RESOLVED: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:43:55] FIRING: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:48:55] RESOLVED: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:54:01] PROBLEM - Host an-worker1188 is DOWN: PING CRITICAL - Packet loss = 100% [01:54:55] FIRING: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:59:56] RESOLVED: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [02:00:05] Deploy window Automatic branching of MediaWiki, extensions, skins, and vendor – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T0200) [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T0200) [02:00:30] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:07:43] (03PS1) 10Dzahn: gitlab: add an IP to abusers [puppet] - 10https://gerrit.wikimedia.org/r/1357025 (https://phabricator.wikimedia.org/T441247) [02:08:07] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) [02:08:17] (03CR) 10CI reject: [V:04-1] gitlab: add an IP to abusers [puppet] - 10https://gerrit.wikimedia.org/r/1357025 (https://phabricator.wikimedia.org/T441247) (owner: 10Dzahn) [02:08:55] FIRING: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [02:09:05] (03PS2) 10Dzahn: gitlab: add an IP to abusers [puppet] - 10https://gerrit.wikimedia.org/r/1357025 (https://phabricator.wikimedia.org/T441247) [02:09:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:13:47] FIRING: [14x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [02:13:55] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [02:14:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:18:38] (03Abandoned) 10Dzahn: gitlab: add an IP to abusers [puppet] - 10https://gerrit.wikimedia.org/r/1357025 (https://phabricator.wikimedia.org/T441247) (owner: 10Dzahn) [02:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:23:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:24:00] RESOLVED: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [02:26:56] btullis@cumin1004 reboot-workers (PID 1915313) is awaiting input [02:28:47] FIRING: KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1152.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [02:28:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:45:55] FIRING: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [02:50:56] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [02:55:56] RESOLVED: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:00:04] Deploy window Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see Heterogeneous deployment/Train deploys (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T0300) [03:13:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:18:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:19:16] FIRING: [3x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 7d 9h 47m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [03:23:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:28:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:38:47] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [04:00:04] Deploy window Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T0400) [04:01:04] !log mwpresync@deploy1003 Pruned MediaWiki: 1.47.0-wmf.22 (duration: 01m 01s) [04:08:56] FIRING: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:13:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:14:16] FIRING: [7x] GnmiInterfaceCountersDrop: asw1-b3-magru is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [04:18:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:18:55] RESOLVED: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:28:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:33:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:53:21] (03CR) 10Giuseppe Lavagetto: [C:03+2] sre.hosts.provision: enable virtualization by default for wikikube-worker [cookbooks] - 10https://gerrit.wikimedia.org/r/1347480 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [04:55:50] (03Merged) 10jenkins-bot: sre.hosts.provision: enable virtualization by default for wikikube-worker [cookbooks] - 10https://gerrit.wikimedia.org/r/1347480 (https://phabricator.wikimedia.org/T438287) (owner: 10Giuseppe Lavagetto) [05:18:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:23:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:24:51] FIRING: [2x] GitLabReplicaDataStale: GitLab - replica gitlab1003:0 serves data older than 30h - https://wikitech.wikimedia.org/wiki/GitLab/Backup_and_Restore - https://grafana.wikimedia.org/d/R_1IvBZnz/gitlab-omnibus-overview - https://alerts.wikimedia.org/?q=alertname%3DGitLabReplicaDataStale [05:28:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:30:28] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1160.eqiad.wmnet with reason: Maintenance [05:33:18] (03PS1) 10Marostegui: mariadb: Remove db1156 [puppet] - 10https://gerrit.wikimedia.org/r/1357137 (https://phabricator.wikimedia.org/T441298) [05:33:25] !log marostegui@cumin1004 START - Cookbook sre.mysql.decommission [05:33:33] !log marostegui@cumin1004 START - Cookbook sre.hosts.decommission for hosts db1156.eqiad.wmnet [05:33:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:35:11] (03PS1) 10Marostegui: wmnet: Failover m2-master [dns] - 10https://gerrit.wikimedia.org/r/1357138 (https://phabricator.wikimedia.org/T441633) [05:35:22] (03CR) 10Marostegui: [C:03+2] mariadb: Remove db1156 [puppet] - 10https://gerrit.wikimedia.org/r/1357137 (https://phabricator.wikimedia.org/T441298) (owner: 10Marostegui) [05:38:35] RECOVERY - dump of s4 in eqiad on backupmon1001 is OK: Last dump for s4 at eqiad (db1265) taken on 2026-10-20 03:02:08 (181 GiB, +0.2 %) https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Rerun_a_failed_backup [05:39:26] !log marostegui@cumin1004 START - Cookbook sre.dns.netbox [05:44:05] !log marostegui@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1156.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004" [05:44:50] !log marostegui@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db1156.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004" [05:44:50] !log marostegui@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [05:44:52] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db1156.eqiad.wmnet [05:44:59] !log marostegui@cumin1004 Removing db1156 from zarcillo T441298 [05:45:02] T441298: decommission db1156.eqiad.wmnet - https://phabricator.wikimedia.org/T441298 [05:45:03] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.decommission (exit_code=0) [05:45:06] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1156.eqiad.wmnet - https://phabricator.wikimedia.org/T441298#12452566 (10ops-monitoring-bot) db1156 has been deleted from zarcillo [05:45:12] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1156.eqiad.wmnet - https://phabricator.wikimedia.org/T441298#12452568 (10ops-monitoring-bot) db1156 has been decommissioned by Data Persistence [05:45:14] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1156.eqiad.wmnet - https://phabricator.wikimedia.org/T441298#12452569 (10ops-monitoring-bot) a:05Marostegui→03None This host is ready for DC-Ops to decommission [05:47:13] PROBLEM - orchestrator resolve cache non-FQDNs on dborch1002 is CRITICAL: CRITICAL: 1 non-FQDN entries in orchestrator resolve cache: https://wikitech.wikimedia.org/wiki/Orchestrator [05:48:13] RECOVERY - orchestrator resolve cache non-FQDNs on dborch1002 is OK: OK: all orchestrator resolve cache entries are FQDNs https://wikitech.wikimedia.org/wiki/Orchestrator [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T0600) [06:00:05] marostegui, cezmunsta, and federico3: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) Primary database switchover deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T0600). [06:03:11] (03CR) 10Arnaudb: [C:03+2] Revert^4 "gitlab: discard firewall throttling on the primary" [puppet] - 10https://gerrit.wikimedia.org/r/1356859 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [06:03:40] !log moving gitlab behind CDN T425441 [06:03:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:03:45] T425441: gitlab behind CDN - https://phabricator.wikimedia.org/T425441 [06:03:47] FIRING: [18x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [06:04:24] (03CR) 10Arnaudb: [C:03+2] Revert^4 "gitlab: point gitlab.wikimedia.org at the CDN" [dns] - 10https://gerrit.wikimedia.org/r/1356858 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [06:04:50] !log arnaudb@dns1006 START - running authdns-update [06:06:13] !log arnaudb@cumin1004 START - Cookbook sre.hosts.reboot-single for host releases2003.codfw.wmnet [06:07:11] !log arnaudb@dns1006 END - running authdns-update [06:10:14] !log arnaudb@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host releases2003.codfw.wmnet [06:18:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:22:11] (03PS1) 10Marostegui: orchestrator.conf.json.erb: Remove Amir/Federico [puppet] - 10https://gerrit.wikimedia.org/r/1357143 [06:23:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:28:47] FIRING: KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1152.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [06:29:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:29:49] !log arnaudb@cumin1004 START - Cookbook sre.hosts.reboot-single for host aphlict1002.eqiad.wmnet [06:30:02] !log arnaudb@cumin1004 START - Cookbook sre.hosts.reboot-single for host phab1005.eqiad.wmnet [06:30:13] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet [06:30:19] !log arnaudb@cumin1004 START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org [06:31:30] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6001.drmrs.wmnet [06:32:45] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [06:33:25] PROBLEM - Gitlab SSH healthcheck git daemon on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [06:33:33] PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [06:33:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:33:48] !log arnaudb@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aphlict1002.eqiad.wmnet [06:34:33] jmm@cumin2003 drain-node (PID 3332700) is awaiting input [06:34:46] jmm@cumin2003 drain-node (PID 3332459) is awaiting input [06:36:23] RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30031 bytes in 0.160 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [06:37:32] !log arnaudb@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host phab1005.eqiad.wmnet [06:37:34] !log arnaudb@cumin1004 START - Cookbook sre.gerrit.reboot-gerrit Rebooting Gerrit on gerrit1003 [06:37:37] !log arnaudb@cumin1004 END (FAIL) - Cookbook sre.gerrit.reboot-gerrit (exit_code=99) Rebooting Gerrit on gerrit1003 [06:38:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:39:15] RECOVERY - Gitlab SSH healthcheck git daemon on gitlab.wikimedia.org is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u10 (protocol 2.0) https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring [06:40:20] !log arnaudb@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org [06:40:25] (03PS1) 10Arnaudb: gerrit: reboot cookbook update [cookbooks] - 10https://gerrit.wikimedia.org/r/1357144 [06:40:37] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet [06:41:03] !log slyngshede@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_magru and not P{cp7002*} and not P{cp7010*} and A:cp [06:41:45] !log arnaudb@cumin1004 START - Cookbook sre.hosts.reboot-single for host zuul1004.eqiad.wmnet [06:43:03] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti6001.drmrs.wmnet [06:43:47] FIRING: [4x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:45:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [06:47:00] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [06:47:13] !log arnaudb@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host zuul1004.eqiad.wmnet [06:47:29] (03CR) 10Arnaudb: [C:03+2] gerrit: reboot cookbook update [cookbooks] - 10https://gerrit.wikimedia.org/r/1357144 (owner: 10Arnaudb) [06:48:38] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet [06:49:04] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6001.drmrs.wmnet [06:49:05] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet [06:49:23] (03Merged) 10jenkins-bot: gerrit: reboot cookbook update [cookbooks] - 10https://gerrit.wikimedia.org/r/1357144 (owner: 10Arnaudb) [06:49:26] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6001.drmrs.wmnet [06:50:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [06:51:34] !log arnaudb@cumin1004 START - Cookbook sre.hosts.reboot-single for host releases1003.eqiad.wmnet [06:52:53] !log slyngshede@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp7001.magru.wmnet [06:53:14] (03Abandoned) 10Nikerabbit: Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1356831 (owner: 10L10n-bot) [06:57:21] (03CR) 10Nikerabbit: [V:03+2] Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1356884 (owner: 10L10n-bot) [06:57:32] !log arnaudb@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host releases1003.eqiad.wmnet [06:59:33] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet [06:59:57] !log failover Ganeti master in drms01 to ganeti6001 [06:59:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:00:04] Amir1, urbanecm, and awight: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for UTC morning backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T0700). [07:00:05] No Gerrit patches in the queue for this window AFAICS. [07:01:10] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host install7002.wikimedia.org [07:02:19] FIRING: [2x] HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [07:03:05] PROBLEM - ganeti-wconfd running on ganeti6003 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 110 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [07:03:47] FIRING: [3x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-magru (195.200.68.139) - group Confed_magru - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [07:04:41] jmm@cumin2003 drain-node (PID 3348050) is awaiting input [07:07:51] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install7002.wikimedia.org [07:16:31] (03CR) 10CWilliams: [C:03+1] "LGTM" [dns] - 10https://gerrit.wikimedia.org/r/1357138 (https://phabricator.wikimedia.org/T441633) (owner: 10Marostegui) [07:17:05] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [07:17:52] (03CR) 10CWilliams: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1357143 (owner: 10Marostegui) [07:19:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:20:42] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [07:22:19] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [07:23:47] FIRING: [3x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 7d 5h 42m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [07:24:16] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for mszwarc - https://phabricator.wikimedia.org/T441400#12452675 (10mszwarc) [07:24:42] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for mszwarc - https://phabricator.wikimedia.org/T441400#12452678 (10mszwarc) >>! In T441400#12451288, @Dzahn wrote: > @mszwarc looks like it's back to you to verify the new keys work and check off that box ;) Verified both keys, both... [07:24:53] (03CR) 10Mszwarc: [C:03+1] "Both FIDO keys work, the old key can be removed" [puppet] - 10https://gerrit.wikimedia.org/r/1356257 (https://phabricator.wikimedia.org/T441400) (owner: 10Mszwarc) [07:28:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:30:38] (03PS2) 10Mszwarc: admin: Remove non-FIDO SSH key for mszwarc [puppet] - 10https://gerrit.wikimedia.org/r/1356257 (https://phabricator.wikimedia.org/T441400) [07:32:29] (03CR) 10Slyngshede: [C:03+2] admin: Remove non-FIDO SSH key for mszwarc [puppet] - 10https://gerrit.wikimedia.org/r/1356257 (https://phabricator.wikimedia.org/T441400) (owner: 10Mszwarc) [07:32:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [07:32:53] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [07:33:04] (03PS1) 10Ayounsi: Add eqiad A/B switches Incinga hostgroups [puppet] - 10https://gerrit.wikimedia.org/r/1357147 [07:33:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:34:11] (03CR) 10Elukey: [C:03+1] Add eqiad A/B switches Incinga hostgroups [puppet] - 10https://gerrit.wikimedia.org/r/1357147 (owner: 10Ayounsi) [07:34:44] !log slyngshede@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp7003.magru.wmnet [07:35:10] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for mszwarc - https://phabricator.wikimedia.org/T441400#12452703 (10SLyngshede-WMF) [07:35:23] (03PS3) 10Filippo Giunchedi: team-wmcs: add backup dashboard [alerts] - 10https://gerrit.wikimedia.org/r/1356386 (https://phabricator.wikimedia.org/T428893) [07:35:40] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for mszwarc - https://phabricator.wikimedia.org/T441400#12452704 (10SLyngshede-WMF) 05In progress→03Resolved a:03SLyngshede-WMF Key will be gone from production hosts in about 30 minutes. [07:35:53] (03CR) 10Ayounsi: [C:03+2] Add eqiad A/B switches Incinga hostgroups [puppet] - 10https://gerrit.wikimedia.org/r/1357147 (owner: 10Ayounsi) [07:36:56] (03PS6) 10Elukey: registryctl: add an option to drop tags containing a date [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1354834 (https://phabricator.wikimedia.org/T437297) [07:36:59] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [07:37:09] (03CR) 10Elukey: registryctl: add an option to drop tags containing a date (031 comment) [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1354834 (https://phabricator.wikimedia.org/T437297) (owner: 10Elukey) [07:37:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [07:38:47] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [07:38:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:40:22] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [07:41:12] (03PS1) 10Dpogorzelski: ml-staging-codfw: install Gateway API and Inference Extension CRDs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357212 (https://phabricator.wikimedia.org/T436650) [07:42:22] (03CR) 10CI reject: [V:04-1] ml-staging-codfw: install Gateway API and Inference Extension CRDs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357212 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [07:44:46] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet [07:45:15] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host install6003.wikimedia.org [07:46:58] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install6003.wikimedia.org [07:47:24] (03PS2) 10Dpogorzelski: ml-staging-codfw: install Gateway API and Inference Extension CRDs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357212 (https://phabricator.wikimedia.org/T436650) [07:48:31] (03CR) 10CI reject: [V:04-1] ml-staging-codfw: install Gateway API and Inference Extension CRDs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357212 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [07:48:52] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Eqiad worker nodes should be connected to 10G NICs - https://phabricator.wikimedia.org/T440696#12452724 (10JMeybohm) >>! In T440696#12442121, @cmooney wrote: >>>! In T440696#12442044, @JMeybohm wrote: >> We can depool... [07:50:28] (03CR) 10JMeybohm: [C:03+2] "Indeed. CI blib maybe..." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344832 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [07:51:12] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet [07:51:20] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet [07:52:01] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5005.eqsin.wmnet [07:53:47] FIRING: [7x] GnmiInterfaceCountersDrop: asw1-b3-magru is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [07:55:08] jmm@cumin2003 drain-node (PID 3356588) is awaiting input [07:55:53] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for mstyles - https://phabricator.wikimedia.org/T441724#12452744 (10Jelto) I reached out to @Mstyles to double-check the key [07:56:08] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [07:56:50] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host an-master1004.eqiad.wmnet [07:58:37] (03PS2) 10Cathal Mooney: Remaining puppet additions for eqiad row a/b switches [puppet] - 10https://gerrit.wikimedia.org/r/1355641 (https://phabricator.wikimedia.org/T439535) [07:58:47] FIRING: [7x] GnmiInterfaceCountersDrop: asw1-b3-magru is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [07:59:57] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet [08:00:28] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-master1004.eqiad.wmnet [08:00:41] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host install5004.wikimedia.org [08:01:10] (03PS1) 10Bartosz Wójtowicz: ml-services: move llm-qwen38-27b to LLMInferenceService [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357214 (https://phabricator.wikimedia.org/T441645) [08:01:37] (03PS1) 10Gkyziridis: ml-services: Replace llm-qwen38-27b with llm-gpt-oss-safeguard-20b in experimental [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357216 (https://phabricator.wikimedia.org/T440307) [08:02:12] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [08:02:19] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [08:02:19] (03CR) 10Ayounsi: [C:03+1] Remaining puppet additions for eqiad row a/b switches [puppet] - 10https://gerrit.wikimedia.org/r/1355641 (https://phabricator.wikimedia.org/T439535) (owner: 10Cathal Mooney) [08:03:42] jmm@cumin2003 drain-node (PID 3356588) is awaiting input [08:03:45] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [08:04:09] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet [08:04:20] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6003.drmrs.wmnet [08:04:20] (03CR) 10Jelto: [C:03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1355775 (https://phabricator.wikimedia.org/T441068) (owner: 10Dzahn) [08:04:51] (03CR) 10Cathal Mooney: [C:03+2] Remaining puppet additions for eqiad row a/b switches [puppet] - 10https://gerrit.wikimedia.org/r/1355641 (https://phabricator.wikimedia.org/T439535) (owner: 10Cathal Mooney) [08:06:08] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [08:06:19] (03CR) 10Jelto: [C:03+1] admin: add obenhmida to analytics-privatedata-users, lvl1 [puppet] - 10https://gerrit.wikimedia.org/r/1356986 (https://phabricator.wikimedia.org/T440464) (owner: 10Dzahn) [08:06:29] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet [08:07:26] jmm@cumin2003 drain-node (PID 3359910) is awaiting input [08:09:43] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install5004.wikimedia.org [08:10:09] (03CR) 10Blake: [C:03+1] site.pp: add mc-misc200X hosts to the codfw memcached pool #1 [puppet] - 10https://gerrit.wikimedia.org/r/1356950 (https://phabricator.wikimedia.org/T440896) (owner: 10Effie Mouzeli) [08:10:21] (03PS3) 10Dpogorzelski: ml-staging-codfw: install Gateway API and Inference Extension CRDs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357212 (https://phabricator.wikimedia.org/T436650) [08:10:33] (03CR) 10Blake: [C:03+1] site.pp: add mc-misc200X hosts to the codfw memcached pool #2 [puppet] - 10https://gerrit.wikimedia.org/r/1356951 (https://phabricator.wikimedia.org/T440896) (owner: 10Effie Mouzeli) [08:10:35] RECOVERY - Host sretest1006 is UP: PING OK - Packet loss = 0%, RTA = 0.39 ms [08:10:45] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet [08:10:52] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5005.eqsin.wmnet [08:11:34] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [08:11:37] !log klausman@cumin1004 START - Cookbook sre.hosts.remove-downtime for ml-serve1015.eqiad.wmnet [08:11:37] !log klausman@cumin1004 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ml-serve1015.eqiad.wmnet [08:12:35] (03CR) 10Gkyziridis: [C:03+1] ml-services: move llm-qwen38-27b to LLMInferenceService [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357214 (https://phabricator.wikimedia.org/T441645) (owner: 10Bartosz Wójtowicz) [08:12:55] RECOVERY - Check correctness of the icinga configuration on alert1002 is OK: Icinga configuration is correct https://wikitech.wikimedia.org/wiki/Icinga [08:13:01] (03PS1) 10Ryan Kemper: airflow-analytics-test: upgrade to Airflow 3.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357219 (https://phabricator.wikimedia.org/T435477) [08:13:08] (03CR) 10Effie Mouzeli: [C:03+1] logstash/rsyslog: route php8 versions correctly [puppet] - 10https://gerrit.wikimedia.org/r/1356954 (owner: 10Hnowlan) [08:13:47] FIRING: [3x] GnmiInterfaceCountersDrop: cloudsw1-b1-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [08:13:51] jmm@cumin2003 drain-node (PID 3359910) is awaiting input [08:15:01] (03CR) 10CI reject: [V:04-1] ml-staging-codfw: install Gateway API and Inference Extension CRDs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357212 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [08:15:12] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: move llm-qwen38-27b to LLMInferenceService [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357214 (https://phabricator.wikimedia.org/T441645) (owner: 10Bartosz Wójtowicz) [08:16:22] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti6003.drmrs.wmnet [08:16:48] (03Merged) 10jenkins-bot: ml-services: move llm-qwen38-27b to LLMInferenceService [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357214 (https://phabricator.wikimedia.org/T441645) (owner: 10Bartosz Wójtowicz) [08:16:52] (03CR) 10Effie Mouzeli: [C:03+2] site.pp: add mc-misc200X hosts to the codfw memcached pool #1 [puppet] - 10https://gerrit.wikimedia.org/r/1356950 (https://phabricator.wikimedia.org/T440896) (owner: 10Effie Mouzeli) [08:16:52] !log failover Ganeti master in eqsin to ganeti5007 [08:16:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:16:57] !log slyngshede@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp7004.magru.wmnet [08:17:56] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [08:18:00] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . [08:19:17] PROBLEM - ganeti-wconfd running on ganeti5004 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 110 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [08:19:18] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host install4004.wikimedia.org [08:20:05] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-restart-varnish rolling restart of Varnish on 1 hosts matching query P{cp3071.*} [08:21:04] (03CR) 10Ladsgroup: [C:03+1] orchestrator.conf.json.erb: Remove Amir/Federico [puppet] - 10https://gerrit.wikimedia.org/r/1357143 (owner: 10Marostegui) [08:22:30] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.cdn.roll-restart-varnish (exit_code=0) rolling restart of Varnish on 1 hosts matching query P{cp3071.*} [08:22:31] (03PS4) 10Dpogorzelski: ml-staging-codfw: install Gateway API and Inference Extension CRDs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357212 (https://phabricator.wikimedia.org/T436650) [08:22:36] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6003.drmrs.wmnet [08:22:41] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6003.drmrs.wmnet [08:23:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:23:48] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host an-coord1004.eqiad.wmnet [08:24:52] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6002.drmrs.wmnet [08:25:53] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install4004.wikimedia.org [08:27:55] jmm@cumin2003 drain-node (PID 3364739) is awaiting input [08:28:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:28:48] (03CR) 10Federico Ceratto: [C:03+1] orchestrator.conf.json.erb: Remove Amir/Federico [puppet] - 10https://gerrit.wikimedia.org/r/1357143 (owner: 10Marostegui) [08:28:58] !log btullis@cumin1004 START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-codfw [08:29:07] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host install3004.wikimedia.org [08:29:44] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-coord1004.eqiad.wmnet [08:31:47] !log jiji@cumin1004 START - Cookbook sre.hosts.reboot-single for host mc-misc2001.codfw.wmnet [08:32:16] PROBLEM - BFD status on lsw1-a7-codfw.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [08:32:45] jmm@cumin2003 drain-node (PID 3364739) is awaiting input [08:33:24] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet [08:33:29] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti6002.drmrs.wmnet [08:33:47] FIRING: [20x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [08:34:21] (03PS1) 10Jelto: miscweb: update annualreport image to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357225 (https://phabricator.wikimedia.org/T412941) [08:35:41] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install3004.wikimedia.org [08:36:33] jmm@cumin2003 drain-node (PID 3365675) is awaiting input [08:37:03] !log jiji@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2001.codfw.wmnet [08:37:16] RECOVERY - BFD status on lsw1-a7-codfw.mgmt is OK: UP: 4 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [08:37:22] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host install2005.wikimedia.org [08:38:47] FIRING: [20x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [08:38:52] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:39:32] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6002.drmrs.wmnet [08:39:54] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6002.drmrs.wmnet [08:41:30] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin rebooting P{lvs4010.*,lvs5006.*,lvs6003.*} and A:liberica [08:41:59] jmm@cumin2003 drain-node (PID 3365675) is awaiting input [08:42:13] (03Abandoned) 10Kevin Bazira: conftool-data: Add tts-section-generator [puppet] - 10https://gerrit.wikimedia.org/r/1311799 (https://phabricator.wikimedia.org/T432308) (owner: 10Kevin Bazira) [08:43:48] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:43:55] !log jiji@cumin1004 START - Cookbook sre.hosts.reboot-single for host mc-misc2002.codfw.wmnet [08:44:02] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti5004.eqsin.wmnet [08:44:09] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install2005.wikimedia.org [08:44:24] !log failover Ganeti master in drms02 to ganeti6002 [08:44:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:45:18] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host install1005.wikimedia.org [08:46:03] 06SRE, 06Infrastructure-Foundations, 10netbox: Traceback in sre.dns.netbox accessing a virtual interface - https://phabricator.wikimedia.org/T396834#12452990 (10ayounsi) 05Open→03Declined Going to close that tasks , hopefully it's rare enough for it to be an issue. Please re-open if it happens again. [08:46:29] PROBLEM - ganeti-wconfd running on ganeti6004 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 110 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [08:47:53] PROBLEM - BFD status on lsw1-c2-codfw.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [08:49:45] !log trueg@deploy1003 helmfile [eqiad] START helmfile.d/services/eventgate-analytics: sync [08:49:47] !log trueg@deploy1003 helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: sync [08:50:39] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5004.eqsin.wmnet [08:50:46] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet [08:50:54] !log klausman@cumin1004 START - Cookbook sre.hosts.reboot-single for host ml-build1001.eqiad.wmnet [08:51:53] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install1005.wikimedia.org [08:51:54] RECOVERY - BFD status on lsw1-c2-codfw.mgmt is OK: UP: 4 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [08:52:39] !log jiji@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc2002.codfw.wmnet [08:53:04] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs4010.*,lvs5006.*,lvs6003.*} and A:liberica [08:53:47] RESOLVED: CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [08:53:47] FIRING: [2x] GnmiInterfaceCountersDrop: fasw2-e16a-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [08:54:17] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin rebooting P{lvs3010.*} and A:liberica [08:55:25] !log klausman@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-build1001.eqiad.wmnet [08:57:31] (03CR) 10Atsuko: [C:03+1] airflow-analytics-test: upgrade to Airflow 3.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357219 (https://phabricator.wikimedia.org/T435477) (owner: 10Ryan Kemper) [08:57:52] !log klausman@cumin1004 START - Cookbook sre.hosts.reboot-single for host ml-lab1002.eqiad.wmnet [08:57:58] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti6004.drmrs.wmnet [08:58:01] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs3010.*} and A:liberica [08:58:47] FIRING: [3x] GnmiInterfaceCountersDrop: cr2-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [08:58:53] !log slyngshede@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp7005.magru.wmnet [08:59:16] (03CR) 10Hashar: [C:03+1] "Yeah you can roll it ahead of the 3.12 deployment" [puppet] - 10https://gerrit.wikimedia.org/r/1356992 (https://phabricator.wikimedia.org/T441113) (owner: 10Dzahn) [08:59:16] FIRING: [3x] GnmiInterfaceCountersDrop: cr2-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [08:59:55] RECOVERY - Host an-worker1188 is UP: PING OK - Packet loss = 0%, RTA = 0.18 ms [08:59:59] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host bast4006.wikimedia.org [09:03:16] PROBLEM - BFD status on lsw1-d2-codfw.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [09:03:47] !log klausman@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-lab1002.eqiad.wmnet [09:04:11] !log root@cumin2003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-codfw [09:04:15] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2002.codfw.wmnet [09:04:41] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti6004.drmrs.wmnet [09:04:50] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2002.codfw.wmnet [09:06:35] (03CR) 10Volans: [C:03+1] "LGTM" [alerts] - 10https://gerrit.wikimedia.org/r/1356386 (https://phabricator.wikimedia.org/T428893) (owner: 10Filippo Giunchedi) [09:07:16] (03CR) 10JavierMonton: [C:03+1] kafka-jumbo: declare Data Engineering topics [puppet] - 10https://gerrit.wikimedia.org/r/1355378 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [09:07:16] RECOVERY - BFD status on lsw1-d2-codfw.mgmt is OK: UP: 4 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [09:08:02] !log herron@cumin1004 END (PASS) - Cookbook sre.opensearch.roll-restart-reboot (exit_code=0) rolling reboot on A:logstash-codfw [09:08:51] (03PS1) 10Arnaudb: gitlab: point the service IP PTRs at the gitlab-ssh names [dns] - 10https://gerrit.wikimedia.org/r/1357229 (https://phabricator.wikimedia.org/T425441) [09:08:52] FIRING: [5x] GnmiInterfaceCountersDrop: cr2-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [09:08:55] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2002.codfw.wmnet [09:08:56] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2002.codfw.wmnet [09:09:02] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2003.codfw.wmnet [09:09:34] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2003.codfw.wmnet [09:10:12] !log btullis@cumin1004 END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-codfw [09:10:39] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti6004.drmrs.wmnet [09:10:45] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti6004.drmrs.wmnet [09:10:49] (03CR) 10JavierMonton: [C:03+1] EventStreamConfig: Declare the mediawiki.user_thank.v1 stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356976 (https://phabricator.wikimedia.org/T441426) (owner: 10Snwachukwu) [09:12:34] (03CR) 10Gkyziridis: [C:03+2] ml-services: Replace llm-qwen38-27b with llm-gpt-oss-safeguard-20b in experimental [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357216 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [09:13:38] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2003.codfw.wmnet [09:13:40] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2003.codfw.wmnet [09:13:46] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2004.codfw.wmnet [09:14:06] (03Merged) 10jenkins-bot: ml-services: Replace llm-qwen38-27b with llm-gpt-oss-safeguard-20b in experimental [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357216 (https://phabricator.wikimedia.org/T440307) (owner: 10Gkyziridis) [09:14:21] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2004.codfw.wmnet [09:15:14] !log btullis@cumin1004 START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad [09:15:50] (03CR) 10Jelto: [C:03+2] miscweb: update annualreport image to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357225 (https://phabricator.wikimedia.org/T412941) (owner: 10Jelto) [09:16:31] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host an-web1001.eqiad.wmnet [09:17:28] (03CR) 10Trueg: "It is not bw-compatible. So I will close this for the time being." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1355527 (owner: 10Trueg) [09:17:29] (03Merged) 10jenkins-bot: miscweb: update annualreport image to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357225 (https://phabricator.wikimedia.org/T412941) (owner: 10Jelto) [09:18:05] (03Abandoned) 10Trueg: Bump Qlever version to the latest master [deployment-charts] - 10https://gerrit.wikimedia.org/r/1355527 (owner: 10Trueg) [09:18:26] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2004.codfw.wmnet [09:18:27] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2004.codfw.wmnet [09:18:33] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2005.codfw.wmnet [09:18:41] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . [09:19:05] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2005.codfw.wmnet [09:19:29] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2027.codfw.wmnet [09:19:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [09:21:42] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast4006.wikimedia.org [09:22:41] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-web1001.eqiad.wmnet [09:22:42] (03CR) 10Filippo Giunchedi: [C:03+2] team-wmcs: add backup dashboard [alerts] - 10https://gerrit.wikimedia.org/r/1356386 (https://phabricator.wikimedia.org/T428893) (owner: 10Filippo Giunchedi) [09:23:09] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2005.codfw.wmnet [09:23:11] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2005.codfw.wmnet [09:23:17] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2006.codfw.wmnet [09:23:19] jmm@cumin2003 drain-node (PID 3378395) is awaiting input [09:23:52] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2006.codfw.wmnet [09:24:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:24:36] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti2027.codfw.wmnet [09:24:38] (03CR) 10JMeybohm: [C:03+1] cronjob: s/cache.mcrouter.deployment/cache.mcrouter.container/ (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356869 (https://phabricator.wikimedia.org/T386694) (owner: 10Blake) [09:24:52] FIRING: [2x] GitLabReplicaDataStale: GitLab - replica gitlab1003:0 serves data older than 30h - https://wikitech.wikimedia.org/wiki/GitLab/Backup_and_Restore - https://grafana.wikimedia.org/d/R_1IvBZnz/gitlab-omnibus-overview - https://alerts.wikimedia.org/?q=alertname%3DGitLabReplicaDataStale [09:26:31] (03PS2) 10Slyngshede: switchdc: Skip set-readonly in live-test [cookbooks] - 10https://gerrit.wikimedia.org/r/1351336 (https://phabricator.wikimedia.org/T436846) [09:26:43] (03CR) 10Slyngshede: switchdc: Skip set-readonly in live-test (032 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1351336 (https://phabricator.wikimedia.org/T436846) (owner: 10Slyngshede) [09:26:52] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host irc2003.wikimedia.org [09:29:13] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2006.codfw.wmnet [09:29:14] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2006.codfw.wmnet [09:29:21] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2007.codfw.wmnet [09:29:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:29:56] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2007.codfw.wmnet [09:30:10] !log restarting MediaWiki branch cutting job # T438222 T441774 [09:30:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:30:15] T438222: 1.48.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T438222 [09:30:16] T441774: wmf/1.48.0-wmf.1 branch has not been cut - https://phabricator.wikimedia.org/T441774 [09:30:18] !log btullis@cumin1004 END (FAIL) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=1) rolling reboot on A:cephosd-eqiad [09:30:26] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2027.codfw.wmnet [09:30:33] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2027.codfw.wmnet [09:30:39] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2003.wikimedia.org [09:32:17] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2158.codfw.wmnet with reason: Maintenance [09:32:19] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2158 (T441633)', diff saved to https://phabricator.wikimedia.org/P96859 and previous config saved to /var/cache/conftool/dbconfig/20261020-093218-ladsgroup.json [09:32:37] (03CR) 10JMeybohm: [C:04-1] cronjob.yaml.tpl: enable switch for k8s native sidecars (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356824 (https://phabricator.wikimedia.org/T386694) (owner: 10Blake) [09:35:16] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2007.codfw.wmnet [09:35:17] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2007.codfw.wmnet [09:35:23] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2008.codfw.wmnet [09:35:26] andre: I have restarted the branch cut job, it failed during the night [09:35:58] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2008.codfw.wmnet [09:36:12] (03PS1) 10Arnaudb: gitlab: restrict 443 on the service IP to Prometheus probes [puppet] - 10https://gerrit.wikimedia.org/r/1357232 (https://phabricator.wikimedia.org/T425441) [09:36:18] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2028.codfw.wmnet [09:38:13] (03PS1) 10TrainBranchBot: Branch commit for wmf/1.48.0-wmf.1 [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1357234 (https://phabricator.wikimedia.org/T438222) [09:38:15] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/1.48.0-wmf.1 [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1357234 (https://phabricator.wikimedia.org/T438222) (owner: 10TrainBranchBot) [09:38:24] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti2028.codfw.wmnet [09:38:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:38:59] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2158 (T441633)', diff saved to https://phabricator.wikimedia.org/P96860 and previous config saved to /var/cache/conftool/dbconfig/20261020-093859-ladsgroup.json [09:40:41] !log slyngshede@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp7006.magru.wmnet [09:41:12] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host debmonitor-dev2001.codfw.wmnet [09:41:21] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2008.codfw.wmnet [09:41:22] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2008.codfw.wmnet [09:41:28] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker2009.codfw.wmnet [09:43:42] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1029.eqiad.wmnet [09:43:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:44:18] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2028.codfw.wmnet [09:44:40] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2028.codfw.wmnet [09:44:44] (03Merged) 10jenkins-bot: Branch commit for wmf/1.48.0-wmf.1 [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1357234 (https://phabricator.wikimedia.org/T438222) (owner: 10TrainBranchBot) [09:45:13] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor-dev2001.codfw.wmnet [09:46:12] !log jelto@deploy1003 Started scap deploy-service miscweb: demo deployment for MVP walk through [09:46:12] !log jelto@deploy1003 scap deploy-service miscweb: [STAGING/DEFAULT] staging-eqiad Deploying miscweb [09:46:31] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker2009.codfw.wmnet [09:46:48] jmm@cumin2003 drain-node (PID 3393724) is awaiting input [09:47:05] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host debmonitor2003.codfw.wmnet [09:47:34] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2029.codfw.wmnet [09:47:52] !log jelto@deploy1003 scap deploy-service miscweb: [PRODUCTION/DEFAULT] codfw Deploying miscweb [09:49:01] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P96862 and previous config saved to /var/cache/conftool/dbconfig/20261020-094901-ladsgroup.json [09:49:39] !log jelto@deploy1003 scap deploy-service miscweb: [PRODUCTION/DEFAULT] eqiad Deploying miscweb [09:49:42] !log cwilliams@cumin1004 START - Cookbook sre.hosts.reboot-single for host backup1010.eqiad.wmnet [09:49:55] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin rebooting P{lvs7002.*} and A:liberica [09:50:11] !log jelto@deploy1003 Finished scap deploy-service miscweb: demo deployment for MVP walk through (duration: 03m 59s) [09:50:16] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1029.eqiad.wmnet [09:50:28] !log cwilliams@cumin1004 START - Cookbook sre.hosts.reboot-single for host backup2010.codfw.wmnet [09:51:04] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor2003.codfw.wmnet [09:51:29] !log ayounsi@cumin1004 START - Cookbook sre.network.peering with action 'configure' for AS: 3292 [09:51:58] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker2009.codfw.wmnet [09:52:00] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker2009.codfw.wmnet [09:52:00] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-codfw [09:52:23] !log ayounsi@cumin1004 END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'configure' for AS: 3292 [09:53:52] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs7002.*} and A:liberica [09:54:27] jmm@cumin2003 drain-node (PID 3399087) is awaiting input [09:54:45] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1029.eqiad.wmnet [09:55:10] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1029.eqiad.wmnet [09:55:18] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1010.eqiad.wmnet [09:55:21] !log cwilliams@cumin1004 START - Cookbook sre.hosts.reboot-single for host backup1011.eqiad.wmnet [09:55:24] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host debmonitor1003.eqiad.wmnet [09:55:30] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti2029.codfw.wmnet [09:55:38] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1030.eqiad.wmnet [09:57:10] !log root@cumin2003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-worker-eqiad [09:57:14] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1002.eqiad.wmnet [09:57:51] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1002.eqiad.wmnet [09:58:06] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1030.eqiad.wmnet [09:58:47] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2010.codfw.wmnet [09:58:51] !log cwilliams@cumin1004 START - Cookbook sre.hosts.reboot-single for host backup2011.codfw.wmnet [09:59:03] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P96863 and previous config saved to /var/cache/conftool/dbconfig/20261020-095902-ladsgroup.json [09:59:08] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor1003.eqiad.wmnet [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1000) [10:00:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-jobrunner - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [10:00:48] (03CR) 10Effie Mouzeli: [C:03+2] site.pp: add mc-misc200X hosts to the codfw memcached pool #2 [puppet] - 10https://gerrit.wikimedia.org/r/1356951 (https://phabricator.wikimedia.org/T440896) (owner: 10Effie Mouzeli) [10:01:58] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1002.eqiad.wmnet [10:02:01] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1002.eqiad.wmnet [10:02:08] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1003.eqiad.wmnet [10:02:29] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1011.eqiad.wmnet [10:02:40] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1003.eqiad.wmnet [10:02:49] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1030.eqiad.wmnet [10:03:16] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1030.eqiad.wmnet [10:03:18] 06SRE, 06Infrastructure-Foundations, 10netops: Network: gnmic monitoring issues and improvements - https://phabricator.wikimedia.org/T441780 (10cmooney) 03NEW p:05Triage→03Medium [10:03:25] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1031.eqiad.wmnet [10:03:42] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:04:10] (03PS1) 10Mvolz: deployment_server::deploy_service::service_catalog: add citoid [puppet] - 10https://gerrit.wikimedia.org/r/1357236 [10:04:30] 06SRE, 06Infrastructure-Foundations, 10netops: Network: gnmic monitoring issues and improvements - https://phabricator.wikimedia.org/T441780#12453413 (10cmooney) [10:04:31] 06SRE, 06Infrastructure-Foundations, 10netops: Missing series for BGP session_state from eqiad CRs since upgrade to 23.4R2-S8.7 - https://phabricator.wikimedia.org/T435909#12453415 (10cmooney) [10:04:33] 06SRE, 06Infrastructure-Foundations, 10netops: Export network fixed-limit usage stats in gnmi - https://phabricator.wikimedia.org/T395998#12453416 (10cmooney) [10:04:44] !log powercycle ganeti2029 [10:04:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:04:57] (03CR) 10CI reject: [V:04-1] deployment_server::deploy_service::service_catalog: add citoid [puppet] - 10https://gerrit.wikimedia.org/r/1357236 (owner: 10Mvolz) [10:05:15] FIRING: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-jobrunner - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [10:05:49] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2011.codfw.wmnet [10:06:01] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1031.eqiad.wmnet [10:06:37] (03PS1) 10Cathal Mooney: gnmic collection: reduce interval for certain subscriptions [puppet] - 10https://gerrit.wikimedia.org/r/1357237 (https://phabricator.wikimedia.org/T441780) [10:06:48] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1003.eqiad.wmnet [10:06:51] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1003.eqiad.wmnet [10:06:56] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1004.eqiad.wmnet [10:07:21] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet [10:07:29] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1004.eqiad.wmnet [10:07:56] (03PS2) 10Mvolz: deployment_server::deploy_service::service_catalog: add citoid [puppet] - 10https://gerrit.wikimedia.org/r/1357236 [10:09:03] !log cwilliams@cumin1004 START - Cookbook sre.hosts.reboot-single for host backupmon1001.eqiad.wmnet [10:09:05] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2158 (T441633)', diff saved to https://phabricator.wikimedia.org/P96864 and previous config saved to /var/cache/conftool/dbconfig/20261020-100904-ladsgroup.json [10:09:24] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2180.codfw.wmnet with reason: Maintenance [10:09:26] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2180 (T441633)', diff saved to https://phabricator.wikimedia.org/P96865 and previous config saved to /var/cache/conftool/dbconfig/20261020-100926-ladsgroup.json [10:09:57] (03CR) 10Ayounsi: [C:03+1] gnmic collection: reduce interval for certain subscriptions [puppet] - 10https://gerrit.wikimedia.org/r/1357237 (https://phabricator.wikimedia.org/T441780) (owner: 10Cathal Mooney) [10:10:15] RESOLVED: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-jobrunner - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [10:10:18] jouncebot: now [10:10:19] For the next 0 hour(s) and 49 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1000) [10:11:03] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2029.codfw.wmnet [10:11:10] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2029.codfw.wmnet [10:11:17] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1004.eqiad.wmnet [10:11:19] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1004.eqiad.wmnet [10:11:24] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1005.eqiad.wmnet [10:12:03] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1031.eqiad.wmnet [10:12:11] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1031.eqiad.wmnet [10:12:51] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backupmon1001.eqiad.wmnet [10:13:17] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet [10:13:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [10:14:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [10:15:56] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin depooling P{lvs7001.*} and A:liberica [10:16:18] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2180 (T441633)', diff saved to https://phabricator.wikimedia.org/P96866 and previous config saved to /var/cache/conftool/dbconfig/20261020-101617-ladsgroup.json [10:16:19] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7001.*} and A:liberica [10:16:28] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1005.eqiad.wmnet [10:17:19] (03PS1) 10Mvolz: deployment_server::deploy_service::service_catalog: add zotero [puppet] - 10https://gerrit.wikimedia.org/r/1357238 [10:20:36] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1005.eqiad.wmnet [10:20:38] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1005.eqiad.wmnet [10:20:45] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet [10:21:23] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet [10:21:48] (03CR) 10Marostegui: [C:03+2] orchestrator.conf.json.erb: Remove Amir/Federico [puppet] - 10https://gerrit.wikimedia.org/r/1357143 (owner: 10Marostegui) [10:22:02] (03CR) 10Marostegui: [C:03+2] wmnet: Failover m2-master [dns] - 10https://gerrit.wikimedia.org/r/1357138 (https://phabricator.wikimedia.org/T441633) (owner: 10Marostegui) [10:22:16] !log marostegui@dns1004 START - running authdns-update [10:22:31] !log slyngshede@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp7007.magru.wmnet [10:23:06] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin rebooting P{lvs7001.*} and A:liberica [10:23:06] !log vgutierrez@cumin1004 END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) rebooting P{lvs7001.*} and A:liberica [10:24:19] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reboot-single for host lvs7001.magru.wmnet [10:24:39] !log marostegui@dns1004 END - running authdns-update [10:26:16] !log btullis@cumin1004 START - Cookbook sre.ceph.roll-restart-reboot-server rolling reboot on A:cephosd-eqiad [10:26:20] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P96867 and previous config saved to /var/cache/conftool/dbconfig/20261020-102619-ladsgroup.json [10:26:33] (03PS8) 10Blake: cronjob.yaml.tpl: enable switch for k8s native sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356824 (https://phabricator.wikimedia.org/T386694) [10:26:35] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet [10:26:37] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet [10:26:43] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet [10:27:07] (03CR) 10Blake: cronjob.yaml.tpl: enable switch for k8s native sidecars (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356824 (https://phabricator.wikimedia.org/T386694) (owner: 10Blake) [10:27:20] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet [10:27:47] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs7001.magru.wmnet [10:28:24] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin pooling P{lvs7001.*} and A:liberica [10:28:47] FIRING: KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1152.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [10:28:52] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:28:58] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7001.*} and A:liberica [10:29:04] PROBLEM - BFD status on lsw1-e1-eqiad.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [10:30:53] (03CR) 10JMeybohm: [C:03+1] cronjob.yaml.tpl: enable switch for k8s native sidecars (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356824 (https://phabricator.wikimedia.org/T386694) (owner: 10Blake) [10:30:54] (03CR) 10Slyngshede: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1355775 (https://phabricator.wikimedia.org/T441068) (owner: 10Dzahn) [10:31:34] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host ceph-admin2001.codfw.wmnet [10:32:35] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet [10:32:37] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet [10:32:43] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet [10:33:04] RECOVERY - BFD status on lsw1-e1-eqiad.mgmt is OK: UP: 4 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [10:33:18] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet [10:33:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:35:24] (03PS1) 10Majavah: P:wmcs::waf: Add known fingerprints file [puppet] - 10https://gerrit.wikimedia.org/r/1357239 (https://phabricator.wikimedia.org/T441785) [10:35:34] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ceph-admin2001.codfw.wmnet [10:36:05] !log jiji@cumin1004 START - Cookbook sre.discovery.service-route check mw-api-int-ro: maintenance [10:36:05] !log jiji@cumin1004 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-api-int-ro: maintenance [10:36:22] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P96868 and previous config saved to /var/cache/conftool/dbconfig/20261020-103621-ladsgroup.json [10:36:27] !log jiji@cumin1004 START - Cookbook sre.discovery.service-route check mw-api-ext-ro: maintenance [10:36:28] !log jiji@cumin1004 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-api-ext-ro: maintenance [10:36:43] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host matomo1003.eqiad.wmnet [10:36:56] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9533/co" [puppet] - 10https://gerrit.wikimedia.org/r/1357239 (https://phabricator.wikimedia.org/T441785) (owner: 10Majavah) [10:37:44] (03CR) 10CI reject: [V:04-1] P:wmcs::waf: Add known fingerprints file [puppet] - 10https://gerrit.wikimedia.org/r/1357239 (https://phabricator.wikimedia.org/T441785) (owner: 10Majavah) [10:38:20] (03PS2) 10Majavah: P:wmcs::waf: Add known fingerprints file [puppet] - 10https://gerrit.wikimedia.org/r/1357239 (https://phabricator.wikimedia.org/T441785) [10:38:28] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet [10:38:30] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet [10:38:36] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet [10:39:15] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1009.eqiad.wmnet [10:39:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:39:56] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reboot-single for host acmechief-test1001.eqiad.wmnet [10:40:37] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host matomo1003.eqiad.wmnet [10:42:10] !log arnaudb@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [10:42:12] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host krb1005.eqiad.wmnet [10:42:37] !log arnaudb@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [10:43:06] jouncebot: next [10:43:06] In 1 hour(s) and 16 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1200) [10:43:41] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test1001.eqiad.wmnet [10:43:54] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reboot-single for host acmechief-test2001.codfw.wmnet [10:44:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:44:38] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1009.eqiad.wmnet [10:44:40] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1009.eqiad.wmnet [10:44:40] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-worker-eqiad [10:45:30] (03CR) 10Phuedx: "Hey Ryan. Thanks for the ping. Could you explain why different DCs would need different partition counts?" [puppet] - 10https://gerrit.wikimedia.org/r/1355381 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [10:46:09] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host krb1005.eqiad.wmnet [10:46:24] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2180 (T441633)', diff saved to https://phabricator.wikimedia.org/P96869 and previous config saved to /var/cache/conftool/dbconfig/20261020-104623-ladsgroup.json [10:46:33] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2193.codfw.wmnet with reason: Maintenance [10:46:35] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2193 (T441633)', diff saved to https://phabricator.wikimedia.org/P96870 and previous config saved to /var/cache/conftool/dbconfig/20261020-104634-ladsgroup.json [10:46:44] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host archiva1002.wikimedia.org [10:47:45] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief-test2001.codfw.wmnet [10:47:59] !log arnaudb@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [10:48:51] !log arnaudb@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [10:50:04] !log gmodena@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [10:50:18] (03PS1) 10Vgutierrez: hieradata: Switch acme-chief clients to eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1357242 (https://phabricator.wikimedia.org/T439010) [10:50:41] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host archiva1002.wikimedia.org [10:52:14] PROBLEM - BFD status on lsw1-e2-eqiad.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [10:52:29] jouncebot: now [10:52:29] For the next 0 hour(s) and 7 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1000) [10:52:32] jouncebot: next [10:52:32] In 1 hour(s) and 7 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1200) [10:52:45] Dear deployers, I am about to depool mw-api-int-ro from eqiad to test the waters a bit, followed by mw-api-ext-ro. Depending on how those will go, I will move to mw-web or not. [10:53:29] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2193 (T441633)', diff saved to https://phabricator.wikimedia.org/P96871 and previous config saved to /var/cache/conftool/dbconfig/20261020-105328-ladsgroup.json [10:53:41] (03PS2) 10Vgutierrez: hieradata: Switch acme-chief clients to eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1357242 (https://phabricator.wikimedia.org/T439010) [10:54:27] (03CR) 10Daniel Kertesz: [C:03+1] hieradata: Switch acme-chief clients to eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1357242 (https://phabricator.wikimedia.org/T439010) (owner: 10Vgutierrez) [10:56:14] RECOVERY - BFD status on lsw1-e2-eqiad.mgmt is OK: UP: 4 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [10:56:14] (03CR) 10Vgutierrez: [C:03+2] hieradata: Switch acme-chief clients to eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1357242 (https://phabricator.wikimedia.org/T439010) (owner: 10Vgutierrez) [10:57:45] !log kamila@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [10:58:27] !log kamila@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [10:58:46] !log kamila@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [10:59:27] !log kamila@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [10:59:46] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti-test2003.codfw.wmnet [10:59:54] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2030.codfw.wmnet [11:00:08] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1032.eqiad.wmnet [11:01:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [11:01:18] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin rebooting P{lvs5005.*,lvs4009.*} and A:liberica [11:02:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [11:03:29] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti2030.codfw.wmnet [11:03:30] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P96872 and previous config saved to /var/cache/conftool/dbconfig/20261020-110330-ladsgroup.json [11:03:31] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1032.eqiad.wmnet [11:03:47] FIRING: [3x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-magru (195.200.68.139) - group Confed_magru - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [11:03:56] FIRING: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:04:20] !log slyngshede@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp7008.magru.wmnet [11:04:41] (03PS1) 10Marostegui: wikireplicas: Use jemalloc [puppet] - 10https://gerrit.wikimedia.org/r/1357243 (https://phabricator.wikimedia.org/T438200) [11:04:52] !log jiji@cumin1004 START - Cookbook sre.discovery.service-route depool mw-api-int-ro in eqiad: T440896 [11:04:53] !log jiji@cumin1004 START - Cookbook sre.dns.wipe-cache mw-api-int-ro.discovery.wmnet on all recursors [11:04:57] !log jiji@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) mw-api-int-ro.discovery.wmnet on all recursors [11:04:57] T440896: Depool eqiad from reads to load test codfw - https://phabricator.wikimedia.org/T440896 [11:05:56] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2003.codfw.wmnet [11:06:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [11:06:19] (03CR) 10Jelto: [C:03+2] deployment_server::deploy_service::service_catalog: add zotero [puppet] - 10https://gerrit.wikimedia.org/r/1357238 (owner: 10Mvolz) [11:06:38] (03CR) 10Muehlenhoff: [C:03+1] admin: add krinkle to codesearch-roots [puppet] - 10https://gerrit.wikimedia.org/r/1355775 (https://phabricator.wikimedia.org/T441068) (owner: 10Dzahn) [11:07:12] (03CR) 10Marostegui: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1357243 (https://phabricator.wikimedia.org/T438200) (owner: 10Marostegui) [11:07:23] (03PS1) 10Gmodena: WDQS: bump chart version 0.1.0 -> 0.1.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357244 (https://phabricator.wikimedia.org/T440560) [11:07:34] PROBLEM - BFD status on lsw1-e3-eqiad.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [11:07:50] (03CR) 10Jelto: [C:03+2] deployment_server::deploy_service::service_catalog: add citoid [puppet] - 10https://gerrit.wikimedia.org/r/1357236 (owner: 10Mvolz) [11:08:16] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs5005.*,lvs4009.*} and A:liberica [11:08:51] FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from gitlab.discovery.wmnet in eqiad #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=eqiad&var-cluster=text&var-origin=gitlab.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [11:08:56] RESOLVED: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:09:00] handling it atm ↑ [11:09:30] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2030.codfw.wmnet [11:09:41] <_joe_> ack [11:09:43] <_joe_> !ack [11:09:44] All incidents are already acked. [11:09:49] <_joe_> thanks arnaudb [11:09:53] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1032.eqiad.wmnet [11:09:56] !log jiji@cumin1004 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool mw-api-int-ro in eqiad: T440896 [11:09:59] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2030.codfw.wmnet [11:10:00] T440896: Depool eqiad from reads to load test codfw - https://phabricator.wikimedia.org/T440896 [11:10:15] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1032.eqiad.wmnet [11:11:34] RECOVERY - BFD status on lsw1-e3-eqiad.mgmt is OK: UP: 4 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [11:11:49] (03PS2) 10Gmodena: WDQS: bump chart version 0.1.0 -> 0.1.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357244 (https://phabricator.wikimedia.org/T440560) [11:11:55] FIRING: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:12:44] (03PS1) 10Kamila Součková: mw-parsoid: temporarily override image to PHP 8.5 in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357245 (https://phabricator.wikimedia.org/T440321) [11:13:32] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P96873 and previous config saved to /var/cache/conftool/dbconfig/20261020-111332-ladsgroup.json [11:13:51] FIRING: [3x] ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from gitlab.discovery.wmnet in codfw #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [11:14:36] (03CR) 10Marostegui: "@fnegri@wikimedia.org this looks good, the puppet failure reported is simply because I mispelled the name of clouddb1031 - see below." [puppet] - 10https://gerrit.wikimedia.org/r/1357243 (https://phabricator.wikimedia.org/T438200) (owner: 10Marostegui) [11:15:09] !incidents [11:15:10] 8628 (ACKED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet eqiad) [11:15:10] 8627 (RESOLVED) NELHigh sre (thanos-rule@main tcp.timed_out) [11:15:10] 8626 (RESOLVED) NELHigh sre (thanos-rule@main tcp.timed_out) [11:15:10] 8625 (RESOLVED) ProbeDown sre (2a02:ec80:300:ed1a::1 ip6 text-https:443 probes/service http_text-https_ip6 esams) [11:16:55] RESOLVED: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:18:51] FIRING: [4x] ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from gitlab.discovery.wmnet in codfw #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [11:18:52] (03PS2) 10Kamila Součková: mw-parsoid: temporarily override image to PHP 8.5 in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357245 (https://phabricator.wikimedia.org/T440321) [11:21:42] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reboot-single for host acmechief1002.eqiad.wmnet [11:21:46] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reboot-single for host acmechief2002.codfw.wmnet [11:21:48] PROBLEM - BFD status on lsw1-f1-eqiad.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [11:23:34] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2193 (T441633)', diff saved to https://phabricator.wikimedia.org/P96874 and previous config saved to /var/cache/conftool/dbconfig/20261020-112333-ladsgroup.json [11:23:42] PROBLEM - Host cloudlb1001 is DOWN: PING CRITICAL - Packet loss = 100% [11:23:46] PROBLEM - BFD status on cloudsw1-c8-eqiad.mgmt is CRITICAL: Down: 1 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [11:23:55] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance [11:24:07] (03CR) 10Clément Goubert: [C:03+1] mw-parsoid: temporarily override image to PHP 8.5 in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357245 (https://phabricator.wikimedia.org/T440321) (owner: 10Kamila Součková) [11:24:14] RECOVERY - Host cloudlb1001 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [11:24:16] FIRING: [3x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 7d 1h 42m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [11:24:45] (03CR) 10Kamila Součková: [C:03+2] mw-parsoid: temporarily override image to PHP 8.5 in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357245 (https://phabricator.wikimedia.org/T440321) (owner: 10Kamila Součková) [11:24:46] RECOVERY - BFD status on cloudsw1-c8-eqiad.mgmt is OK: UP: 16 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [11:25:27] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief1002.eqiad.wmnet [11:25:35] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host acmechief2002.codfw.wmnet [11:25:48] RECOVERY - BFD status on lsw1-f1-eqiad.mgmt is OK: UP: 4 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [11:26:08] (03Merged) 10jenkins-bot: mw-parsoid: temporarily override image to PHP 8.5 in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357245 (https://phabricator.wikimedia.org/T440321) (owner: 10Kamila Součková) [11:26:20] PROBLEM - Host cloudlb1002 is DOWN: PING CRITICAL - Packet loss = 100% [11:26:46] PROBLEM - BFD status on cloudsw1-d5-eqiad.mgmt is CRITICAL: Down: 1 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [11:27:14] RECOVERY - Host cloudlb1002 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [11:27:46] RECOVERY - BFD status on cloudsw1-d5-eqiad.mgmt is OK: UP: 16 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [11:27:48] (03PS1) 10Kevin Bazira: ml-services: update the tts isvc to autoscale with demand [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357247 [11:28:33] !log kamila@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [11:28:35] !log kamila@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [11:28:47] FIRING: [7x] CoreBGPDown: Core BGP session down between cloudsw1-c8-eqiad and cloudlb1001 (172.20.1.2) - group cloud_host - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [11:28:47] !log kamila@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [11:28:52] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:29:38] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2214.codfw.wmnet with reason: Maintenance [11:29:40] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2214 (T441633)', diff saved to https://phabricator.wikimedia.org/P96875 and previous config saved to /var/cache/conftool/dbconfig/20261020-112940-ladsgroup.json [11:30:09] (03CR) 10Kevin Bazira: [C:03+2] ml-services: update the tts isvc to autoscale with demand [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357247 (owner: 10Kevin Bazira) [11:32:21] (03Merged) 10jenkins-bot: ml-services: update the tts isvc to autoscale with demand [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357247 (owner: 10Kevin Bazira) [11:33:47] FIRING: [7x] CoreBGPDown: Core BGP session down between cloudsw1-c8-eqiad and cloudlb1001 (172.20.1.2) - group cloud_host - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [11:33:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:35:32] !log kamila@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [11:35:40] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [11:35:44] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [11:36:34] PROBLEM - BFD status on lsw1-f2-eqiad.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [11:36:35] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2214 (T441633)', diff saved to https://phabricator.wikimedia.org/P96876 and previous config saved to /var/cache/conftool/dbconfig/20261020-113634-ladsgroup.json [11:37:48] !log btullis@cumin1004 START - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash rolling reboot on A:apifeatureusage [11:37:59] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [11:38:02] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . [11:38:51] RESOLVED: [3x] ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from gitlab.discovery.wmnet in codfw #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [11:40:34] RECOVERY - BFD status on lsw1-f2-eqiad.mgmt is OK: UP: 4 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [11:43:36] PROBLEM - Host an-worker1200 is DOWN: PING CRITICAL - Packet loss = 100% [11:43:40] !log btullis@cumin1004 END (PASS) - Cookbook sre.ceph.roll-restart-reboot-server (exit_code=0) rolling reboot on A:cephosd-eqiad [11:43:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:43:51] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1156.eqiad.wmnet - https://phabricator.wikimedia.org/T441298#12453799 (10Jclark-ctr) a:03Jclark-ctr [11:44:39] !log cwilliams@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on backup1014.eqiad.wmnet with reason: Maintenance [11:45:26] !log cwilliams@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on 8 hosts with reason: Maintenance [11:45:42] !log slyngshede@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp7009.magru.wmnet [11:45:43] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_magru and not P{cp7002*} and not P{cp7010*} and A:cp [11:46:37] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P96877 and previous config saved to /var/cache/conftool/dbconfig/20261020-114636-ladsgroup.json [11:46:46] RECOVERY - Host an-worker1200 is UP: PING OK - Packet loss = 0%, RTA = 0.14 ms [11:47:34] !log filippo@cumin1004 START - Cookbook sre.hosts.reboot-single for host cloudvirtlocal1001.eqiad.wmnet [11:48:26] PROBLEM - SSH on an-worker1200 is CRITICAL: connect to address 10.64.143.6 and port 22: Connection refused https://wikitech.wikimedia.org/wiki/SSH/monitoring [11:48:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:49:33] !log btullis@cumin1004 END (PASS) - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash (exit_code=0) rolling reboot on A:apifeatureusage [11:54:14] !log filippo@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirtlocal1001.eqiad.wmnet [11:55:42] FIRING: JobUnavailable: Reduced availability for job bacula in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:56:03] ^ maintenance [11:56:36] !log cwilliams@cumin1004 START - Cookbook sre.hosts.reboot-single for host backup1008.eqiad.wmnet [11:56:38] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P96878 and previous config saved to /var/cache/conftool/dbconfig/20261020-115638-ladsgroup.json [11:57:45] Dear deployers, I am about to depool mw-api-ext-ro from eqiad to test the waters a bit Depending on how those will go, I will move to mw-web or not. [11:57:53] jouncebot: next [11:57:53] In 0 hour(s) and 2 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1200) [11:58:00] jouncebot: now [11:58:00] No deployments scheduled for the next 0 hour(s) and 1 minute(s) [11:58:34] !log jiji@cumin1004 START - Cookbook sre.discovery.service-route depool mw-api-ext-ro in eqiad: T440896 [11:58:39] T440896: Depool eqiad from reads to load test codfw - https://phabricator.wikimedia.org/T440896 [11:59:12] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1156.eqiad.wmnet - https://phabricator.wikimedia.org/T441298#12453836 (10Jclark-ctr) [11:59:19] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops, 10decommission-hardware: decommission db1156.eqiad.wmnet - https://phabricator.wikimedia.org/T441298#12453837 (10Jclark-ctr) 05Open→03Resolved [11:59:32] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1008.eqiad.wmnet [11:59:35] !log cwilliams@cumin1004 START - Cookbook sre.hosts.reboot-single for host backup1009.eqiad.wmnet [12:00:04] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1200) [12:00:05] 10ops-eqiad, 06SRE, 06DC-Ops, 10decommission-hardware, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Decommission krb1003 - https://phabricator.wikimedia.org/T441019#12453839 (10Jclark-ctr) 05In progress→03Resolved [12:00:09] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:00:12] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:02:52] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1009.eqiad.wmnet [12:02:55] !log cwilliams@cumin1004 START - Cookbook sre.hosts.reboot-single for host backup1012.eqiad.wmnet [12:03:17] btullis@cumin1004 reboot-workers (PID 1915313) is awaiting input [12:03:38] !log jiji@cumin1004 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool mw-api-ext-ro in eqiad: T440896 [12:03:45] T440896: Depool eqiad from reads to load test codfw - https://phabricator.wikimedia.org/T440896 [12:04:49] !log filippo@cumin1004 START - Cookbook sre.hosts.reboot-single for host cloudvirtlocal1002.eqiad.wmnet [12:06:40] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2214 (T441633)', diff saved to https://phabricator.wikimedia.org/P96879 and previous config saved to /var/cache/conftool/dbconfig/20261020-120640-ladsgroup.json [12:07:00] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2217.codfw.wmnet with reason: Maintenance [12:07:02] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2217 (T441633)', diff saved to https://phabricator.wikimedia.org/P96880 and previous config saved to /var/cache/conftool/dbconfig/20261020-120701-ladsgroup.json [12:07:32] (03PS3) 10Btullis: deployment_server: Restrict airflow kubeconfigs to team groups [puppet] - 10https://gerrit.wikimedia.org/r/1356996 (https://phabricator.wikimedia.org/T429558) [12:07:32] (03PS3) 10Btullis: admin: Make airflow-deployers a managed list [puppet] - 10https://gerrit.wikimedia.org/r/1356997 (https://phabricator.wikimedia.org/T429558) [12:07:47] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1012.eqiad.wmnet [12:07:50] !log cwilliams@cumin1004 START - Cookbook sre.hosts.reboot-single for host backup2008.codfw.wmnet [12:08:26] RECOVERY - SSH on an-worker1200 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u10 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring [12:09:36] 10ops-codfw, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T441801 (10phaultfinder) 03NEW [12:10:54] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbproxy1024.eqiad.wmnet with reason: Reboot [12:10:56] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2008.codfw.wmnet [12:10:59] !log cwilliams@cumin1004 START - Cookbook sre.hosts.reboot-single for host backup2009.codfw.wmnet [12:11:37] !log filippo@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirtlocal1002.eqiad.wmnet [12:13:45] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2009.codfw.wmnet [12:13:49] !log cwilliams@cumin1004 START - Cookbook sre.hosts.reboot-single for host backup2012.codfw.wmnet [12:13:54] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2217 (T441633)', diff saved to https://phabricator.wikimedia.org/P96881 and previous config saved to /var/cache/conftool/dbconfig/20261020-121354-ladsgroup.json [12:14:02] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbproxy1023.eqiad.wmnet with reason: Reboot [12:14:16] !log filippo@cumin1004 START - Cookbook sre.hosts.reboot-single for host cloudvirtlocal1003.eqiad.wmnet [12:16:46] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2012.codfw.wmnet [12:16:50] !log cwilliams@cumin1004 START - Cookbook sre.hosts.reboot-single for host backup2014.codfw.wmnet [12:19:12] (03PS1) 10Marostegui: wmnet: Switchover m3-master [dns] - 10https://gerrit.wikimedia.org/r/1357263 (https://phabricator.wikimedia.org/T441633) [12:20:51] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2014.codfw.wmnet [12:21:06] !log filippo@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirtlocal1003.eqiad.wmnet [12:23:13] (03PS1) 10DCausse: Add cirrus-streaming-updater defition to scap deploy-service [puppet] - 10https://gerrit.wikimedia.org/r/1357266 [12:23:43] (03PS2) 10DCausse: Add cirrus-streaming-updater definition to scap deploy-service [puppet] - 10https://gerrit.wikimedia.org/r/1357266 [12:23:56] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P96882 and previous config saved to /var/cache/conftool/dbconfig/20261020-122356-ladsgroup.json [12:23:59] !log cwilliams@cumin1004 START - Cookbook sre.hosts.reboot-single for host backup1014.eqiad.wmnet [12:24:44] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host ceph-admin1001.eqiad.wmnet [12:26:06] (03PS3) 10DCausse: Add cirrus-streaming-updater definition to scap deploy-service [puppet] - 10https://gerrit.wikimedia.org/r/1357266 [12:26:22] (03CR) 10Trueg: [C:03+1] WDQS: bump chart version 0.1.0 -> 0.1.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357244 (https://phabricator.wikimedia.org/T440560) (owner: 10Gmodena) [12:28:16] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1014.eqiad.wmnet [12:28:32] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ceph-admin1001.eqiad.wmnet [12:29:39] !log btullis@cumin1004 START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-jumbo-eqiad [12:32:07] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dbstore1010.eqiad.wmnet [12:33:52] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:33:58] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P96883 and previous config saved to /var/cache/conftool/dbconfig/20261020-123357-ladsgroup.json [12:37:24] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dbstore1010.eqiad.wmnet [12:38:47] FIRING: [18x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:104:10:192:48:14 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [12:38:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:40:38] (03CR) 10Elukey: [C:03+2] profile::pki::intermediates: update kafka's public cert [puppet] - 10https://gerrit.wikimedia.org/r/1354357 (https://phabricator.wikimedia.org/T438703) (owner: 10Elukey) [12:43:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:43:56] PROBLEM - Host cloudweb1003 is DOWN: PING CRITICAL - Packet loss = 100% [12:44:00] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2217 (T441633)', diff saved to https://phabricator.wikimedia.org/P96884 and previous config saved to /var/cache/conftool/dbconfig/20261020-124359-ladsgroup.json [12:44:20] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2224.codfw.wmnet with reason: Maintenance [12:44:22] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2224 (T441633)', diff saved to https://phabricator.wikimedia.org/P96885 and previous config saved to /var/cache/conftool/dbconfig/20261020-124421-ladsgroup.json [12:44:26] RECOVERY - Host cloudweb1003 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [12:49:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:49:57] FIRING: ProbeDown: Service pki:443 has failed probes (http_pki_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#pki:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:51:13] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2224 (T441633)', diff saved to https://phabricator.wikimedia.org/P96886 and previous config saved to /var/cache/conftool/dbconfig/20261020-125112-ladsgroup.json [12:51:35] !incidents [12:51:35] 8629 (ACKED) ProbeDown sre (10.2.2.96 ip4 pki:443 probes/service http_pki_ip4 eqiad) [12:51:36] 8628 (RESOLVED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet eqiad) [12:51:36] 8627 (RESOLVED) NELHigh sre (thanos-rule@main tcp.timed_out) [12:51:36] 8626 (RESOLVED) NELHigh sre (thanos-rule@main tcp.timed_out) [12:51:36] 8625 (RESOLVED) ProbeDown sre (2a02:ec80:300:ed1a::1 ip6 text-https:443 probes/service http_text-https_ip6 esams) [12:52:11] <_joe_> elukey: is that you right? [12:52:33] FIRING: [3x] ProbeDown: Service kafka-jumbo1010:9093 has failed probes (tcp_kafka_broker_tls_kafka_jumbo1010_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:53:41] _joe_ yeah not sure why [12:54:27] no issues on the pki nodes so this is a current mistery, no idea why jumbo has problems [12:54:28] checking [12:55:25] <_joe_> do you need me to jump out of meetings and help? [12:55:26] ferm was down for an unrelated issue afaics [12:55:30] !log filippo@cumin1004 START - Cookbook sre.hosts.reboot-single for host cloudrabbit1001.eqiad.wmnet [12:55:31] nono thanks [12:55:41] I disabled puppet everywhere for kafka [12:56:55] jouncebot: next [12:56:55] In 0 hour(s) and 3 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1300) [12:57:33] RESOLVED: ProbeDown: Service pki:443 has failed probes (http_pki_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#pki:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:58:03] !log sukhe@cumin1004 START - Cookbook sre.dns.roll-reboot rolling reboot on P{dns7002* or dns6001* or dns4003*} and (A:dnsbox) [12:58:03] !log sukhe@cumin1004 cookbooks.sre.dns.roll-reboot begin reboot of dns4003.wikimedia.org [12:58:41] (03PS1) 10Kamila Součková: deployment_server: switch mw-parsoid to PHP 8.5 [puppet] - 10https://gerrit.wikimedia.org/r/1357273 (https://phabricator.wikimedia.org/T440321) [13:00:04] Lucas_WMDE, urbanecm, and TheresNoTime: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1300). [13:00:04] No Gerrit patches in the queue for this window AFAICS. [13:00:26] !log jiji@deploy1003 helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply [13:00:56] !log jiji@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply [13:01:11] yup, nothing to deploy [13:01:15] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P96887 and previous config saved to /var/cache/conftool/dbconfig/20261020-130114-ladsgroup.json [13:01:18] !log filippo@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1001.eqiad.wmnet [13:02:06] Lucas_WMDE: grand! [13:02:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [13:02:16] (03CR) 10Kamila Součková: "I will deploy this after parsoid RT testing on 8.3 / 8.5 completed successfully (and after reverting I586925c47b6b361ad2c328dc660c9871edf0" [puppet] - 10https://gerrit.wikimedia.org/r/1357273 (https://phabricator.wikimedia.org/T440321) (owner: 10Kamila Součková) [13:02:33] FIRING: ProbeDown: Service pki:443 has failed probes (http_pki_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#pki:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:03:33] !log filippo@cumin1004 START - Cookbook sre.hosts.reboot-single for host cloudrabbit1002.eqiad.wmnet [13:04:35] (03CR) 10Jelto: [C:03+2] Add cirrus-streaming-updater definition to scap deploy-service [puppet] - 10https://gerrit.wikimedia.org/r/1357266 (owner: 10DCausse) [13:08:47] FIRING: [3x] GnmiInterfaceCountersDrop: lsw1-a5-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [13:09:13] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host druid-internal1001.eqiad.wmnet [13:09:39] !log filippo@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1002.eqiad.wmnet [13:10:18] (03PS1) 10Elukey: service: simplify pki's probe [puppet] - 10https://gerrit.wikimedia.org/r/1357279 (https://phabricator.wikimedia.org/T438703) [13:10:41] !log filippo@cumin1004 START - Cookbook sre.hosts.reboot-single for host cloudrabbit1003.eqiad.wmnet [13:11:17] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P96888 and previous config saved to /var/cache/conftool/dbconfig/20261020-131116-ladsgroup.json [13:12:33] RESOLVED: ProbeDown: Service pki:443 has failed probes (http_pki_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#pki:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:13:53] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [13:14:25] (03CR) 10Btullis: [C:03+1] "Looks good to me." [puppet] - 10https://gerrit.wikimedia.org/r/1357279 (https://phabricator.wikimedia.org/T438703) (owner: 10Elukey) [13:14:28] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid-internal1001.eqiad.wmnet [13:14:31] (03CR) 10Xcollazo: [C:03+1] kafka-jumbo: declare Data Engineering topics (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1355378 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [13:14:36] !log sukhe@cumin1004 cookbooks.sre.dns.roll-reboot finished rebooting dns4003.wikimedia.org [13:16:47] !log filippo@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1003.eqiad.wmnet [13:19:26] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host druid-internal1002.eqiad.wmnet [13:20:04] !log klausman@cumin1004 START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2001.codfw.wmnet [13:20:38] (03CR) 10FNegri: [C:03+1] "Thank you, LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/1357243 (https://phabricator.wikimedia.org/T438200) (owner: 10Marostegui) [13:21:11] (03PS5) 10Blake: cronjob: s/cache.mcrouter.deployment/cache.mcrouter.container/ [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356869 (https://phabricator.wikimedia.org/T386694) [13:21:18] (03CR) 10Blake: cronjob: s/cache.mcrouter.deployment/cache.mcrouter.container/ (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356869 (https://phabricator.wikimedia.org/T386694) (owner: 10Blake) [13:21:19] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2224 (T441633)', diff saved to https://phabricator.wikimedia.org/P96889 and previous config saved to /var/cache/conftool/dbconfig/20261020-132118-ladsgroup.json [13:22:31] !log klausman@cumin1004 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2001.codfw.wmnet [13:22:51] !log klausman@cumin1004 START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2002.codfw.wmnet [13:25:03] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid-internal1002.eqiad.wmnet [13:25:08] (03CR) 10Elukey: [C:03+2] service: simplify pki's probe [puppet] - 10https://gerrit.wikimedia.org/r/1357279 (https://phabricator.wikimedia.org/T438703) (owner: 10Elukey) [13:25:11] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host druid-internal1003.eqiad.wmnet [13:25:17] !log klausman@cumin1004 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2002.codfw.wmnet [13:25:27] !log klausman@cumin1004 START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2003.codfw.wmnet [13:26:56] (03CR) 10Marostegui: [C:03+2] wikireplicas: Use jemalloc [puppet] - 10https://gerrit.wikimedia.org/r/1357243 (https://phabricator.wikimedia.org/T438200) (owner: 10Marostegui) [13:27:21] !log elukey@cumin1004 START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-test-eqiad [13:27:50] !log klausman@cumin1004 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2003.codfw.wmnet [13:27:57] !log klausman@cumin1004 START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-ctrl2001.codfw.wmnet [13:29:36] !log sukhe@cumin1004 cookbooks.sre.dns.roll-reboot begin reboot of dns6001.wikimedia.org [13:29:56] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Restart [13:30:33] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid-internal1003.eqiad.wmnet [13:31:20] !log klausman@cumin1004 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-ctrl2001.codfw.wmnet [13:31:40] PROBLEM - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-presto1006 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 1 Failed : virtual_disk: 1 Dgrd : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [13:31:42] ACKNOWLEDGEMENT - Dell PowerEdge or Supermicro Broadcom RAID Controller on an-presto1006 is CRITICAL: communication: 0 OK : controller: 1 Needs Attention : physical_disk: 1 Failed : virtual_disk: 1 Dgrd : bbu: 0 OK : enclosure: 0 OK : CLI Version = 007.1910.0000.0000 Oct 08, 2021 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T441831 https://wikitech.wikimedia.org/wiki/PERCCli%23Monitoring [13:31:48] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-presto1006 - https://phabricator.wikimedia.org/T441831 (10ops-monitoring-bot) 03NEW [13:31:53] !log klausman@cumin1004 START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-ctrl2002.codfw.wmnet [13:32:33] FIRING: ProbeDown: Service pki:443 has failed probes (http_pki_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#pki:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:33:47] FIRING: [6x] BFDdown: BFD session down between asw1-b12-drmrs and 185.15.58.5 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:33:50] PROBLEM - BFD status on asw1-b12-drmrs.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [13:33:52] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:33:57] (03PS1) 10Bartosz Wójtowicz: ml-services: deploy gpt-oss-safeguard-20b LLMInferenceService [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357282 (https://phabricator.wikimedia.org/T441822) [13:34:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:34:50] RECOVERY - BFD status on asw1-b12-drmrs.mgmt is OK: UP: 7 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [13:34:57] RESOLVED: ProbeDown: Service pki:443 has failed probes (http_pki_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#pki:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:35:17] !log klausman@cumin1004 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-ctrl2002.codfw.wmnet [13:35:34] !log klausman@cumin1004 START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2001.codfw.wmnet [13:37:08] !log upload liberica 0.28 to trixie-wikimedia (apt.wm.o) - T401832 [13:37:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:37:14] T401832: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832 [13:37:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [13:37:33] RESOLVED: ProbeDown: Service pki:443 has failed probes (http_pki_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#pki:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [13:38:47] FIRING: [6x] BFDdown: BFD session down between asw1-b12-drmrs and 185.15.58.5 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [13:38:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:38:51] !log ayounsi@cumin1004 START - Cookbook sre.network.peering with action 'configure' for AS: 44034 [13:39:27] !log ayounsi@cumin1004 END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'configure' for AS: 44034 [13:39:33] !log klausman@cumin1004 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2001.codfw.wmnet [13:39:49] 06SRE, 10docker-pkg, 10Projects-Cleanup: Move docker-pkg to gitlab, archive gerrit repo - https://phabricator.wikimedia.org/T441088#12454608 (10hashar) [13:40:40] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reimage for host lvs7003.magru.wmnet with OS trixie [13:41:26] !log klausman@cumin1004 START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2002.codfw.wmnet [13:43:51] !log klausman@cumin1004 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2002.codfw.wmnet [13:43:57] !log klausman@cumin1004 START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2003.codfw.wmnet [13:45:35] !log sukhe@cumin1004 cookbooks.sre.dns.roll-reboot finished rebooting dns6001.wikimedia.org [13:46:21] !log klausman@cumin1004 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2003.codfw.wmnet [13:46:51] !log elukey@cumin1004 END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-test-eqiad [13:47:01] !log klausman@cumin1004 START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1001.eqiad.wmnet [13:48:12] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-presto1006 - https://phabricator.wikimedia.org/T441831#12454666 (10Jclark-ctr) a:03Jclark-ctr This server is out of warranty. I can check to see if we have any 4 TB SAS drives available, but I don't believe we have any from our decommissioned servers. Plea... [13:48:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:48:53] (03CR) 10Kevin Bazira: [C:03+1] ml-services: deploy gpt-oss-safeguard-20b LLMInferenceService [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357282 (https://phabricator.wikimedia.org/T441822) (owner: 10Bartosz Wójtowicz) [13:49:23] !log klausman@cumin1004 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1001.eqiad.wmnet [13:49:32] !log klausman@cumin1004 START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1002.eqiad.wmnet [13:51:42] !log klausman@cumin1004 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1002.eqiad.wmnet [13:51:48] !log dkertesz@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P{cp7001.*} and A:cp - upgrade haproxy to 3.2.26-1 (T441435) [13:51:54] T441435: Upgrade to HAProxy 3.2.26 - https://phabricator.wikimedia.org/T441435 [13:52:34] !log klausman@cumin1004 START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1003.eqiad.wmnet [13:53:38] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: deploy gpt-oss-safeguard-20b LLMInferenceService [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357282 (https://phabricator.wikimedia.org/T441822) (owner: 10Bartosz Wójtowicz) [13:53:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:54:39] !log elukey@cumin1004 START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-codfw [13:54:56] !log klausman@cumin1004 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1003.eqiad.wmnet [13:55:05] (03Merged) 10jenkins-bot: ml-services: deploy gpt-oss-safeguard-20b LLMInferenceService [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357282 (https://phabricator.wikimedia.org/T441822) (owner: 10Bartosz Wójtowicz) [13:55:15] !log klausman@cumin1004 START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl1001.eqiad.wmnet [13:56:19] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-presto1006 - https://phabricator.wikimedia.org/T441831#12454724 (10Jclark-ctr) We do not have any of these drives in stock from spare servers [13:56:29] (03PS1) 10Arnaudb: wdqs: fetch the allowlist through urldownloader [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357290 (https://phabricator.wikimedia.org/T441768) [13:58:11] !log a-pizzata@deploy1003 Started deploy [analytics/refinery@68687dc] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@68687dc4] [13:58:19] (03PS1) 10Effie Mouzeli: mw deployments: add codfw replicas for forthcoming switchover [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357291 (https://phabricator.wikimedia.org/T439674) [13:58:38] !log klausman@cumin1004 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl1001.eqiad.wmnet [13:58:44] !log klausman@cumin1004 START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl1002.eqiad.wmnet [13:58:58] !log a-pizzata@deploy1003 Finished deploy [analytics/refinery@68687dc] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@68687dc4] (duration: 00m 46s) [13:59:14] !log a-pizzata@deploy1003 Started deploy [analytics/refinery@68687dc]: Regular analytics weekly train [analytics/refinery@68687dc4] [13:59:16] (03Abandoned) 10Hashar: Add option to use the public hostname of a registry [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1178068 (https://phabricator.wikimedia.org/T401733) (owner: 10Hashar) [13:59:18] (03Abandoned) 10Hashar: Print log file for any action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1337551 (owner: 10Hashar) [13:59:20] (03Abandoned) 10Hashar: Normalize root path to avoid extra . or / [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1337559 (owner: 10Hashar) [13:59:23] (03Abandoned) 10Hashar: Log info to stdout when run interactively [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1337562 (owner: 10Hashar) [13:59:26] 06SRE, 06Collaboration-Services, 10Infrastructure Security, 06Infrastructure-Foundations, and 7 others: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838 (10bking) 03NEW [13:59:29] (03Abandoned) 10Hashar: Change update to exactly match the given image name [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1166856 (owner: 10Hashar) [13:59:33] (03Abandoned) 10Hashar: dockerfile: Add verify_stage template function [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1354049 (https://phabricator.wikimedia.org/T439490) (owner: 10Dduvall) [14:00:05] Deploy window Test Kitchen UI Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1400) [14:00:10] 06SRE, 06Collaboration-Services, 10Infrastructure Security, 06Infrastructure-Foundations, and 7 others: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838#12454745 (10bking) 05Open→03In progress p:05Triage→03Medium a:03bking [14:00:35] !log sukhe@cumin1004 cookbooks.sre.dns.roll-reboot begin reboot of dns7002.wikimedia.org [14:00:36] (03PS1) 10Arnaudb: wdqs: fetch the allowlist through urldownloader [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357290 (https://phabricator.wikimedia.org/T441768) [14:00:55] (03PS1) 10Arnaudb: wdqs: fetch the allowlist from internal GitLab [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357292 (https://phabricator.wikimedia.org/T441768) [14:01:42] (03CR) 10Hashar: Refactor tests to run inline with build (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345668 (https://phabricator.wikimedia.org/T439490) (owner: 10Dduvall) [14:02:06] (03CR) 10Trueg: [C:03+2] WDQS: bump chart version 0.1.0 -> 0.1.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357244 (https://phabricator.wikimedia.org/T440560) (owner: 10Gmodena) [14:02:06] !log klausman@cumin1004 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl1002.eqiad.wmnet [14:03:07] (03CR) 10Lerickson: [C:03+1] WDQS: bump chart version 0.1.0 -> 0.1.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357244 (https://phabricator.wikimedia.org/T440560) (owner: 10Gmodena) [14:03:08] !log klausman@cumin1004 START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2001.codfw.wmnet [14:03:40] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on lvs7003.magru.wmnet with reason: host reimage [14:03:43] (03Merged) 10jenkins-bot: WDQS: bump chart version 0.1.0 -> 0.1.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357244 (https://phabricator.wikimedia.org/T440560) (owner: 10Gmodena) [14:03:47] FIRING: [16x] GnmiInterfaceCountersDrop: cr1-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:04:16] FIRING: [16x] GnmiInterfaceCountersDrop: cr1-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:04:44] PROBLEM - BFD status on asw1-b4-magru.mgmt is CRITICAL: Down: 2 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [14:04:53] !log a-pizzata@deploy1003 Finished deploy [analytics/refinery@68687dc]: Regular analytics weekly train [analytics/refinery@68687dc4] (duration: 05m 39s) [14:06:44] RECOVERY - BFD status on asw1-b4-magru.mgmt is OK: UP: 2 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [14:06:54] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [14:07:15] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [14:08:04] !log klausman@cumin1004 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl2001.codfw.wmnet [14:08:13] !log klausman@cumin1004 START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2002.codfw.wmnet [14:08:25] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs7003.magru.wmnet with reason: host reimage [14:08:46] 06SRE, 10SRE-swift-storage: Remove obsolete schemas from hosts.yaml - https://phabricator.wikimedia.org/T441841 (10MatthewVernon) 03NEW [14:08:47] FIRING: [6x] BFDdown: BFD session down between asw1-b4-magru and 195.200.68.37 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [14:08:47] FIRING: [18x] GnmiInterfaceCountersDrop: cr1-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:09:16] (03PS1) 10MVernon: swift: add ms-be1[098-100] to the rings [puppet] - 10https://gerrit.wikimedia.org/r/1357295 (https://phabricator.wikimedia.org/T424895) [14:09:33] 06SRE, 06Collaboration-Services, 10Infrastructure Security, 06Infrastructure-Foundations, and 6 others: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838#12454810 (10ssingh) [14:10:05] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [14:10:23] !log sukhe@cumin1004 cookbooks.sre.dns.roll-reboot finished rebooting dns7002.wikimedia.org [14:10:24] !log sukhe@cumin1004 END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on P{dns7002* or dns6001* or dns4003*} and (A:dnsbox) [14:10:45] !log hashar@deploy1003 Started deploy [integration/docroot@5cf09c8]: opensource: docker-pkg has moved to GitLab - T441088 [14:10:50] T441088: Move docker-pkg to gitlab, archive gerrit repo - https://phabricator.wikimedia.org/T441088 [14:10:55] !log hashar@deploy1003 Finished deploy [integration/docroot@5cf09c8]: opensource: docker-pkg has moved to GitLab - T441088 (duration: 00m 10s) [14:11:15] 06SRE, 06Collaboration-Services, 10Infrastructure Security, 06Infrastructure-Foundations, and 5 others: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838#12454822 (10taavi) Not sure why we were tagged here? [14:11:17] (03PS1) 10Trueg: WDQSv2: Bump wdqs-proxy to 0.14.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357296 [14:11:53] !log elukey@cumin1004 END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-codfw [14:12:04] (03PS1) 10CDanis: cache::haproxy: Remove X-Provenance sent by clients [puppet] - 10https://gerrit.wikimedia.org/r/1357297 [14:12:30] jouncebot: now [14:12:30] For the next 0 hour(s) and 17 minute(s): Test Kitchen UI Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1400) [14:12:34] jouncebot: next [14:12:34] In 0 hour(s) and 17 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1430) [14:12:40] !log elukey@cumin1004 START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-eqiad [14:13:11] !log klausman@cumin1004 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl2002.codfw.wmnet [14:13:50] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [14:14:08] 06SRE, 10Infrastructure Security, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), and 3 others: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838#12454844 (10bking) [14:14:57] 06SRE, 10Infrastructure Security, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), and 3 others: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838#12454857 (10bking) Apologies to all, I think those tags were inherited from the parent. I've just r... [14:15:24] !log klausman@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-staging-worker [14:15:26] !log klausman@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2001.codfw.wmnet [14:17:24] !log hashar@deploy1003 Started deploy [integration/docroot@be38d5a]: opensource: docker-pkg has moved to GitLab - T441088 [14:17:30] T441088: Move docker-pkg to gitlab, archive gerrit repo - https://phabricator.wikimedia.org/T441088 [14:17:34] !log hashar@deploy1003 Finished deploy [integration/docroot@be38d5a]: opensource: docker-pkg has moved to GitLab - T441088 (duration: 00m 10s) [14:17:44] (03PS6) 10Ayounsi: WIP: Add cookbook to sync BGP sessions with Netbox [cookbooks] - 10https://gerrit.wikimedia.org/r/1347441 (https://phabricator.wikimedia.org/T439408) [14:18:45] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [14:18:45] !log bking@cumin2003 END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) [14:18:58] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [14:18:59] !log bking@cumin2003 END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) [14:19:08] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [14:19:10] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [14:19:35] (03CR) 10Effie Mouzeli: [C:03+2] "decided on irc to go for it" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357291 (https://phabricator.wikimedia.org/T439674) (owner: 10Effie Mouzeli) [14:19:36] (03CR) 10Vgutierrez: [C:03+1] "nice catch" [puppet] - 10https://gerrit.wikimedia.org/r/1357297 (owner: 10CDanis) [14:19:43] 06SRE, 10docker-pkg, 10Projects-Cleanup, 13Patch-For-Review: Move docker-pkg to gitlab, archive gerrit repo - https://phabricator.wikimedia.org/T441088#12454898 (10hashar) 05Open→03Resolved a:03hashar I have: - amended https://gerrit.wikimedia.org/r/c/integration/config/+/1354921 and deployed it... [14:20:04] (03CR) 10CDanis: [C:03+2] "another Claude driveby while digging into something else 🫠" [puppet] - 10https://gerrit.wikimedia.org/r/1357297 (owner: 10CDanis) [14:20:19] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [14:20:28] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [14:20:31] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . [14:21:12] (03Merged) 10jenkins-bot: mw deployments: add codfw replicas for forthcoming switchover [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357291 (https://phabricator.wikimedia.org/T439674) (owner: 10Effie Mouzeli) [14:22:48] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [14:24:30] !log elukey@cumin1004 START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-codfw [14:25:32] !log klausman@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2001.codfw.wmnet [14:25:34] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [14:26:39] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [14:28:47] FIRING: KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1152.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [14:29:12] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [14:29:21] !log ayounsi@cumin1004 START - Cookbook sre.network.sync-bgp-netbox [14:29:21] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.sync-bgp-netbox (exit_code=0) [14:29:29] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [14:29:38] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [14:29:43] !log jiji@deploy1003 helmfile [codfw] START helmfile.d/services/mw-api-ext: apply [14:29:59] !log jiji@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1430) [14:30:19] !log klausman@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2001.codfw.wmnet [14:30:21] !log klausman@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2001.codfw.wmnet [14:30:24] !log klausman@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2002.codfw.wmnet [14:30:56] (03CR) 10CWilliams: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1357295 (https://phabricator.wikimedia.org/T424895) (owner: 10MVernon) [14:32:05] !log elukey@cumin1004 END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-eqiad [14:32:08] (03CR) 10MVernon: [C:03+2] swift: add ms-be1[098-100] to the rings [puppet] - 10https://gerrit.wikimedia.org/r/1357295 (https://phabricator.wikimedia.org/T424895) (owner: 10MVernon) [14:34:11] !log jiji@deploy1003 helmfile [codfw] START helmfile.d/services/mw-web: apply [14:34:23] (03CR) 10Ottomata: EventStreamConfig: Declare the mediawiki.user_thank.v1 stream (033 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356976 (https://phabricator.wikimedia.org/T441426) (owner: 10Snwachukwu) [14:34:31] !log jiji@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-web: apply [14:34:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:34:31] (03CR) 10Ottomata: [C:04-1] EventStreamConfig: Declare the mediawiki.user_thank.v1 stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356976 (https://phabricator.wikimedia.org/T441426) (owner: 10Snwachukwu) [14:37:12] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [14:37:29] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [14:37:50] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Degraded RAID on an-presto1006 - https://phabricator.wikimedia.org/T441831#12455007 (10Jclark-ctr) [14:37:52] !log cscott@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [14:37:56] !log cscott@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [14:38:16] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [14:38:34] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Eqiad worker nodes should be connected to 10G NICs - https://phabricator.wikimedia.org/T440696#12455010 (10JMeybohm) [14:39:17] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [14:39:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:40:14] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Degraded RAID on an-presto1006 - https://phabricator.wikimedia.org/T441831#12455022 (10Jclark-ctr) @RobH if Data platform wants this drive replaced it will have to be ordered like previous 4tb sas drives >>! In T434505#12204573... [14:40:32] !log klausman@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2002.codfw.wmnet [14:40:44] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs7003.magru.wmnet with OS trixie [14:41:14] 06SRE, 10Infrastructure Security, 06Infrastructure-Foundations, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), and 3 others: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838#12455028 (10bking) [14:41:30] !log upload ipip-multiqueue-optimizer 0.3+deb13u1 to trixie-wikimedia (apt.wm.o) - T401832 [14:41:32] (03CR) 10Arnaudb: [C:04-1] "do not merge this yet" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357292 (https://phabricator.wikimedia.org/T441768) (owner: 10Arnaudb) [14:41:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:41:35] T401832: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832 [14:43:51] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [14:44:16] !log elukey@cumin1004 END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-codfw [14:44:34] jouncebot: next [14:44:34] In 0 hour(s) and 15 minute(s): SRE Collaboration Services office hours (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1500) [14:44:39] jouncebot: now [14:44:39] For the next 0 hour(s) and 15 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1430) [14:45:05] headsup, I will attempt to depool the web traffic from eqiad [14:45:22] !log klausman@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2002.codfw.wmnet [14:45:24] !log klausman@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2002.codfw.wmnet [14:45:27] !log klausman@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2003.codfw.wmnet [14:45:47] !log elukey@cumin1004 START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-eqiad [14:47:29] !log jiji@cumin1004 START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: T440896 [14:47:35] T440896: Depool eqiad from reads to load test codfw - https://phabricator.wikimedia.org/T440896 [14:48:03] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [14:49:01] FIRING: [4x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:49:05] RESOLVED: [3x] PKICertificateExpiry: Intermediate certificate in the trust chain for kafka expires in 6d 22h 21m 25s - https://wikitech.wikimedia.org/wiki/PKI/CA_Operations - TODO - https://alerts.wikimedia.org/?q=alertname%3DPKICertificateExpiry [14:51:10] (03CR) 10Hnowlan: [C:03+2] logstash/rsyslog: route php8 versions correctly [puppet] - 10https://gerrit.wikimedia.org/r/1356954 (owner: 10Hnowlan) [14:51:15] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Degraded RAID on an-presto1006 - https://phabricator.wikimedia.org/T441831#12455095 (10RobH) >>! In T441831#12455020, @Jclark-ctr wrote: > @RobH if Data platform wants this drive replaced it will have to be ordered like previous... [14:52:11] !log dkertesz@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P{cp7001.*} and A:cp - upgrade haproxy to 3.2.26-1 (T441435) [14:52:17] T441435: Upgrade to HAProxy 3.2.26 - https://phabricator.wikimedia.org/T441435 [14:52:32] !log jiji@cumin1004 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool mw-web-ro in eqiad: T440896 [14:52:38] T440896: Depool eqiad from reads to load test codfw - https://phabricator.wikimedia.org/T440896 [14:53:47] FIRING: [4x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:55:37] !log klausman@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2003.codfw.wmnet [14:56:16] (03PS1) 10Trueg: WDQSv2: fetch the allowlist from internal GitLab [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357305 (https://phabricator.wikimedia.org/T441768) [14:56:33] (03Abandoned) 10Trueg: WDQSv2: Bump wdqs-proxy to 0.14.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357296 (owner: 10Trueg) [14:57:18] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin depooling P{lvs7002.*} and A:liberica [14:57:23] !log depooled eqiad for mw-web T440896 [14:57:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:57:30] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7002.*} and A:liberica [14:57:46] !log cscott@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [14:57:49] !log cscott@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [14:58:15] (03CR) 10Lerickson: [C:03+1] WDQSv2: fetch the allowlist from internal GitLab [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357305 (https://phabricator.wikimedia.org/T441768) (owner: 10Trueg) [14:58:43] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [14:59:17] 06SRE, 06Infrastructure-Foundations: Upgrade rubocop ruby linter on operations/puppet - https://phabricator.wikimedia.org/T441847 (10hashar) 03NEW [14:59:35] (03CR) 10Jelto: gitlab: restrict 443 on the service IP to Prometheus probes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1357232 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [15:00:04] !log klausman@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2003.codfw.wmnet [15:00:05] jelto, arnoldokoth, mutante, and arnaudb: SRE Collaboration Services office hours (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1500). Please do the needful. [15:00:06] !log klausman@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2003.codfw.wmnet [15:00:06] !log klausman@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-staging-worker [15:00:44] (03CR) 10Vgutierrez: gitlab: restrict 443 on the service IP to Prometheus probes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1357232 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [15:01:11] jouncebot nowandnext [15:01:11] For the next 0 hour(s) and 58 minute(s): SRE Collaboration Services office hours (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1500) [15:01:11] In 0 hour(s) and 58 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1600) [15:01:33] (03PS1) 10TrainBranchBot: testwikis to 1.48.0-wmf.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357306 (https://phabricator.wikimedia.org/T438222) [15:01:35] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by dancy@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357306 (https://phabricator.wikimedia.org/T438222) (owner: 10TrainBranchBot) [15:02:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:02:34] (03Merged) 10jenkins-bot: testwikis to 1.48.0-wmf.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357306 (https://phabricator.wikimedia.org/T438222) (owner: 10TrainBranchBot) [15:02:44] (03CR) 10Majavah: "is this not covered by the blanked rule allowing prometheus nodes in `profile::firewall`?" [puppet] - 10https://gerrit.wikimedia.org/r/1357232 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [15:02:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [15:03:52] (03CR) 10Ahmon Dancy: [C:03+1] registryctl: add an option to drop tags containing a date [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1354834 (https://phabricator.wikimedia.org/T437297) (owner: 10Elukey) [15:04:31] !log dancy@deploy1003 Started scap sync-world: testwikis to 1.48.0-wmf.1 refs T438222 [15:04:38] T438222: 1.48.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T438222 [15:05:07] (03PS3) 10Hashar: Gemfile: bump rake and add ruby 3.3 gems [puppet] - 10https://gerrit.wikimedia.org/r/1348698 (https://phabricator.wikimedia.org/T441847) [15:05:09] (03PS2) 10Hashar: rubocop: enable Lint/UnneededDisable [puppet] - 10https://gerrit.wikimedia.org/r/1348795 (https://phabricator.wikimedia.org/T441847) [15:05:12] (03PS4) 10Hashar: rubocop: upgrade and target ruby 2.7 [puppet] - 10https://gerrit.wikimedia.org/r/1348797 (https://phabricator.wikimedia.org/T441847) [15:05:13] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [15:05:25] !log elukey@cumin1004 END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-eqiad [15:06:43] (03PS3) 10Hashar: ipmi: ruby: stop using then for multiline when statement [puppet] - 10https://gerrit.wikimedia.org/r/1348873 (https://phabricator.wikimedia.org/T441847) [15:07:37] (03PS1) 10Gkyziridis: ml-services: Remove the llm-gpt-oss-safeguard-20b model from the experimental ns [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357310 (https://phabricator.wikimedia.org/T440307) [15:07:50] (03CR) 10Elukey: [C:03+2] registryctl: add an option to drop tags containing a date (031 comment) [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1354834 (https://phabricator.wikimedia.org/T437297) (owner: 10Elukey) [15:07:56] (03CR) 10Elukey: [C:03+2] docker_report: fix k8s browser on newer package versions [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1354837 (owner: 10Elukey) [15:08:57] (03PS1) 10Majavah: sslcert: Add more generic sslcert::deselect resource [puppet] - 10https://gerrit.wikimedia.org/r/1357311 [15:08:57] (03PS1) 10Majavah: P:base: Untrust emSign/eMudra root certificates fleet-wide [puppet] - 10https://gerrit.wikimedia.org/r/1357312 [15:11:20] FIRING: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh [15:12:28] (03PS3) 10Hashar: wmflib: fix rubocop ignore missing a department [puppet] - 10https://gerrit.wikimedia.org/r/1348875 (https://phabricator.wikimedia.org/T441847) [15:13:14] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin pooling P{lvs7002.*} and A:liberica [15:13:40] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7002.*} and A:liberica [15:16:20] RESOLVED: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh [15:17:46] (03CR) 10Majavah: [V:03+1 C:03+2] P:base: Untrust emSign/eMudra root certificates fleet-wide [puppet] - 10https://gerrit.wikimedia.org/r/1357312 (owner: 10Majavah) [15:17:54] (03CR) 10Elukey: [V:03+2 C:03+2] docker_report: fix k8s browser on newer package versions [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1354837 (owner: 10Elukey) [15:18:02] (03CR) 10Elukey: [V:03+2 C:03+2] "recheck" [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1354837 (owner: 10Elukey) [15:19:40] (03CR) 10CI reject: [V:04-1] registryctl: add an option to drop tags containing a date [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1354834 (https://phabricator.wikimedia.org/T437297) (owner: 10Elukey) [15:21:09] (03CR) 10Blake: [C:03+2] cronjob: s/cache.mcrouter.deployment/cache.mcrouter.container/ [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356869 (https://phabricator.wikimedia.org/T386694) (owner: 10Blake) [15:21:44] !log btullis@cumin1004 START - Cookbook sre.hosts.provision for host druid-internal1004.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [15:23:32] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [15:23:45] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [15:26:04] 10ops-codfw, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T441855 (10phaultfinder) 03NEW [15:26:41] (03PS3) 10Hashar: jupyterhub: change Object#kind_of? to Object#is_a? [puppet] - 10https://gerrit.wikimedia.org/r/1348958 (https://phabricator.wikimedia.org/T441847) [15:27:04] (03PS4) 10Hashar: logstash: change Object#kind_of? to Object#is_a? [puppet] - 10https://gerrit.wikimedia.org/r/1348957 (https://phabricator.wikimedia.org/T441847) [15:27:08] (03PS4) 10Hashar: wmflib: fix rubocop ignore missing a department [puppet] - 10https://gerrit.wikimedia.org/r/1348875 (https://phabricator.wikimedia.org/T441847) [15:27:16] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host druid-internal1004.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [15:28:20] (03PS9) 10Blake: cronjob.yaml.tpl: enable switch for k8s native sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356824 (https://phabricator.wikimedia.org/T386694) [15:28:40] (03CR) 10CI reject: [V:04-1] cronjob.yaml.tpl: enable switch for k8s native sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356824 (https://phabricator.wikimedia.org/T386694) (owner: 10Blake) [15:29:32] (03PS2) 10Trueg: WDQSv2: fetch the allowlist from internal GitLab [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357305 (https://phabricator.wikimedia.org/T441768) [15:30:29] (03PS10) 10Blake: cronjob.yaml.tpl: enable switch for k8s native sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356824 (https://phabricator.wikimedia.org/T386694) [15:30:34] (03CR) 10JMeybohm: [C:04-1] sre.k8s: add upgrade-kubelets-inplace cookbook (035 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1346036 (https://phabricator.wikimedia.org/T437480) (owner: 10Jelto) [15:31:00] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [15:31:03] (03CR) 10Lerickson: [C:03+1] WDQSv2: fetch the allowlist from internal GitLab [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357305 (https://phabricator.wikimedia.org/T441768) (owner: 10Trueg) [15:31:39] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [15:31:59] !log mvernon@cumin1004 START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe [15:32:20] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [15:32:30] !log mvernon@cumin2003 START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe [15:32:48] (03CR) 10Trueg: [C:03+2] WDQSv2: fetch the allowlist from internal GitLab [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357305 (https://phabricator.wikimedia.org/T441768) (owner: 10Trueg) [15:33:44] (03CR) 10JMeybohm: [C:04-1] sre.k8s: add upgrade-kubelets-inplace cookbook (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1346036 (https://phabricator.wikimedia.org/T437480) (owner: 10Jelto) [15:33:47] FIRING: [3x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-magru (195.200.68.139) - group Confed_magru - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:34:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:34:39] (03Merged) 10jenkins-bot: WDQSv2: fetch the allowlist from internal GitLab [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357305 (https://phabricator.wikimedia.org/T441768) (owner: 10Trueg) [15:36:41] !log mvernon@cumin2003 END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe [15:38:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:40:32] (03PS1) 10Jelto: miscweb: update annualreport image to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357316 (https://phabricator.wikimedia.org/T412941) [15:41:39] !log dancy@deploy1003 Finished scap sync-world: testwikis to 1.48.0-wmf.1 refs T438222 (duration: 37m 07s) [15:41:46] T438222: 1.48.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T438222 [15:42:01] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [15:42:51] FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from gitlab.discovery.wmnet in esams #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=esams&var-cluster=text&var-origin=gitlab.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [15:43:00] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [15:43:41] <_joe_> topranks: can you take a look? I'm in a meeting [15:43:45] <_joe_> I can get out if needed [15:43:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:44:16] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [15:44:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T441855#12455371 (10phaultfinder) [15:45:14] _joe_: yep looking [15:45:15] also in meeting [15:45:31] 06SRE, 06Infrastructure-Foundations, 10Puppet-Core: facter3: use structured facts - https://phabricator.wikimedia.org/T222160#12455374 (10jhathaway) 05Open→03Resolved a:03jhathaway [15:46:57] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [15:46:59] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Degraded RAID on an-presto1006 - https://phabricator.wikimedia.org/T441831#12455387 (10BTullis) HI @RobH, @Jclark-ctr - Thank you. I've answered in T441844#12455369 You can skip ordering this disk. I will delete and recreate the... [15:47:28] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [15:47:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-install - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [15:48:03] !log mvernon@cumin1004 END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe [15:48:08] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [15:48:30] !log T435477 `[airflow-analytics-test]` gearing up to start upgrade to Airflow 3.3, pausing all associated DAGs shortly [15:48:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:48:36] T435477: Upgrade airflow-analytics-test to 3.3.x - https://phabricator.wikimedia.org/T435477 [15:48:41] !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [15:50:09] (03CR) 10CWilliams: [C:03+1] "LGTM" [dns] - 10https://gerrit.wikimedia.org/r/1357263 (https://phabricator.wikimedia.org/T441633) (owner: 10Marostegui) [15:50:28] (03CR) 10Marostegui: [C:03+2] wmnet: Switchover m3-master [dns] - 10https://gerrit.wikimedia.org/r/1357263 (https://phabricator.wikimedia.org/T441633) (owner: 10Marostegui) [15:51:19] !log marostegui@dns1004 START - running authdns-update [15:52:03] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [15:52:51] RESOLVED: ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from gitlab.discovery.wmnet in esams #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=esams&var-cluster=text&var-origin=gitlab.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [15:53:44] !log marostegui@dns1004 END - running authdns-update [15:53:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:54:22] !log T435477 `[airflow-analytics-test]` all associated DAGs paused [15:54:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:54:28] T435477: Upgrade airflow-analytics-test to 3.3.x - https://phabricator.wikimedia.org/T435477 [15:55:16] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [15:55:58] !log trueg@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [15:56:21] (03PS5) 10Herron: sre.thanos.roll-reboot: new cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/1342361 [15:56:21] (03CR) 10Herron: "used this to roll reboot the codfw/eqiad thanos clusters today via test-cookbook, here's the output from eqiad https://phabricator.wikimed" [cookbooks] - 10https://gerrit.wikimedia.org/r/1342361 (owner: 10Herron) [15:58:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:59:19] !log herron@cumin1004 START - Cookbook sre.opensearch.roll-restart-reboot rolling reboot on A:logstash-eqiad [15:59:19] !log trueg@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [16:00:04] jhathaway and rzl: How many deployers does it take to do Puppet request window deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1600). [16:00:05] No Gerrit patches in the queue for this window AFAICS. [16:00:53] (03CR) 10Ryan Kemper: [C:03+2] airflow-analytics-test: upgrade to Airflow 3.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357219 (https://phabricator.wikimedia.org/T435477) (owner: 10Ryan Kemper) [16:02:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [16:02:43] (03Merged) 10jenkins-bot: airflow-analytics-test: upgrade to Airflow 3.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357219 (https://phabricator.wikimedia.org/T435477) (owner: 10Ryan Kemper) [16:06:05] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1165.eqiad.wmnet with reason: Maintenance [16:06:07] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1165 (T441633)', diff saved to https://phabricator.wikimedia.org/P96892 and previous config saved to /var/cache/conftool/dbconfig/20261020-160606-ladsgroup.json [16:07:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [16:08:47] FIRING: [10x] CertAlmostExpired: Certificate for service kafka-jumbo1010:9093 is about to expire - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:09:06] !log ryankemper@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [16:09:52] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T441801#12455507 (10phaultfinder) [16:10:01] !log ryankemper@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [16:10:47] (03CR) 10BCornwall: [C:03+1] gitlab: point the service IP PTRs at the gitlab-ssh names [dns] - 10https://gerrit.wikimedia.org/r/1357229 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [16:10:59] !log T435477 `[airflow-analytics-test]` applying https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1357219, so this marks the official cutover [16:11:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:11:05] T435477: Upgrade airflow-analytics-test to 3.3.x - https://phabricator.wikimedia.org/T435477 [16:11:30] (03CR) 10BCornwall: [C:03+1] trafficserver: add maps for codesearch [puppet] - 10https://gerrit.wikimedia.org/r/1342804 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:12:06] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [16:13:18] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1165 (T441633)', diff saved to https://phabricator.wikimedia.org/P96893 and previous config saved to /var/cache/conftool/dbconfig/20261020-161317-ladsgroup.json [16:13:42] (03PS1) 10Marostegui: wmnet: Failover m5-master [dns] - 10https://gerrit.wikimedia.org/r/1357319 (https://phabricator.wikimedia.org/T441633) [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:18:48] (03PS1) 10Ladsgroup: thumbor: Remove VIPS for jpg files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357320 (https://phabricator.wikimedia.org/T220171) [16:20:18] (03PS5) 10Dduvall: Refactor tests to run inline with build [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345668 (https://phabricator.wikimedia.org/T439490) [16:20:26] (03PS1) 10CDanis: trafficserver: Enforce requestctl cached_only requests [puppet] - 10https://gerrit.wikimedia.org/r/1357323 [16:20:26] (03PS1) 10CDanis: varnish: Handle ATS cached_only denials [puppet] - 10https://gerrit.wikimedia.org/r/1357324 [16:20:26] (03PS1) 10CDanis: cache::haproxy: Add requestctl all scope, handle cached_only headers [puppet] - 10https://gerrit.wikimedia.org/r/1357325 [16:20:28] (03CR) 10Dduvall: Refactor tests to run inline with build (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1345668 (https://phabricator.wikimedia.org/T439490) (owner: 10Dduvall) [16:20:39] (03PS1) 10Clément Goubert: lac-lambdas: Move more configuration to values.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357326 [16:20:46] (03PS1) 10CDanis: cloud-hosts/-vrf: deny to kubernetes ranges [homer/public] - 10https://gerrit.wikimedia.org/r/1357327 [16:21:32] (03CR) 10CI reject: [V:04-1] varnish: Handle ATS cached_only denials [puppet] - 10https://gerrit.wikimedia.org/r/1357324 (owner: 10CDanis) [16:22:46] (03CR) 10Cathal Mooney: [C:03+1] "LGTM thanks." [homer/public] - 10https://gerrit.wikimedia.org/r/1357327 (owner: 10CDanis) [16:22:53] (03CR) 10CI reject: [V:04-1] trafficserver: Enforce requestctl cached_only requests [puppet] - 10https://gerrit.wikimedia.org/r/1357323 (owner: 10CDanis) [16:23:19] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P96895 and previous config saved to /var/cache/conftool/dbconfig/20261020-162319-ladsgroup.json [16:23:20] FIRING: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh [16:23:23] (03PS2) 10CDanis: cloud-hosts/-vrf: deny to kubernetes ranges [homer/public] - 10https://gerrit.wikimedia.org/r/1357327 [16:23:30] (03CR) 10CI reject: [V:04-1] cache::haproxy: Add requestctl all scope, handle cached_only headers [puppet] - 10https://gerrit.wikimedia.org/r/1357325 (owner: 10CDanis) [16:25:00] (03CR) 10CDanis: [C:03+2] cloud-hosts/-vrf: deny to kubernetes ranges [homer/public] - 10https://gerrit.wikimedia.org/r/1357327 (owner: 10CDanis) [16:25:56] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host druid-internal1005.eqiad.wmnet [16:25:57] (03Merged) 10jenkins-bot: cloud-hosts/-vrf: deny to kubernetes ranges [homer/public] - 10https://gerrit.wikimedia.org/r/1357327 (owner: 10CDanis) [16:27:49] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [16:28:02] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [16:28:02] !log bking@cumin2003 END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) [16:28:19] (03CR) 10Jelto: [C:03+1] "verified on meet" [puppet] - 10https://gerrit.wikimedia.org/r/1356982 (https://phabricator.wikimedia.org/T441724) (owner: 10Mstyles) [16:28:20] RESOLVED: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh [16:28:47] FIRING: [10x] CertAlmostExpired: Certificate for service kafka-jumbo1010:9093 is about to expire - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:29:18] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [16:31:14] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid-internal1005.eqiad.wmnet [16:31:18] (03CR) 10Ottomata: [C:03+1] lac-lambdas: Move more configuration to values.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357326 (owner: 10Clément Goubert) [16:31:19] <_joe_> elukey: ^^ the cert about to expire warning for kafka-jumbo is you? [16:31:41] (03CR) 10Jelto: [C:03+2] admin: Add FIDO SSH keys for mstyles [puppet] - 10https://gerrit.wikimedia.org/r/1356982 (https://phabricator.wikimedia.org/T441724) (owner: 10Mstyles) [16:32:46] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [16:33:21] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P96897 and previous config saved to /var/cache/conftool/dbconfig/20261020-163321-ladsgroup.json [16:33:37] !log brett@cumin2003 START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp70[11-16].magru.wmnet} and A:cp [16:34:03] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for mstyles - https://phabricator.wikimedia.org/T441724#12455587 (10Jelto) @Mstyles you should be able to use the keys in the next 30 minutes [16:34:21] (03CR) 10Jelto: [C:03+2] miscweb: update annualreport image to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357316 (https://phabricator.wikimedia.org/T412941) (owner: 10Jelto) [16:34:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:35:20] (03PS2) 10CDanis: trafficserver: Enforce requestctl cached_only requests [puppet] - 10https://gerrit.wikimedia.org/r/1357323 [16:35:20] (03PS2) 10CDanis: varnish: Handle ATS cached_only denials [puppet] - 10https://gerrit.wikimedia.org/r/1357324 [16:35:20] (03PS2) 10CDanis: cache::haproxy: Add requestctl all scope, handle cached_only headers [puppet] - 10https://gerrit.wikimedia.org/r/1357325 [16:35:30] PROBLEM - Host wdqs1011 is DOWN: PING CRITICAL - Packet loss = 100% [16:35:39] (03CR) 10Clément Goubert: [C:03+2] lac-lambdas: Move more configuration to values.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357326 (owner: 10Clément Goubert) [16:36:00] (03Merged) 10jenkins-bot: miscweb: update annualreport image to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357316 (https://phabricator.wikimedia.org/T412941) (owner: 10Jelto) [16:36:29] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host druid-internal1006.eqiad.wmnet [16:36:58] RECOVERY - Host wdqs1011 is UP: PING OK - Packet loss = 0%, RTA = 0.45 ms [16:37:15] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns entries new cloudsw - cmooney@cumin1004" [16:37:19] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns entries new cloudsw - cmooney@cumin1004" [16:37:19] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [16:37:40] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster [16:38:01] (03Merged) 10jenkins-bot: lac-lambdas: Move more configuration to values.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357326 (owner: 10Clément Goubert) [16:38:30] !log cgoubert@deploy1003 helmfile [staging] START helmfile.d/services/lac-lambdas: apply [16:38:32] !log cgoubert@deploy1003 helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply [16:38:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:38:52] FIRING: [8x] CertAlmostExpired: Certificate for service kafka-jumbo1010:9093 is about to expire - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:38:58] (03PS3) 10CDanis: trafficserver: Enforce requestctl cached_only requests [puppet] - 10https://gerrit.wikimedia.org/r/1357323 [16:38:58] (03PS3) 10CDanis: varnish: Handle ATS cached_only denials [puppet] - 10https://gerrit.wikimedia.org/r/1357324 [16:38:58] (03PS3) 10CDanis: cache::haproxy: Add requestctl all scope, handle cached_only headers [puppet] - 10https://gerrit.wikimedia.org/r/1357325 [16:39:18] (03CR) 10CDanis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1357323 (owner: 10CDanis) [16:39:22] (03CR) 10CDanis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1357324 (owner: 10CDanis) [16:39:26] (03CR) 10CDanis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1357325 (owner: 10CDanis) [16:40:16] (03PS1) 10BCornwall: ncmonitor: Ignore wikipedia.ai [puppet] - 10https://gerrit.wikimedia.org/r/1357333 [16:41:20] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device cloudsw3-c8-eqiad [16:41:21] !log cmooney@cumin1004 END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device cloudsw3-c8-eqiad [16:41:46] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid-internal1006.eqiad.wmnet [16:41:52] (03CR) 10JHathaway: [C:03+2] Gemfile: bump rake and add ruby 3.3 gems [puppet] - 10https://gerrit.wikimedia.org/r/1348698 (https://phabricator.wikimedia.org/T441847) (owner: 10Hashar) [16:42:09] (03PS1) 10Clément Goubert: DRAFT: REST gateway: add support for /api/ routes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1354040 (https://phabricator.wikimedia.org/T433547) (owner: 10Daniel Kinzler) [16:42:11] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device cloudsw3-c8-eqiad [16:42:11] !log cmooney@cumin1004 END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device cloudsw3-c8-eqiad [16:42:36] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device cloudsw3-c8-eqiad [16:42:37] !log cmooney@cumin1004 END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device cloudsw3-c8-eqiad [16:43:23] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1165 (T441633)', diff saved to https://phabricator.wikimedia.org/P96898 and previous config saved to /var/cache/conftool/dbconfig/20261020-164322-ladsgroup.json [16:43:41] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance [16:43:43] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1168 (T441633)', diff saved to https://phabricator.wikimedia.org/P96899 and previous config saved to /var/cache/conftool/dbconfig/20261020-164343-ladsgroup.json [16:44:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:44:46] PROBLEM - Host wdqs1012 is DOWN: PING CRITICAL - Packet loss = 100% [16:45:01] !log brett@cumin2003 cookbooks.sre.cdn.roll-reboot finished rebooting cp7011.magru.wmnet [16:45:02] (03CR) 10BCornwall: [C:03+1] wmnet: remove graphite, statsd [dns] - 10https://gerrit.wikimedia.org/r/1332687 (https://phabricator.wikimedia.org/T435340) (owner: 10Hnowlan) [16:45:11] (03CR) 10Ssingh: [C:03+1] ncmonitor: Ignore wikipedia.ai [puppet] - 10https://gerrit.wikimedia.org/r/1357333 (owner: 10BCornwall) [16:45:15] (03CR) 10Dzahn: [C:03+2] admin: add krinkle to codesearch-roots [puppet] - 10https://gerrit.wikimedia.org/r/1355775 (https://phabricator.wikimedia.org/T441068) (owner: 10Dzahn) [16:45:28] (03CR) 10BCornwall: [C:03+1] wmnet: Failover m5-master [dns] - 10https://gerrit.wikimedia.org/r/1357319 (https://phabricator.wikimedia.org/T441633) (owner: 10Marostegui) [16:45:40] RECOVERY - Host wdqs1012 is UP: PING OK - Packet loss = 0%, RTA = 0.46 ms [16:46:10] (03CR) 10BCornwall: [C:03+2] ncmonitor: Ignore wikipedia.ai [puppet] - 10https://gerrit.wikimedia.org/r/1357333 (owner: 10BCornwall) [16:46:57] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to codesearch-roots for krinkle - https://phabricator.wikimedia.org/T441068#12455671 (10Dzahn) Will resolve this within the next few hours. All done but have puppet disabled for other reasons on the affected machine. [16:47:17] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to codesearch-roots for krinkle - https://phabricator.wikimedia.org/T441068#12455674 (10Dzahn) a:03Dzahn [16:47:31] (03CR) 10BCornwall: [V:03+1 C:03+2] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9536/console" [puppet] - 10https://gerrit.wikimedia.org/r/1357333 (owner: 10BCornwall) [16:50:45] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1168 (T441633)', diff saved to https://phabricator.wikimedia.org/P96900 and previous config saved to /var/cache/conftool/dbconfig/20261020-165043-ladsgroup.json [16:52:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [16:52:49] PROBLEM - Host wdqs1013 is DOWN: PING CRITICAL - Packet loss = 100% [16:53:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:53:47] FIRING: [4x] CertAlmostExpired: Certificate for service kafka-jumbo1010:9093 is about to expire - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:54:21] RECOVERY - Host wdqs1013 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [16:54:50] (03CR) 10Volans: "To me, very personal opinion, running `test-cookbook` of a large patch like this one on a full cluster, without any review (at least there" [cookbooks] - 10https://gerrit.wikimedia.org/r/1342361 (owner: 10Herron) [16:57:12] 06SRE, 06Infrastructure-Foundations, 10netops, 13Patch-For-Review: Network: gnmic monitoring issues and improvements - https://phabricator.wikimedia.org/T441780#12455759 (10cmooney) @ayounsi it strikes me we might need better profiling of devices than simply basing things on the manufacturer. For instance... [16:57:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [16:57:18] (03PS1) 10Clément Goubert: cache::haproxy: Block mathoid as a service endpoint [puppet] - 10https://gerrit.wikimedia.org/r/1357339 [16:57:32] !log btullis@cumin1004 END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-jumbo-eqiad [16:57:48] 10ops-eqiad, 06DC-Ops: hw troubleshooting: power-on failure for druid-internal1004.eqiad.wmnet - https://phabricator.wikimedia.org/T441866 (10BTullis) 03NEW [16:58:07] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad: WMCS switch refresh 2026 (C8 & D5) - https://phabricator.wikimedia.org/T440305#12455778 (10cmooney) [16:58:20] FIRING: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh [16:58:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:58:47] FIRING: [6x] CertAlmostExpired: Certificate for service kafka-jumbo1010:9093 is about to expire - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:58:52] FIRING: [3x] GnmiInterfaceCountersDrop: lsw1-a5-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [16:59:32] !log jelto@deploy1003 Started scap deploy-service miscweb: demo deployment for MVP walk through [16:59:32] !log jelto@deploy1003 scap deploy-service miscweb: [STAGING/DEFAULT] staging-eqiad Deploying miscweb [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1700) [17:00:34] !log jelto@deploy1003 scap deploy-service miscweb: [PRODUCTION/DEFAULT] codfw Deploying miscweb [17:00:47] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P96901 and previous config saved to /var/cache/conftool/dbconfig/20261020-170046-ladsgroup.json [17:01:31] !log jelto@deploy1003 scap deploy-service miscweb: [PRODUCTION/DEFAULT] eqiad Deploying miscweb [17:01:43] PROBLEM - Host wdqs1014 is DOWN: PING CRITICAL - Packet loss = 100% [17:02:09] !log jelto@deploy1003 Finished scap deploy-service miscweb: demo deployment for MVP walk through (duration: 02m 37s) [17:02:21] (03CR) 10Volans: [C:03+1] "LGTM and it's a noop on the prod hosts." [puppet] - 10https://gerrit.wikimedia.org/r/1357239 (https://phabricator.wikimedia.org/T441785) (owner: 10Majavah) [17:02:35] !log bking@cumin2003 END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) [17:02:57] !log T435477 `[airflow-analytics-test]` upgrade complete; all previously-unpaused DAGs have been unpaused again. ofc, we'll still watch for anything weird that breaks over the next couple days [17:03:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:03:03] T435477: Upgrade airflow-analytics-test to 3.3.x - https://phabricator.wikimedia.org/T435477 [17:03:15] RECOVERY - Host wdqs1014 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [17:03:47] RESOLVED: [4x] CertAlmostExpired: Certificate for service kafka-jumbo1010:9093 is about to expire - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Renew_TLS_certificate - https://grafana.wikimedia.org/d/K1dRhGCnz/probes-tls-dashboard - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [17:04:29] (03PS2) 10Clément Goubert: REST gateway: add support for /api/ routes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1354040 (https://phabricator.wikimedia.org/T433547) (owner: 10Daniel Kinzler) [17:05:02] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [17:06:26] !log btullis@cumin1004 START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad [17:06:32] RECOVERY - Host an-worker1142 is UP: PING OK - Packet loss = 0%, RTA = 0.12 ms [17:07:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [17:08:12] (03CR) 10BCornwall: [C:03+2] NCRedirRedirects: Automated MarkMonitor domain sync [puppet] - 10https://gerrit.wikimedia.org/r/1353514 (owner: 10Ncmonitor) [17:08:47] FIRING: [3x] GnmiInterfaceCountersDrop: lsw1-a5-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [17:10:46] PROBLEM - Host wdqs1016 is DOWN: PING CRITICAL - Packet loss = 100% [17:10:49] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P96903 and previous config saved to /var/cache/conftool/dbconfig/20261020-171048-ladsgroup.json [17:12:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [17:12:16] RECOVERY - Host wdqs1016 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [17:13:08] !log bking@cumin2003 START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838 [17:13:14] T441838: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838 [17:13:16] !log bking@cumin2003 END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838 [17:13:28] !log bking@cumin2003 START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838 [17:13:47] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [17:13:48] !log btullis@cumin1004 START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster [17:13:48] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster [17:14:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [17:14:14] !log btullis@cumin1004 START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster [17:14:15] !log btullis@cumin1004 END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster [17:15:13] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host an-worker1142.eqiad.wmnet [17:17:48] (03PS4) 10CDanis: trafficserver: Enforce requestctl cached_only requests [puppet] - 10https://gerrit.wikimedia.org/r/1357323 [17:17:48] (03PS4) 10CDanis: varnish: Handle X-Requestctl-Cached-Only-Denied response [puppet] - 10https://gerrit.wikimedia.org/r/1357324 [17:17:48] (03PS4) 10CDanis: cache::haproxy: Add requestctl all scope, support cached_only [puppet] - 10https://gerrit.wikimedia.org/r/1357325 [17:18:59] PROBLEM - Host wdqs1017 is DOWN: PING CRITICAL - Packet loss = 100% [17:19:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [17:19:09] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [17:20:40] !log bking@cumin2003 END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838 [17:20:46] T441838: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838 [17:20:51] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1168 (T441633)', diff saved to https://phabricator.wikimedia.org/P96904 and previous config saved to /var/cache/conftool/dbconfig/20261020-172050-ladsgroup.json [17:21:08] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance [17:21:11] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1187 (T441633)', diff saved to https://phabricator.wikimedia.org/P96905 and previous config saved to /var/cache/conftool/dbconfig/20261020-172110-ladsgroup.json [17:21:17] RECOVERY - Host wdqs1017 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms [17:22:07] (03CR) 10Blake: [C:03+2] cronjob.yaml.tpl: enable switch for k8s native sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356824 (https://phabricator.wikimedia.org/T386694) (owner: 10Blake) [17:23:40] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1142.eqiad.wmnet [17:24:56] (03Merged) 10jenkins-bot: cronjob.yaml.tpl: enable switch for k8s native sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356824 (https://phabricator.wikimedia.org/T386694) (owner: 10Blake) [17:26:04] !log blake@deploy1003 Started scap sync-world: bump chart version for T386694 [17:26:09] !log brett@cumin2003 cookbooks.sre.cdn.roll-reboot finished rebooting cp7012.magru.wmnet [17:26:10] T386694: Replace k8s-controller-sidecars with built in Sidecar containers on k8s 1.31 - https://phabricator.wikimedia.org/T386694 [17:27:02] (03PS1) 10Andrew Bogott: wmcs-drain-hypervisor: format with black [puppet] - 10https://gerrit.wikimedia.org/r/1357344 [17:27:02] (03PS1) 10Andrew Bogott: wmcs-drain-hypervisor.py: add 'rebootprep' arg [puppet] - 10https://gerrit.wikimedia.org/r/1357345 [17:27:02] (03PS1) 10Andrew Bogott: wmcs-drain-hypervisor.py: force VMs in VERIFY_RESIZE state to ACTIVE [puppet] - 10https://gerrit.wikimedia.org/r/1357346 [17:28:03] !log re-enabled puppet on codesearch1001/2001 - might start to index git repos on gitlab and gerrit [17:28:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:28:08] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1187 (T441633)', diff saved to https://phabricator.wikimedia.org/P96906 and previous config saved to /var/cache/conftool/dbconfig/20261020-172807-ladsgroup.json [17:28:12] (03CR) 10CI reject: [V:04-1] wmcs-drain-hypervisor.py: add 'rebootprep' arg [puppet] - 10https://gerrit.wikimedia.org/r/1357345 (owner: 10Andrew Bogott) [17:28:31] (03CR) 10CI reject: [V:04-1] wmcs-drain-hypervisor.py: force VMs in VERIFY_RESIZE state to ACTIVE [puppet] - 10https://gerrit.wikimedia.org/r/1357346 (owner: 10Andrew Bogott) [17:28:47] PROBLEM - Host wdqs1018 is DOWN: PING CRITICAL - Packet loss = 100% [17:29:26] (03CR) 10Herron: "Thanks for the feedback. For background, the reviewers on the patch have been involved with this via discussion in meetings and via irc. W" [cookbooks] - 10https://gerrit.wikimedia.org/r/1342361 (owner: 10Herron) [17:30:02] !log blake@deploy1003 Finished scap sync-world: bump chart version for T386694 (duration: 04m 54s) [17:30:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:30:46] (03PS2) 10Andrew Bogott: wmcs-drain-hypervisor.py: add 'rebootprep' arg [puppet] - 10https://gerrit.wikimedia.org/r/1357345 [17:30:46] (03PS2) 10Andrew Bogott: wmcs-drain-hypervisor.py: force VMs in VERIFY_RESIZE state to ACTIVE [puppet] - 10https://gerrit.wikimedia.org/r/1357346 [17:30:53] RECOVERY - Host wdqs1018 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [17:32:19] (03CR) 10Blake: "@brouberol@wikimedia.org, if you don't mind having a look at the mediawiki-dumps-legacy part of this, I'd appreciate it!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356812 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [17:34:40] 10ops-codfw, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T441869 (10phaultfinder) 03NEW [17:35:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:37:40] PROBLEM - Host wdqs1019 is DOWN: PING CRITICAL - Packet loss = 100% [17:38:10] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P96908 and previous config saved to /var/cache/conftool/dbconfig/20261020-173809-ladsgroup.json [17:38:59] blake: OK if I deploy a new scap release? [17:39:28] !log brett@cumin2003 START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling reboot on A:ncredir [17:40:18] RECOVERY - Host wdqs1019 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [17:40:59] !log dancy@deploy1003 Installing scap version "4.302.0" for 3 host(s) [17:41:13] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [17:41:43] !log btullis@cumin1004 END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad [17:42:53] !log dancy@deploy1003 Installation of scap version "4.302.0" completed for 3 hosts [17:43:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:44:26] (03CR) 10Volans: "This has always been left to the sensibility of the operators as there can be so many scenarios, like self-merging patches, and similar th" [cookbooks] - 10https://gerrit.wikimedia.org/r/1342361 (owner: 10Herron) [17:48:11] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P96909 and previous config saved to /var/cache/conftool/dbconfig/20261020-174811-ladsgroup.json [17:48:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:54:31] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:55:07] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to codesearch-roots for krinkle - https://phabricator.wikimedia.org/T441068#12456137 (10Dzahn) 05In progress→03Resolved ` [codesearch1001:~] $ id krinkle uid=2008(krinkle) gid=500(wikidev) groups=500(wikidev),770(codesearch-roots)... [17:58:13] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1187 (T441633)', diff saved to https://phabricator.wikimedia.org/P96910 and previous config saved to /var/cache/conftool/dbconfig/20261020-175812-ladsgroup.json [17:58:30] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357352 [17:58:32] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1201.eqiad.wmnet with reason: Maintenance [17:58:34] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1201 (T441633)', diff saved to https://phabricator.wikimedia.org/P96911 and previous config saved to /var/cache/conftool/dbconfig/20261020-175833-ladsgroup.json [18:00:05] dancy and hashar: Deploy window MediaWiki train - Utc-7+Utc-0 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T1800) [18:00:10] o/ [18:00:50] (03PS1) 10TrainBranchBot: group0 to 1.48.0-wmf.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357353 (https://phabricator.wikimedia.org/T438222) [18:00:53] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by dancy@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357353 (https://phabricator.wikimedia.org/T438222) (owner: 10TrainBranchBot) [18:01:40] (03Merged) 10jenkins-bot: group0 to 1.48.0-wmf.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357353 (https://phabricator.wikimedia.org/T438222) (owner: 10TrainBranchBot) [18:03:17] (03CR) 10Dzahn: "." [puppet] - 10https://gerrit.wikimedia.org/r/1357025 (https://phabricator.wikimedia.org/T441247) (owner: 10Dzahn) [18:03:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:05:13] (03CR) 10Dzahn: gitlab: restrict 443 on the service IP to Prometheus probes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1357232 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [18:05:37] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1201 (T441633)', diff saved to https://phabricator.wikimedia.org/P96912 and previous config saved to /var/cache/conftool/dbconfig/20261020-180536-ladsgroup.json [18:05:49] (03CR) 10Dzahn: [C:03+2] gerrit: ensure future gerrit is NOT generating host keys [puppet] - 10https://gerrit.wikimedia.org/r/1356992 (https://phabricator.wikimedia.org/T441113) (owner: 10Dzahn) [18:07:50] !log brett@cumin2003 cookbooks.sre.cdn.roll-reboot finished rebooting cp7013.magru.wmnet [18:08:47] FIRING: [4x] BFDdown: BFD session down between cr1-codfw and 195.200.68.139 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [18:10:39] !log bking@cumin2003 START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838 [18:10:45] T441838: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838 [18:12:01] decentralized [18:13:37] (03CR) 10Dzahn: [C:03+2] "thanks, deployed! no service restarts." [puppet] - 10https://gerrit.wikimedia.org/r/1356992 (https://phabricator.wikimedia.org/T441113) (owner: 10Dzahn) [18:14:22] !log dancy@deploy1003 rebuilt and synchronized wikiversions files: group0 to 1.48.0-wmf.1 refs T438222 [18:14:28] T438222: 1.48.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T438222 [18:15:39] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P96914 and previous config saved to /var/cache/conftool/dbconfig/20261020-181538-ladsgroup.json [18:23:28] (03PS1) 10Sbisson: ArticleGuidance: Enable the reference risk model in production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357358 (https://phabricator.wikimedia.org/T440586) [18:23:47] FIRING: [4x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:24:18] !log bking@cumin2003 END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838 [18:24:22] (03CR) 10CI reject: [V:04-1] ArticleGuidance: Enable the reference risk model in production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357358 (https://phabricator.wikimedia.org/T440586) (owner: 10Sbisson) [18:24:25] T441838: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838 [18:25:02] (03PS1) 10Dzahn: codesearch: show a preview banner on the production frontend [puppet] - 10https://gerrit.wikimedia.org/r/1357359 (https://phabricator.wikimedia.org/T268199) [18:25:41] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P96915 and previous config saved to /var/cache/conftool/dbconfig/20261020-182540-ladsgroup.json [18:26:01] (03PS1) 10Ottomata: Prefetch the LAC revision_feature_counts artifact on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357360 (https://phabricator.wikimedia.org/T436695) [18:26:13] (03CR) 10Dzahn: [C:03+2] admin: add obenhmida to analytics-privatedata-users, lvl1 [puppet] - 10https://gerrit.wikimedia.org/r/1356986 (https://phabricator.wikimedia.org/T440464) (owner: 10Dzahn) [18:28:47] FIRING: KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1152.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:29:00] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users, lvl 1 for OBenhmida - https://phabricator.wikimedia.org/T440464#12456328 (10Dzahn) @OBenhmida Hi, I merged the change that I think fixes this. Can you please try half an hour after this comment and check if a... [18:32:09] (03PS2) 10Ottomata: Prefetch the LAC revision_feature_counts artifact on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357360 (https://phabricator.wikimedia.org/T436695) [18:32:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [18:32:32] (03PS2) 10Dzahn: codesearch: show a preview banner on the production frontend [puppet] - 10https://gerrit.wikimedia.org/r/1357359 (https://phabricator.wikimedia.org/T268199) [18:33:48] PROBLEM - BFD status on cloudsw1-c8-eqiad.mgmt is CRITICAL: Down: 1 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [18:35:43] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1201 (T441633)', diff saved to https://phabricator.wikimedia.org/P96916 and previous config saved to /var/cache/conftool/dbconfig/20261020-183542-ladsgroup.json [18:35:48] RECOVERY - BFD status on cloudsw1-c8-eqiad.mgmt is OK: UP: 16 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [18:36:02] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance [18:37:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [18:38:47] FIRING: [5x] CoreBGPDown: Core BGP session down between cloudsw1-c8-eqiad and cloudservices1006 (172.20.1.5) - group cloud_host - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [18:41:22] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1241.eqiad.wmnet with reason: Maintenance [18:41:24] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1241 (T441633)', diff saved to https://phabricator.wikimedia.org/P96917 and previous config saved to /var/cache/conftool/dbconfig/20261020-184123-ladsgroup.json [18:41:26] (03CR) 10Dzahn: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1357359 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [18:41:32] !log bking@cumin2003 START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838 [18:41:38] T441838: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838 [18:43:47] FIRING: [4x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:48:07] !log brett@cumin2003 END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling reboot on A:ncredir [18:48:39] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1241 (T441633)', diff saved to https://phabricator.wikimedia.org/P96918 and previous config saved to /var/cache/conftool/dbconfig/20261020-184838-ladsgroup.json [18:48:47] FIRING: [4x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:48:56] FIRING: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:49:07] !log brett@cumin2003 cookbooks.sre.cdn.roll-reboot finished rebooting cp7014.magru.wmnet [18:53:56] RESOLVED: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:54:02] (03PS1) 10Ottomata: changeprop-jobqueue: Give LAC revision_feature_counts prefetch its own rule [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357365 (https://phabricator.wikimedia.org/T436695) [18:58:38] PROBLEM - Check unit status of push_cross_cluster_settings_9400 on cloudelastic1009 is CRITICAL: CRITICAL: Status of the systemd unit push_cross_cluster_settings_9400 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:58:41] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P96919 and previous config saved to /var/cache/conftool/dbconfig/20261020-185840-ladsgroup.json [18:58:47] FIRING: [4x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:00:06] PROBLEM - Check unit status of push_cross_cluster_settings_9200 on cloudelastic1009 is CRITICAL: CRITICAL: Status of the systemd unit push_cross_cluster_settings_9200 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [19:00:39] (03PS1) 10Arlolra: Add "proofread-index" to LINTABLE_CONTENT_MODELS [extensions/Linter] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1357368 (https://phabricator.wikimedia.org/T441874) [19:01:08] !log dancy@deploy1003 Installing scap version "4.302.0" for 158 host(s) [19:02:33] !log bking@cumin2003 START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply kernel updates - bking@cumin2003 - T441838 [19:02:39] (03PS1) 10Arlolra: Add "proofread-index" to LINTABLE_CONTENT_MODELS [extensions/Linter] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1357370 (https://phabricator.wikimedia.org/T441874) [19:02:41] T441838: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838 [19:03:25] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 20 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [extensions/Linter] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1357368 (https://phabricator.wikimedia.org/T441874) (owner: 10Arlolra) [19:03:35] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, October 20 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [extensions/Linter] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1357370 (https://phabricator.wikimedia.org/T441874) (owner: 10Arlolra) [19:03:45] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [19:03:45] !log bking@cumin2003 END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) [19:03:47] FIRING: [4x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:05:12] !log dancy@deploy1003 Installation of scap version "4.302.0" completed for 158 hosts [19:05:34] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [19:06:39] (03CR) 10Ryan Kemper: "The short answer is, in principle they don't *need* to" [puppet] - 10https://gerrit.wikimedia.org/r/1355381 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [19:06:58] !log bking@cumin2003 END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838 [19:07:29] (03PS1) 10Mstyles: admin: Remove non-FIDO SSH key for mstyles [puppet] - 10https://gerrit.wikimedia.org/r/1357373 (https://phabricator.wikimedia.org/T441724) [19:08:43] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P96920 and previous config saved to /var/cache/conftool/dbconfig/20261020-190842-ladsgroup.json [19:09:39] (03PS1) 10Ahmon Dancy: scap: Remove the statsd_port setting [puppet] - 10https://gerrit.wikimedia.org/r/1357374 (https://phabricator.wikimedia.org/T436513) [19:11:32] PROBLEM - Host wdqs1023 is DOWN: PING CRITICAL - Packet loss = 100% [19:13:47] FIRING: [4x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:14:00] RECOVERY - Host wdqs1023 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [19:15:41] (03CR) 10Ryan Kemper: "You may have already seen, but it's there just hiding at the bottom of the diff :D" [puppet] - 10https://gerrit.wikimedia.org/r/1355378 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [19:17:28] (03CR) 10Ryan Kemper: [C:03+1] "reapplying +1 after comment change" [puppet] - 10https://gerrit.wikimedia.org/r/1355379 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [19:18:38] RECOVERY - Check unit status of push_cross_cluster_settings_9400 on cloudelastic1009 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9400 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [19:18:45] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1241 (T441633)', diff saved to https://phabricator.wikimedia.org/P96921 and previous config saved to /var/cache/conftool/dbconfig/20261020-191844-ladsgroup.json [19:18:47] FIRING: [4x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:19:03] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1279.eqiad.wmnet with reason: Maintenance [19:19:24] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance [19:19:26] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1279 (T441633)', diff saved to https://phabricator.wikimedia.org/P96922 and previous config saved to /var/cache/conftool/dbconfig/20261020-191925-ladsgroup.json [19:19:49] 06SRE, 10Infrastructure Security, 06Data-Platform-SRE (2026-10-09 - 2026-10-30), 07Security: Reboot Data Platform-owned Opensearch hosts (relforge, cloudelastic, cirrussearch) - https://phabricator.wikimedia.org/T441883 (10bking) 03NEW [19:20:06] RECOVERY - Check unit status of push_cross_cluster_settings_9200 on cloudelastic1009 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9200 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [19:20:26] PROBLEM - Host wdqs1024 is DOWN: PING CRITICAL - Packet loss = 100% [19:21:31] (03PS1) 10Ahmon Dancy: scap::master: Pass SCAP_CALLER through sudo [puppet] - 10https://gerrit.wikimedia.org/r/1357382 [19:22:18] RECOVERY - Host wdqs1024 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms [19:23:21] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [19:24:25] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1279 (T441633)', diff saved to https://phabricator.wikimedia.org/P96923 and previous config saved to /var/cache/conftool/dbconfig/20261020-192424-ladsgroup.json [19:28:44] !log brett@cumin2003 cookbooks.sre.cdn.roll-reboot finished rebooting cp7015.magru.wmnet [19:34:27] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1279', diff saved to https://phabricator.wikimedia.org/P96924 and previous config saved to /var/cache/conftool/dbconfig/20261020-193426-ladsgroup.json [19:40:35] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [19:44:29] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1279', diff saved to https://phabricator.wikimedia.org/P96925 and previous config saved to /var/cache/conftool/dbconfig/20261020-194428-ladsgroup.json [19:48:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:53:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:54:31] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1279 (T441633)', diff saved to https://phabricator.wikimedia.org/P96926 and previous config saved to /var/cache/conftool/dbconfig/20261020-195430-ladsgroup.json [19:54:48] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1282.eqiad.wmnet with reason: Maintenance [19:54:50] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1282 (T441633)', diff saved to https://phabricator.wikimedia.org/P96927 and previous config saved to /var/cache/conftool/dbconfig/20261020-195450-ladsgroup.json [19:58:08] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [19:58:47] FIRING: [4x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:59:31] FIRING: [4x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:59:52] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1282 (T441633)', diff saved to https://phabricator.wikimedia.org/P96928 and previous config saved to /var/cache/conftool/dbconfig/20261020-195951-ladsgroup.json [20:00:05] RoanKattouw, urbanecm, TheresNoTime, and cjming: Time to do the UTC late backport window deploy. Don't look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T2000). [20:00:05] arlolra: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:01:24] I will get started on my patches in a few minutes unless anyone says otherwise [20:04:31] FIRING: [4x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:06:31] !log arlolra@deploy1003 Started scap sync-world: Backport for Add "proofread-index" to LINTABLE_CONTENT_MODELS (T441874) ([[gerrit:1357368|1.48.0-wmf.1]], [[gerrit:1357370|1.47.0-wmf.24]]) [20:06:38] T441874: Consolidate configuration for when Linting content models - https://phabricator.wikimedia.org/T441874 [20:07:30] hey, arlolra , could we squeeze in another patch in this window? [20:07:42] plenty of room [20:08:13] !log brett@cumin2003 cookbooks.sre.cdn.roll-reboot finished rebooting cp7016.magru.wmnet [20:08:13] !log brett@cumin2003 END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp70[11-16].magru.wmnet} and A:cp [20:09:53] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1282', diff saved to https://phabricator.wikimedia.org/P96929 and previous config saved to /var/cache/conftool/dbconfig/20261020-200953-ladsgroup.json [20:10:46] !log arlolra@deploy1003 arlolra: Backport for Add "proofread-index" to LINTABLE_CONTENT_MODELS (T441874) ([[gerrit:1357368|1.48.0-wmf.1]], [[gerrit:1357370|1.47.0-wmf.24]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:11:10] !log bking@cumin2003 START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838 [20:11:15] !log bking@cumin2003 END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply kernel updates - bking@cumin2003 - T441838 [20:11:17] T441838: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838 [20:11:46] !log bking@cumin2003 START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply kernel updates - bking@cumin2003 - T441838 [20:11:55] !log arlolra@deploy1003 arlolra: Continuing with deployment [20:14:15] PROBLEM - OpenSearch health check for shards on 9400 on cloudelastic1010 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1390, relocating_shards: 0, initializing_shards: 0, unassigned_sha [20:14:15] , delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.38332333533293 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:14:15] PROBLEM - OpenSearch health check for shards on 9600 on cloudelastic1008 is CRITICAL: CRITICAL - elasticsearch inactive shards 272 threshold =0.15 breach: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1367, relocating_shards: 0, initializing_shards: 0, unassigned_shard [20:14:15] delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.40451494813911 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:14:15] PROBLEM - OpenSearch health check for shards on 9400 on cloudelastic1008 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1390, relocating_shards: 0, initializing_shards: 0, unassigned_sha [20:14:16] , delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.38332333533293 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:14:16] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1007 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 875, active_shards: 1388, relocating_shards: 0, initializing_shards: 0, unassigned_shards: [20:14:17] ayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.36336336336336 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:14:17] PROBLEM - OpenSearch health check for shards on 9400 on cloudelastic1009 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1390, relocating_shards: 0, initializing_shards: 0, unassigned_sha [20:14:18] , delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.38332333533293 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:14:19] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1009 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 875, active_shards: 1388, relocating_shards: 0, initializing_shards: 0, unassigned_shards: [20:14:19] ayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.36336336336336 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:14:33] PROBLEM - OpenSearch health check for shards on 9400 on cloudelastic1007 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1390, relocating_shards: 0, initializing_shards: 0, unassigned_sha [20:14:33] , delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.38332333533293 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:14:33] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1008 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 875, active_shards: 1388, relocating_shards: 0, initializing_shards: 0, unassigned_shards: [20:14:33] ayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.36336336336336 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:14:39] PROBLEM - OpenSearch health check for shards on 9600 on cloudelastic1011 is CRITICAL: CRITICAL - elasticsearch inactive shards 272 threshold =0.15 breach: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1367, relocating_shards: 0, initializing_shards: 0, unassigned_shard [20:14:39] delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.40451494813911 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:14:41] PROBLEM - OpenSearch health check for shards on 9600 on cloudelastic1007 is CRITICAL: CRITICAL - elasticsearch inactive shards 272 threshold =0.15 breach: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1367, relocating_shards: 0, initializing_shards: 0, unassigned_shard [20:14:41] delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.40451494813911 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:14:41] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1011 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 875, active_shards: 1388, relocating_shards: 0, initializing_shards: 0, unassigned_shards: [20:14:41] ayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.36336336336336 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:14:43] PROBLEM - OpenSearch health check for shards on 9600 on cloudelastic1010 is CRITICAL: CRITICAL - elasticsearch inactive shards 272 threshold =0.15 breach: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1367, relocating_shards: 0, initializing_shards: 0, unassigned_shard [20:14:43] delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.40451494813911 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:14:49] PROBLEM - OpenSearch health check for shards on 9600 on cloudelastic1009 is CRITICAL: CRITICAL - elasticsearch inactive shards 272 threshold =0.15 breach: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1367, relocating_shards: 0, initializing_shards: 0, unassigned_shard [20:14:49] delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.40451494813911 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:14:59] PROBLEM - OpenSearch health check for shards on 9400 on cloudelastic1011 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1390, relocating_shards: 0, initializing_shards: 0, unassigned_sha [20:14:59] , delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.38332333533293 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:15:05] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1010 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 875, active_shards: 1388, relocating_shards: 0, initializing_shards: 0, unassigned_shards: [20:15:05] ayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.36336336336336 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:15:42] hmm, deploying to the canaries failed with [20:15:43] Error: Failed to render chart: exit status 1: Error: no cached repo found. (try 'helm repo update'): error loading /var/cache/helm/repository/wmf-stable-index.yaml: empty index.yaml file Error: plugin "diff" exited with error [20:15:49] RECOVERY - OpenSearch health check for shards on 9600 on cloudelastic1009 is OK: OK - elasticsearch status cloudelastic-psi-eqiad: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1402, relocating_shards: 0, initializing_shards: 2, unassigned_shards: 236, delayed_unassign [20:15:49] s: 0, number_of_pending_tasks: 2, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 54, active_shards_percent_as_number: 85.48780487804878 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:15:59] RECOVERY - OpenSearch health check for shards on 9400 on cloudelastic1011 is OK: OK - elasticsearch status cloudelastic-omega-eqiad: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1543, relocating_shards: 0, initializing_shards: 3, unassigned_shards: 122, delayed_unas [20:15:59] hards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 92.50599520383693 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:16:05] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1010 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: yellow, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 895, active_shards: 1541, relocating_shards: 0, initializing_shards: 6, unassigned_shards: 119, delayed_unassign [20:16:05] s: 0, number_of_pending_tasks: 2, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 79, active_shards_percent_as_number: 92.4969987995198 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:16:15] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1007 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: yellow, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 895, active_shards: 1593, relocating_shards: 0, initializing_shards: 6, unassigned_shards: 67, delayed_unassigne [20:16:15] : 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 95.61824729891957 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:16:15] RECOVERY - OpenSearch health check for shards on 9400 on cloudelastic1010 is OK: OK - elasticsearch status cloudelastic-omega-eqiad: cluster_name: cloudelastic-omega-eqiad, status: green, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1668, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassig [20:16:15] ds: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:16:15] RECOVERY - OpenSearch health check for shards on 9600 on cloudelastic1008 is OK: OK - elasticsearch status cloudelastic-psi-eqiad: cluster_name: cloudelastic-psi-eqiad, status: green, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1640, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_ [20:16:16] 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:16:16] RECOVERY - OpenSearch health check for shards on 9400 on cloudelastic1008 is OK: OK - elasticsearch status cloudelastic-omega-eqiad: cluster_name: cloudelastic-omega-eqiad, status: green, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1668, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassig [20:16:17] ds: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:16:17] RECOVERY - OpenSearch health check for shards on 9400 on cloudelastic1009 is OK: OK - elasticsearch status cloudelastic-omega-eqiad: cluster_name: cloudelastic-omega-eqiad, status: green, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1668, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassig [20:16:18] ds: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:16:19] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1009 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: yellow, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 895, active_shards: 1620, relocating_shards: 0, initializing_shards: 4, unassigned_shards: 42, delayed_unassigne [20:16:19] : 0, number_of_pending_tasks: 3, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 191, active_shards_percent_as_number: 97.23889555822329 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:16:33] RECOVERY - OpenSearch health check for shards on 9400 on cloudelastic1007 is OK: OK - elasticsearch status cloudelastic-omega-eqiad: cluster_name: cloudelastic-omega-eqiad, status: green, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1668, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassig [20:16:33] ds: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:16:33] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1008 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: green, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 895, active_shards: 1666, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_ [20:16:33] 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:16:39] RECOVERY - OpenSearch health check for shards on 9600 on cloudelastic1011 is OK: OK - elasticsearch status cloudelastic-psi-eqiad: cluster_name: cloudelastic-psi-eqiad, status: green, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1640, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_ [20:16:39] 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:16:41] RECOVERY - OpenSearch health check for shards on 9600 on cloudelastic1007 is OK: OK - elasticsearch status cloudelastic-psi-eqiad: cluster_name: cloudelastic-psi-eqiad, status: green, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1640, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_ [20:16:41] 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:16:41] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1011 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: green, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 895, active_shards: 1666, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_ [20:16:41] 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:16:43] RECOVERY - OpenSearch health check for shards on 9600 on cloudelastic1010 is OK: OK - elasticsearch status cloudelastic-psi-eqiad: cluster_name: cloudelastic-psi-eqiad, status: green, timed_out: False, number_of_nodes: 6, number_of_data_nodes: 6, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1640, relocating_shards: 0, initializing_shards: 0, unassigned_shards: 0, delayed_unassigned_ [20:16:43] 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 100.0 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:18:28] sukhe: any thoughts? [20:19:49] I will retry deployment [20:19:55] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1282', diff saved to https://phabricator.wikimedia.org/P96930 and previous config saved to /var/cache/conftool/dbconfig/20261020-201954-ladsgroup.json [20:20:59] PROBLEM - OpenSearch health check for shards on 9400 on cloudelastic1011 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1390, relocating_shards: 0, initializing_shards: 0, unassigned_sha [20:20:59] , delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.38332333533293 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:21:05] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1010 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 874, active_shards: 1388, relocating_shards: 0, initializing_shards: 0, unassigned_shards: [20:21:05] ayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.36336336336336 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:21:15] PROBLEM - OpenSearch health check for shards on 9400 on cloudelastic1010 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1390, relocating_shards: 0, initializing_shards: 0, unassigned_sha [20:21:15] , delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.38332333533293 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:21:15] PROBLEM - OpenSearch health check for shards on 9600 on cloudelastic1012 is CRITICAL: CRITICAL - elasticsearch inactive shards 272 threshold =0.15 breach: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1367, relocating_shards: 0, initializing_shards: 0, unassigned_shard [20:21:15] delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.40451494813911 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:21:15] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1007 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 874, active_shards: 1388, relocating_shards: 0, initializing_shards: 0, unassigned_shards: [20:21:16] ayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.36336336336336 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:21:17] PROBLEM - OpenSearch health check for shards on 9400 on cloudelastic1009 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1390, relocating_shards: 0, initializing_shards: 0, unassigned_sha [20:21:17] , delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.38332333533293 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:21:17] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1012 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 874, active_shards: 1388, relocating_shards: 0, initializing_shards: 0, unassigned_shards: [20:21:18] ayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.36336336336336 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:21:19] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1009 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 874, active_shards: 1388, relocating_shards: 0, initializing_shards: 0, unassigned_shards: [20:21:19] ayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.36336336336336 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:21:33] PROBLEM - OpenSearch health check for shards on 9400 on cloudelastic1007 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1390, relocating_shards: 0, initializing_shards: 0, unassigned_sha [20:21:33] , delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.38332333533293 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:21:39] PROBLEM - OpenSearch health check for shards on 9600 on cloudelastic1011 is CRITICAL: CRITICAL - elasticsearch inactive shards 272 threshold =0.15 breach: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1367, relocating_shards: 0, initializing_shards: 0, unassigned_shard [20:21:39] delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.40451494813911 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:21:41] PROBLEM - OpenSearch health check for shards on 9600 on cloudelastic1007 is CRITICAL: CRITICAL - elasticsearch inactive shards 272 threshold =0.15 breach: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1367, relocating_shards: 0, initializing_shards: 0, unassigned_shard [20:21:41] delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.40451494813911 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:21:41] PROBLEM - OpenSearch health check for shards on 9200 on cloudelastic1011 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-chi-eqiad, status: red, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 874, active_shards: 1388, relocating_shards: 0, initializing_shards: 0, unassigned_shards: [20:21:41] ayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.36336336336336 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:21:43] PROBLEM - OpenSearch health check for shards on 9600 on cloudelastic1010 is CRITICAL: CRITICAL - elasticsearch inactive shards 272 threshold =0.15 breach: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1367, relocating_shards: 0, initializing_shards: 0, unassigned_shard [20:21:43] delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.40451494813911 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:21:45] PROBLEM - OpenSearch health check for shards on 9400 on cloudelastic1012 is CRITICAL: CRITICAL - elasticsearch inactive shards 277 threshold =0.15 breach: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1390, relocating_shards: 0, initializing_shards: 0, unassigned_sha [20:21:45] , delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.38332333533293 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:21:49] PROBLEM - OpenSearch health check for shards on 9600 on cloudelastic1009 is CRITICAL: CRITICAL - elasticsearch inactive shards 272 threshold =0.15 breach: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1367, relocating_shards: 0, initializing_shards: 0, unassigned_shard [20:21:49] delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 83.40451494813911 https://wikitech.wikimedia.org/wiki/Search%23Administration [20:24:42] arlolra: concerns about squeezing in another patch? [20:25:30] !log arlolra@deploy1003 Finished scap sync-world: Backport for Add "proofread-index" to LINTABLE_CONTENT_MODELS (T441874) ([[gerrit:1357368|1.48.0-wmf.1]], [[gerrit:1357370|1.47.0-wmf.24]]) (duration: 18m 59s) [20:25:36] T441874: Consolidate configuration for when Linting content models - https://phabricator.wikimedia.org/T441874 [20:26:26] none I guess :) [20:27:38] sukhe: no, there was a failure when trying to deploy, see above [20:29:12] !log arlolra@deploy1003 Started scap sync-world: Backport for [[gerrit:1357389|Undeploy Parsoid readviews on hewikisource (T441087)]] [20:29:19] 16:15:43 < arlolra> Error: Failed to render chart: exit status 1: Error: no cached repo found. (try 'helm repo update'): error loading /var/cache/helm/repository/wmf-stable-index.yaml: empty index.yaml file [20:29:19] T441087: Parsoid deletes text between rendering-transparent nodes (category links, empty
tags) inside a block element in template output - https://phabricator.wikimedia.org/T441087 [20:29:32] arlolra: so did it work this time? [20:29:47] It did go through when retrying, yes [20:29:49] I have no idea what it is anyway but I can ask around if it didn't [20:29:52] ok! [20:29:57] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1282 (T441633)', diff saved to https://phabricator.wikimedia.org/P96931 and previous config saved to /var/cache/conftool/dbconfig/20261020-202957-ladsgroup.json [20:30:15] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1287.eqiad.wmnet with reason: Maintenance [20:30:17] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1287 (T441633)', diff saved to https://phabricator.wikimedia.org/P96932 and previous config saved to /var/cache/conftool/dbconfig/20261020-203016-ladsgroup.json [20:30:50] !log bking@cumin2003 START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet [20:33:23] !log arlolra@deploy1003 ssastry, arlolra: Backport for [[gerrit:1357389|Undeploy Parsoid readviews on hewikisource (T441087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:34:01] !log arlolra@deploy1003 ssastry, arlolra: Continuing with deployment [20:35:21] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1287 (T441633)', diff saved to https://phabricator.wikimedia.org/P96933 and previous config saved to /var/cache/conftool/dbconfig/20261020-203520-ladsgroup.json [20:37:59] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet [20:38:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:39:42] !log arlolra@deploy1003 Finished scap sync-world: Backport for [[gerrit:1357389|Undeploy Parsoid readviews on hewikisource (T441087)]] (duration: 10m 37s) [20:39:48] T441087: Parsoid deletes text between rendering-transparent nodes (category links, empty
tags) inside a block element in template output - https://phabricator.wikimedia.org/T441087 [20:41:00] I am done with my patches if anyone else needs the window [20:41:13] I would like to deploy some security patches yes [20:42:43] All yours. Note in the scrollback that I saw a transient error when deployiing [20:43:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:45:23] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1287', diff saved to https://phabricator.wikimedia.org/P96934 and previous config saved to /var/cache/conftool/dbconfig/20261020-204522-ladsgroup.json [20:48:20] RESOLVED: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh [20:48:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:52:51] FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from gitlab.discovery.wmnet in ulsfo #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=ulsfo&var-cluster=text&var-origin=gitlab.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [20:53:13] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [20:53:13] !log bking@cumin2003 END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) [20:53:26] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [20:53:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:55:24] !incidents [20:55:24] 8631 (ACKED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet ulsfo) [20:55:25] 8630 (RESOLVED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet esams) [20:55:25] 8629 (RESOLVED) ProbeDown sre (10.2.2.96 ip4 pki:443 probes/service http_pki_ip4 eqiad) [20:55:25] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1287', diff saved to https://phabricator.wikimedia.org/P96935 and previous config saved to /var/cache/conftool/dbconfig/20261020-205524-ladsgroup.json [20:55:25] 8628 (RESOLVED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet eqiad) [20:55:25] 8627 (RESOLVED) NELHigh sre (thanos-rule@main tcp.timed_out) [20:56:20] FIRING: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh [20:58:47] FIRING: [3x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261020T2100) [21:01:20] RESOLVED: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh [21:04:56] FIRING: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:05:10] Please wait, I'm still working on preparing the security deployment [21:05:27] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1287 (T441633)', diff saved to https://phabricator.wikimedia.org/P96937 and previous config saved to /var/cache/conftool/dbconfig/20261020-210526-ladsgroup.json [21:05:45] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance [21:08:20] FIRING: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh [21:08:50] FIRING: GnmiInterfaceCountersDrop: ... [21:08:56] lsw1-a5-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=lsw1-a5-codfw:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [21:09:56] RESOLVED: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:10:36] Deploying now [21:12:39] RoanKattouw: after this deploy, please ping me [21:13:20] RESOLVED: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh [21:13:47] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-magru:xe-0/1/3 (reserved for moved telxius transport to eqiad) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-magru:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [21:14:39] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [21:16:20] FIRING: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh [21:18:25] PROBLEM - Host cloudelastic1008 is DOWN: PING CRITICAL - Packet loss = 100% [21:21:20] RESOLVED: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh [21:21:58] kostajh: I'm done [21:22:16] RoanKattouw: thanks! [21:24:57] !log thcipriani@deploy1003 Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week [21:27:51] RESOLVED: ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from gitlab.discovery.wmnet in ulsfo #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=ulsfo&var-cluster=text&var-origin=gitlab.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [21:37:56] FIRING: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:38:58] RECOVERY - OpenSearch health check for shards on 9400 on cloudelastic1011 is OK: OK - elasticsearch status cloudelastic-omega-eqiad: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1422, relocating_shards: 0, initializing_shards: 35, unassigned_shards: 210, delayed_una [21:38:58] shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 85.30293941211757 https://wikitech.wikimedia.org/wiki/Search%23Administration [21:39:07] !log bking@cumin2003 END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838 [21:39:14] T441838: Reboot w[cq]s servers to apply updates - https://phabricator.wikimedia.org/T441838 [21:39:14] RECOVERY - OpenSearch health check for shards on 9400 on cloudelastic1010 is OK: OK - elasticsearch status cloudelastic-omega-eqiad: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1446, relocating_shards: 0, initializing_shards: 35, unassigned_shards: 186, delayed_una [21:39:14] shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 86.74265146970606 https://wikitech.wikimedia.org/wiki/Search%23Administration [21:39:16] RECOVERY - OpenSearch health check for shards on 9600 on cloudelastic1012 is OK: OK - elasticsearch status cloudelastic-psi-eqiad: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1428, relocating_shards: 0, initializing_shards: 35, unassigned_shards: 176, delayed_unassig [21:39:16] ds: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 87.12629652226967 https://wikitech.wikimedia.org/wiki/Search%23Administration [21:39:18] RECOVERY - OpenSearch health check for shards on 9400 on cloudelastic1009 is OK: OK - elasticsearch status cloudelastic-omega-eqiad: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1449, relocating_shards: 0, initializing_shards: 35, unassigned_shards: 183, delayed_una [21:39:18] shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 86.92261547690462 https://wikitech.wikimedia.org/wiki/Search%23Administration [21:39:20] RECOVERY - OpenSearch health check for shards on 9600 on cloudelastic1007 is OK: OK - elasticsearch status cloudelastic-psi-eqiad: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1432, relocating_shards: 0, initializing_shards: 35, unassigned_shards: 172, delayed_unassig [21:39:20] ds: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 87.37034777303234 https://wikitech.wikimedia.org/wiki/Search%23Administration [21:39:34] RECOVERY - OpenSearch health check for shards on 9400 on cloudelastic1007 is OK: OK - elasticsearch status cloudelastic-omega-eqiad: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1472, relocating_shards: 0, initializing_shards: 35, unassigned_shards: 160, delayed_una [21:39:34] shards: 0, number_of_pending_tasks: 3, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 35, active_shards_percent_as_number: 88.30233953209358 https://wikitech.wikimedia.org/wiki/Search%23Administration [21:39:38] RECOVERY - OpenSearch health check for shards on 9600 on cloudelastic1011 is OK: OK - elasticsearch status cloudelastic-psi-eqiad: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1462, relocating_shards: 0, initializing_shards: 34, unassigned_shards: 143, delayed_unassig [21:39:38] ds: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.2007321537523 https://wikitech.wikimedia.org/wiki/Search%23Administration [21:39:44] RECOVERY - OpenSearch health check for shards on 9600 on cloudelastic1010 is OK: OK - elasticsearch status cloudelastic-psi-eqiad: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1468, relocating_shards: 0, initializing_shards: 34, unassigned_shards: 137, delayed_unassig [21:39:44] ds: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.56680902989628 https://wikitech.wikimedia.org/wiki/Search%23Administration [21:39:46] RECOVERY - OpenSearch health check for shards on 9400 on cloudelastic1012 is OK: OK - elasticsearch status cloudelastic-omega-eqiad: cluster_name: cloudelastic-omega-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 832, active_shards: 1486, relocating_shards: 0, initializing_shards: 35, unassigned_shards: 146, delayed_una [21:39:46] shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.14217156568687 https://wikitech.wikimedia.org/wiki/Search%23Administration [21:39:48] RECOVERY - OpenSearch health check for shards on 9600 on cloudelastic1009 is OK: OK - elasticsearch status cloudelastic-psi-eqiad: cluster_name: cloudelastic-psi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 818, active_shards: 1471, relocating_shards: 0, initializing_shards: 34, unassigned_shards: 134, delayed_unassig [21:39:48] ds: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 89.74984746796827 https://wikitech.wikimedia.org/wiki/Search%23Administration [21:40:16] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1007 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 769, active_shards: 1310, relocating_shards: 0, initializing_shards: 35, unassigned_shards: 194, delayed_unassig [21:40:16] ds: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 85.12020792722547 https://wikitech.wikimedia.org/wiki/Search%23Administration [21:40:18] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1012 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 769, active_shards: 1310, relocating_shards: 0, initializing_shards: 35, unassigned_shards: 194, delayed_unassig [21:40:18] ds: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 85.12020792722547 https://wikitech.wikimedia.org/wiki/Search%23Administration [21:40:20] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1009 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 769, active_shards: 1310, relocating_shards: 0, initializing_shards: 35, unassigned_shards: 194, delayed_unassig [21:40:20] ds: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 85.12020792722547 https://wikitech.wikimedia.org/wiki/Search%23Administration [21:40:30] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1011 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 769, active_shards: 1312, relocating_shards: 0, initializing_shards: 35, unassigned_shards: 192, delayed_unassig [21:40:30] ds: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 85.25016244314489 https://wikitech.wikimedia.org/wiki/Search%23Administration [21:41:06] RECOVERY - OpenSearch health check for shards on 9200 on cloudelastic1010 is OK: OK - elasticsearch status cloudelastic-chi-eqiad: cluster_name: cloudelastic-chi-eqiad, status: yellow, timed_out: False, number_of_nodes: 5, number_of_data_nodes: 5, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 769, active_shards: 1316, relocating_shards: 0, initializing_shards: 35, unassigned_shards: 188, delayed_unassig [21:41:06] ds: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 85.51007147498375 https://wikitech.wikimedia.org/wiki/Search%23Administration [21:41:55] !log bking@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cloudelastic1008.eqiad.wmnet with reason: backplane error [21:44:51] FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from gitlab.discovery.wmnet in ulsfo #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=ulsfo&var-cluster=text&var-origin=gitlab.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [21:45:09] !incidents [21:45:09] 8632 (UNACKED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet ulsfo) [21:45:10] 8631 (RESOLVED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet ulsfo) [21:45:10] 8630 (RESOLVED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet esams) [21:45:10] 8629 (RESOLVED) ProbeDown sre (10.2.2.96 ip4 pki:443 probes/service http_pki_ip4 eqiad) [21:45:10] 8628 (RESOLVED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet eqiad) [21:45:15] !ack 8632 [21:45:15] 8632 (ACKED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet ulsfo) [21:47:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [21:47:55] RESOLVED: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:48:55] FIRING: ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:49:10] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [21:49:11] !log bking@cumin2003 END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) [21:49:31] FIRING: [2x] SystemdUnitFailed: generate_vrts_aliases.service on mx-in2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:51:57] !log bking@cumin2003 START - Cookbook sre.wdqs.reboot [21:52:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [21:53:11] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:53:56] RESOLVED: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:54:59] !log bking@cumin2003 START - Cookbook sre.hosts.reboot-single for host wdqs1027.eqiad.wmnet [21:57:55] FIRING: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [21:59:51] FIRING: [3x] ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from gitlab.discovery.wmnet in eqiad #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [22:00:29] !incidents [22:00:34] 8632 (ACKED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet ulsfo) [22:00:34] 8631 (RESOLVED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet ulsfo) [22:00:34] 8630 (RESOLVED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet esams) [22:00:34] 8629 (RESOLVED) ProbeDown sre (10.2.2.96 ip4 pki:443 probes/service http_pki_ip4 eqiad) [22:00:35] 8628 (RESOLVED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet eqiad) [22:02:13] !log bking@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1027.eqiad.wmnet [22:02:56] RESOLVED: [2x] ProbeDown: Service gitlab1004:443 has failed probes (http_gitlab_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gitlab1004:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:03:47] FIRING: [3x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:04:51] FIRING: [3x] ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from gitlab.discovery.wmnet in eqiad #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [22:05:32] !incidents [22:05:32] 8632 (ACKED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet ulsfo) [22:05:33] 8631 (RESOLVED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet ulsfo) [22:05:33] 8630 (RESOLVED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet esams) [22:05:33] 8629 (RESOLVED) ProbeDown sre (10.2.2.96 ip4 pki:443 probes/service http_pki_ip4 eqiad) [22:05:33] 8628 (RESOLVED) ATSBackendErrorsHigh cache_text sre (gitlab.discovery.wmnet eqiad) [22:08:47] FIRING: [4x] BFDdown: BFD session down between cr1-codfw and 195.200.68.139 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [22:08:47] FIRING: [2x] SystemdUnitFailed: gitlab-package-puller.service on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:10:34] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) [22:13:28] PROBLEM - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [22:14:51] RESOLVED: [2x] ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from gitlab.discovery.wmnet in eqiad #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [22:28:47] FIRING: KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1152.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [22:38:47] FIRING: [3x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-magru (195.200.68.139) - group Confed_magru - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [22:38:47] RESOLVED: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:39:31] FIRING: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:49:57] FIRING: [4x] ProbeDown: Service doc1004.eqiad.wmnet:443 has failed probes (http_doc1004_eqiad_wmnet_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:58:38] PROBLEM - Check unit status of push_cross_cluster_settings_9400 on cirrussearch2100 is CRITICAL: CRITICAL: Status of the systemd unit push_cross_cluster_settings_9400 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [22:59:58] PROBLEM - Check unit status of push_cross_cluster_settings_9200 on cirrussearch2081 is CRITICAL: CRITICAL: Status of the systemd unit push_cross_cluster_settings_9200 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [22:59:58] PROBLEM - Check unit status of push_cross_cluster_settings_9600 on cirrussearch2083 is CRITICAL: CRITICAL: Status of the systemd unit push_cross_cluster_settings_9600 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [23:07:20] FIRING: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh [23:08:38] RECOVERY - Check unit status of push_cross_cluster_settings_9400 on cirrussearch2100 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9400 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [23:08:38] PROBLEM - Check unit status of push_cross_cluster_settings_9600 on cirrussearch2080 is CRITICAL: CRITICAL: Status of the systemd unit push_cross_cluster_settings_9600 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [23:08:38] PROBLEM - Check unit status of push_cross_cluster_settings_9400 on cirrussearch2092 is CRITICAL: CRITICAL: Status of the systemd unit push_cross_cluster_settings_9400 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [23:09:58] PROBLEM - Check unit status of push_cross_cluster_settings_9200 on cirrussearch2093 is CRITICAL: CRITICAL: Status of the systemd unit push_cross_cluster_settings_9200 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [23:09:58] RECOVERY - Check unit status of push_cross_cluster_settings_9600 on cirrussearch2083 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9600 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [23:09:58] RECOVERY - Check unit status of push_cross_cluster_settings_9200 on cirrussearch2081 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9200 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [23:11:49] !log thcipriani@deploy1003 Forcefully removing global lock: Pausing deploy lock for quick backport [23:12:02] !log thcipriani@deploy1003 Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 107m 04s) [23:13:28] RECOVERY - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [23:15:18] !log thcipriani@deploy1003 Started scap sync-world: Backport for PhpVersion: Show only the major and minor PHP version ([[gerrit:1357432|1.47.0-wmf.24]], [[gerrit:1357431|1.48.0-wmf.1]]) [23:17:20] RESOLVED: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@eqiad to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh [23:18:38] RECOVERY - Check unit status of push_cross_cluster_settings_9600 on cirrussearch2080 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9600 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [23:18:38] RECOVERY - Check unit status of push_cross_cluster_settings_9400 on cirrussearch2092 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9400 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [23:19:34] !log thcipriani@deploy1003 thcipriani: Backport for PhpVersion: Show only the major and minor PHP version ([[gerrit:1357432|1.47.0-wmf.24]], [[gerrit:1357431|1.48.0-wmf.1]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.