[00:00:09] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox: sync [00:00:17] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [00:00:20] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [00:00:28] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [00:02:49] Amir1: ^ just curious, is there something that doesn't work if you don't run it as root? [00:02:56] (if that's you) [00:05:07] Amir1: and also, no rush but lmk whenever you're done and I'll borrow mw-debug to test an envoy bump [00:07:25] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.17 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:09:02] rzl: that is not me [00:09:08] oh! sorry [00:09:36] I deployed mw via spiderpig and it was done ten minutes ago [00:09:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12482359 (10phaultfinder) [00:09:58] same question to whomstever that is, then :) [00:10:45] Go for it. Already turned off the pc [00:10:55] thanks! have a good night [00:20:04] !log rzl@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-debug: apply [00:20:35] !log rzl@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [00:39:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12482444 (10phaultfinder) [00:40:46] (03PS1) 10RLazarus: shellbox-syntaxhighlight: Update to envoy 1.40.1 for validation [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359430 (https://phabricator.wikimedia.org/T436338) [00:42:24] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1359431 [00:42:24] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1359431 (owner: 10TrainBranchBot) [00:46:34] (03PS1) 10Dzahn: codesearch: bump production frontend image [puppet] - 10https://gerrit.wikimedia.org/r/1359432 (https://phabricator.wikimedia.org/T268199) [00:47:10] (03CR) 10Dzahn: [C:03+2] codesearch: bump production frontend image [puppet] - 10https://gerrit.wikimedia.org/r/1359432 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [00:49:11] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1359431 (owner: 10TrainBranchBot) [00:50:15] 06SRE, 10SRE-Access-Requests, 06Data-Engineering: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12482472 (10Dzahn) [00:53:17] (03CR) 10DLynch: "recheck" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359396 (https://phabricator.wikimedia.org/T432286) (owner: 10DLynch) [00:54:32] 06SRE, 10SRE-Access-Requests, 06Data-Engineering: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12482478 (10Dzahn) [00:54:58] 06SRE, 10SRE-Access-Requests, 06Data-Engineering: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12482482 (10Dzahn) 05Open→03In progress [00:55:20] 06SRE, 10SRE-Access-Requests, 06Data-Engineering: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12482484 (10Dzahn) [00:57:03] (03CR) 10DLynch: [C:03+2] editcheck-headless: upgrade image to 2026-10-27-200626-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359396 (https://phabricator.wikimedia.org/T432286) (owner: 10DLynch) [00:58:01] (03PS1) 10RLazarus: rest-gateway: Update to Envoy v1.40.1 in staging for validation [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359438 (https://phabricator.wikimedia.org/T436338) [00:58:03] (03PS2) 10RLazarus: shellbox-syntaxhighlight: Update to Envoy 1.40.1 for validation [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359430 (https://phabricator.wikimedia.org/T436338) [00:58:05] (03PS1) 10RLazarus: mw-debug: Update to Envoy v1.40.1 for validation [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359439 (https://phabricator.wikimedia.org/T436338) [00:58:09] 06SRE, 10SRE-Access-Requests, 06Data-Engineering: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12482489 (10Dzahn) I sent an email for the out-of-band key verification. [00:58:50] (03Merged) 10jenkins-bot: editcheck-headless: upgrade image to 2026-10-27-200626-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359396 (https://phabricator.wikimedia.org/T432286) (owner: 10DLynch) [01:01:05] !log kemayo@deploy1003 helmfile [staging] START helmfile.d/services/editcheck-headless: apply [01:01:41] !log kemayo@deploy1003 helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply [01:06:06] (03PS2) 10Dzahn: codesearch: show a preview banner on the production frontend [puppet] - 10https://gerrit.wikimedia.org/r/1357359 (https://phabricator.wikimedia.org/T268199) [01:07:58] !log rzl@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-debug: apply [01:08:13] !log rzl@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [01:08:17] finished testing on mw-debug [01:09:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12482517 (10phaultfinder) [01:12:26] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1359446 [01:12:26] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1359446 (owner: 10TrainBranchBot) [01:13:46] (03PS1) 10DLynch: python-webapp: Bump version to 0.0.20 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359447 (https://phabricator.wikimedia.org/T436689) [01:16:34] (03CR) 10DLynch: [C:03+2] python-webapp: Bump version to 0.0.20 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359447 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [01:18:47] (03Merged) 10jenkins-bot: python-webapp: Bump version to 0.0.20 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359447 (https://phabricator.wikimedia.org/T436689) (owner: 10DLynch) [01:19:16] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1359446 (owner: 10TrainBranchBot) [01:19:38] !log kemayo@deploy1003 helmfile [staging] START helmfile.d/services/editcheck-headless: apply [01:19:40] !log kemayo@deploy1003 helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply [01:20:54] !log kemayo@deploy1003 helmfile [staging] START helmfile.d/services/editcheck-headless: apply [01:21:28] !log kemayo@deploy1003 helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply [01:23:28] 06SRE, 10Wikimedia-Mailing-lists, 07Chinese-Sites: Alternative mailing lists for Chinese ArbCom - https://phabricator.wikimedia.org/T442488#12482534 (101F616EMO) The English ArbCom have already set up arbcom-en-b and arbcom-en-c for the same purposes. The only difference is that they don’t need NDA counterpa... [01:25:52] !log kemayo@deploy1003 helmfile [codfw] START helmfile.d/services/editcheck-headless: apply [01:27:01] !log kemayo@deploy1003 helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply [01:29:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12482543 (10phaultfinder) [01:32:42] !log kemayo@deploy1003 helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply [01:33:37] !log kemayo@deploy1003 helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply [01:44:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12482562 (10phaultfinder) [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T0200) [02:01:21] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:04:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12482592 (10phaultfinder) [02:08:58] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 37s) [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:19:13] 10SRE-Access-Requests: Add FIDO-backed SSH keys for fab - https://phabricator.wikimedia.org/T442665 (10fkaelin) 03NEW [02:19:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12482608 (10phaultfinder) [02:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:24:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12482610 (10phaultfinder) [02:31:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at codfw: 15.35% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [02:36:13] FIRING: HaproxyUnavailable: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [02:37:13] FIRING: VarnishUnavailable: varnish-text has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/Varnish#Diagnosing_Varnish_alerts - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DVarnishUnavailable [02:39:51] FIRING: [3x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [02:41:17] !incidents [02:41:18] 8661 (UNACKED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [02:41:18] 8662 (UNACKED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [02:41:18] 8660 (RESOLVED) ATSBackendErrorsHigh cache_text sre (eventstreams.discovery.wmnet drmrs) [02:42:44] FIRING: KubernetesDeploymentUnavailableReplicas: ... [02:42:44] Deployment mw-api-ext.codfw.main in mw-api-ext at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=codfw&var-cluster=k8s&var-namespace=mw-api-ext&var-deployment=mw-api-ext.codfw.main - ... [02:42:44] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [02:44:21] (03CR) 10DLynch: [C:03+1] kafka-jumbo: declare Editing topics [puppet] - 10https://gerrit.wikimedia.org/r/1355383 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [02:44:30] o/ [02:44:51] FIRING: [5x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [02:45:02] not sure what's going on above, yet [02:46:23] seems mw-api-ext is unhappy, pods are crash looping [02:46:35] in codfw [02:47:58] yeah [02:48:10] tons of 503s for /w/api.php across various wikis [02:49:51] FIRING: [6x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [02:49:57] FIRING: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [02:54:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12482625 (10phaultfinder) [02:54:51] FIRING: [6x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [02:54:57] RESOLVED: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [02:57:39] FIRING: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [02:59:51] FIRING: [6x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [03:00:22] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [03:00:57] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to edit enwiki. - https://phabricator.wikimedia.org/T442667#12482648 (10AntiCompositeNumber) SRE are aware and responding to the incident. [03:00:57] FIRING: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:01:11] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to edit enwiki. - https://phabricator.wikimedia.org/T442667#12482650 (10Sumanuil) [03:03:30] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to edit enwiki. - https://phabricator.wikimedia.org/T442667#12482651 (10Sumanuil) Note: I have rollbacker right on enwiki, and that still works. [03:04:51] FIRING: [6x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [03:07:24] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to edit enwiki. - https://phabricator.wikimedia.org/T442667#12482655 (10VeritasSapientiae) p:05Triage→03Unbreak! [03:09:51] FIRING: [6x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [03:13:59] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to edit enwiki. - https://phabricator.wikimedia.org/T442667#12482675 (10ZauberViolino) That's happening everywhere. [03:14:51] FIRING: [5x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [03:15:57] RESOLVED: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:16:39] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to edit enwiki. - https://phabricator.wikimedia.org/T442667#12482678 (10Oficial_K) I was trying to upload an image through the Commons Android app, and it also says "unsuccessful response". Wikipedia in Spanish also shows the 503 error code. [03:16:57] FIRING: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:19:29] that doesn't look good [03:19:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12482680 (10phaultfinder) [03:19:51] FIRING: [5x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [03:20:35] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482681 (10suzuneu) [03:21:15] 06SRE, 07Wikimedia-Incident, 07Wikimedia-production-error: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482684 (10suzuneu) [03:21:57] RESOLVED: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:24:51] FIRING: [6x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [03:27:44] FIRING: [2x] KubernetesDeploymentUnavailableReplicas: Deployment mw-api-ext.codfw.canary in mw-api-ext at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [03:29:51] FIRING: [6x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [03:32:18] 06SRE, 07Wikimedia-Incident, 07Wikimedia-production-error: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482701 (10JJPMaster) It should be noted that Echo is working quite inconsistently for me on English Wikibooks. [03:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:34:51] FIRING: [5x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [03:34:57] FIRING: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:39:51] FIRING: [6x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [03:39:57] RESOLVED: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:41:57] FIRING: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:44:51] FIRING: [6x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [03:49:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12482709 (10phaultfinder) [03:49:51] FIRING: [6x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [03:51:57] RESOLVED: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:54:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12482712 (10phaultfinder) [03:54:51] RESOLVED: [4x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [03:56:13] RESOLVED: HaproxyUnavailable: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [03:56:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at codfw: 23.15% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [03:57:12] RESOLVED: VarnishUnavailable: varnish-text has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/Varnish#Diagnosing_Varnish_alerts - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DVarnishUnavailable [03:59:22] 06SRE, 07Wikimedia-Incident, 07Wikimedia-production-error: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482718 (10Aplucas0703) Someone should probably update the Wikimedia Status page to note the issues with editing so people know: https://www.wik... [04:00:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [04:05:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [04:10:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [04:11:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at codfw: 16.48% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:14:43] FIRING: HaproxyUnavailable: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [04:15:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [04:15:13] FIRING: VarnishUnavailable: varnish-text has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/Varnish#Diagnosing_Varnish_alerts - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DVarnishUnavailable [04:16:51] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [04:18:57] FIRING: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:20:08] !incidents [04:20:08] 8663 (ACKED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [04:20:08] 8664 (ACKED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [04:20:09] 8662 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [04:20:09] 8661 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [04:20:09] 8660 (RESOLVED) ATSBackendErrorsHigh cache_text sre (eventstreams.discovery.wmnet drmrs) [04:20:35] 06SRE, 07Wikimedia-Incident, 07Wikimedia-production-error: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482735 (10Aplucas0703) Appears fixed per the Wikimedia Stats. Might need to hard refresh Wikipedia pages to start editing again. https://www.wi... [04:21:51] FIRING: [4x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [04:23:57] RESOLVED: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:25:42] 06SRE, 07Wikimedia-Incident, 07Wikimedia-production-error: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482741 (10NguoiDungKhongDinhDanh) > Appears fixed per the Wikimedia Stats. @Aplucas0703 Are you sure? It says //No incidents reported today//... [04:26:51] FIRING: [5x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [04:29:42] !incidents [04:29:42] 8663 (ACKED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [04:29:42] 8664 (ACKED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [04:29:43] 8662 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [04:29:43] 8661 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [04:29:43] 8660 (RESOLVED) ATSBackendErrorsHigh cache_text sre (eventstreams.discovery.wmnet drmrs) [04:30:57] FIRING: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:31:51] FIRING: [5x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [04:31:51] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482747 (10JJMC89) [04:35:51] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482755 (10AntiCompositeNumber) {F111062099} Things improved, then un-improved. The status page is manually updated by the folks who are currently trying to fix the site inste... [04:36:51] FIRING: [5x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [04:37:06] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482781 (10AntiCompositeNumber) [04:37:23] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482784 (1020261028.1319) Dokdo Korea [04:37:25] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482783 (10Aplucas0703) >>! In T442667#12482741, @NguoiDungKhongDinhDanh wrote: >> Appears fixed per the Wikimedia Stats. > > @Aplucas0703 Are you sure? It says //No incidents... [04:37:29] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482785 (1020261028.1319) 독도는 우리땅 [04:38:00] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482787 (10Marostegui) We are working on this. [04:40:57] RESOLVED: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:41:50] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482793 (10261028.1308) I WANT TO VANDALIZE PHABRICATOR UNTIL 503 ERROR IS RESOLVED. [04:41:51] FIRING: [6x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [04:44:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12482796 (10phaultfinder) [04:45:45] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482798 (10261028.1307) DOKDO IS KOREAN TERRITORY!! CAPITAL PUNISHMEMT TO KIM JONG UN!! [04:45:57] FIRING: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:46:25] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482799 (10261028.1307) '''천안문''', 사라진 목소리, 당신도 그 일부가 될 수 있습니다. [04:46:43] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482800 (10261028.1307) **Tiananmen Square**, the vanished voice, you may be a part of it too. [04:46:51] FIRING: [6x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [04:47:22] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482801 (10261028.1307) **天安门**,消失的声音,你也可以成为其中的一部分。 [04:47:37] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [04:47:39] RESOLVED: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [04:47:39] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482802 (10261028.1307) **天安門**,消失的聲音,你也可以成爲其中的一部分。 [04:47:51] PROBLEM - Host wikikube-worker2337 is DOWN: PING CRITICAL - Packet loss = 100% [04:49:38] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482819 (10AntiCompositeNumber) [04:50:06] Kemayo: you around? [04:50:57] RESOLVED: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [04:51:51] FIRING: [6x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [04:52:42] marostegui: I could be if needed. [04:53:19] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [04:53:25] Kemayo: sorry, we are troubleshooting an issue with pods and I was wondering if your 1AM UTC deployment could be related but effie has pointed ouit that it is unlikely [04:54:43] RESOLVED: HaproxyUnavailable: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [04:55:05] marostegui: I don’t think my deploy should have touched anything outside of editcheck-headless, at least. [04:55:13] RESOLVED: VarnishUnavailable: varnish-text has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/Varnish#Diagnosing_Varnish_alerts - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DVarnishUnavailable [04:55:14] yep, thanks Kemayo! [04:56:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at codfw: 22.81% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [04:56:51] RESOLVED: [4x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [04:58:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [05:04:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12482845 (10phaultfinder) [05:04:55] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST pods) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [05:14:55] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST pods) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [05:18:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [05:26:39] (03CR) 10Physikerwelt: "there are two problems. I can not access wmflabs at the moment due to the php security issue and second I don’t have enough quota to test " [puppet] - 10https://gerrit.wikimedia.org/r/1238335 (https://phabricator.wikimedia.org/T439324) (owner: 10Physikerwelt) [05:30:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at codfw: 17.33% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:34:12] FIRING: VarnishUnavailable: varnish-text has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/Varnish#Diagnosing_Varnish_alerts - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DVarnishUnavailable [05:34:13] FIRING: HaproxyUnavailable: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [05:34:21] !ack [05:34:22] !incidents [05:34:22] 8665 (ACKED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [05:34:22] 8666 (ACKED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [05:34:23] 8665 (ACKED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [05:34:23] 8666 (ACKED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [05:34:23] 8664 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [05:34:23] 8663 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [05:34:23] 8662 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [05:34:24] 8661 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [05:35:40] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482862 (10Thilio) {F111066232} The same issue bot backbone shows, error 503 Server Error: Service Unavailable [05:39:12] RESOLVED: VarnishUnavailable: varnish-text has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/Varnish#Diagnosing_Varnish_alerts - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DVarnishUnavailable [05:39:13] RESOLVED: HaproxyUnavailable: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [05:40:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at codfw: 23.83% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [05:42:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [05:47:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [05:48:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST pods) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [05:49:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12482868 (10phaultfinder) [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T0600) [06:17:44] RESOLVED: [2x] KubernetesDeploymentUnavailableReplicas: Deployment mw-api-ext.codfw.canary in mw-api-ext at codfw has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [06:18:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST pods) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=codfw&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [06:19:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12482890 (10phaultfinder) [06:27:28] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482894 (10Marostegui) We've mitigated the issue. [06:33:50] 06SRE, 07Wikimedia-Incident: Error 503 when attempting to request Wikimedia wikis via API - https://phabricator.wikimedia.org/T442667#12482897 (10Marostegui) 05Open→03Resolved a:03Marostegui [06:54:10] (03CR) 10Hashar: [C:03+1] "Thanks for dealing with it! +1 cause I am happy to see being addressed 😎" [puppet] - 10https://gerrit.wikimedia.org/r/1359258 (https://phabricator.wikimedia.org/T441498) (owner: 10Hnowlan) [07:00:05] Amir1, urbanecm, and awight: It is that lovely time of the day again! You are hereby commanded to deploy UTC morning backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T0700). [07:00:05] No Gerrit patches in the queue for this window AFAICS. [07:01:44] :) [07:02:01] then I am going to have a coffee! [07:11:09] (03PS1) 10Hashar: Update Zuul plugin to align CSS styling [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1359738 (https://phabricator.wikimedia.org/T441684) [07:20:56] I am going to do a quick deploky for Gerrit in order to update the Zuul plugin ^ [07:21:14] (03CR) 10Hashar: [C:03+2] Update Zuul plugin to align CSS styling [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1359738 (https://phabricator.wikimedia.org/T441684) (owner: 10Hashar) [07:21:51] (03Merged) 10jenkins-bot: Update Zuul plugin to align CSS styling [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1359738 (https://phabricator.wikimedia.org/T441684) (owner: 10Hashar) [07:22:36] !log hashar@deploy1003 Started deploy [gerrit/gerrit@b02d300]: Update Zuul plugin to align CSS styling - T441684 [07:22:44] T441684: Gerrit Zuul plugin style mismatches related changes styling - https://phabricator.wikimedia.org/T441684 [07:22:51] !log hashar@deploy1003 Finished deploy [gerrit/gerrit@b02d300]: Update Zuul plugin to align CSS styling - T441684 (duration: 00m 14s) [07:24:53] (03PS1) 10Muehlenhoff: Remove email addresses from offboarded users [puppet] - 10https://gerrit.wikimedia.org/r/1359739 [07:30:07] (03CR) 10Slyngshede: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1359739 (owner: 10Muehlenhoff) [07:33:28] restarting Gerrit now [07:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:35:46] FIRING: GerritHAProxyBackendUnavailable: Gerrit backend is unavilable for tcp-proxy (HAProxy) gerrit_ssh - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyBackendUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyBackendUnavailable [07:36:46] FIRING: [2x] GerritHAProxyServiceUnavailable: Gerrit tcp-proxy (HAProxy) service gerrit_ssh is DOWN in codfw - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyServiceUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyServiceUnavailable [07:36:55] !log Restarted Gerrit on gerrit1003 [07:36:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:37:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:39:46] FIRING: GerritHigh5xxRatio: High rate of HTTP 5xx responses on gerrit1003 - TODO - https://grafana.wikimedia.org/d/nX8li17Sk/gerrit-overview-sre-collab - https://alerts.wikimedia.org/?q=alertname%3DGerritHigh5xxRatio [07:40:46] RESOLVED: [4x] GerritHAProxyBackendUnavailable: Gerrit backend is unavilable for tcp-proxy (HAProxy) gerrit_ssh - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyBackendUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyBackendUnavailable [07:41:46] RESOLVED: [2x] GerritHAProxyServiceUnavailable: Gerrit tcp-proxy (HAProxy) service gerrit_ssh is DOWN in codfw - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyServiceUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyServiceUnavailable [07:42:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:43:51] (03CR) 10Muehlenhoff: [C:03+2] Remove email addresses from offboarded users [puppet] - 10https://gerrit.wikimedia.org/r/1359739 (owner: 10Muehlenhoff) [07:44:46] RESOLVED: GerritHigh5xxRatio: High rate of HTTP 5xx responses on gerrit1003 - TODO - https://grafana.wikimedia.org/d/nX8li17Sk/gerrit-overview-sre-collab - https://alerts.wikimedia.org/?q=alertname%3DGerritHigh5xxRatio [07:52:09] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1359311 (https://phabricator.wikimedia.org/T441854) (owner: 10CWilliams) [07:58:12] (03PS1) 10Marostegui: db2252: Remove note. [puppet] - 10https://gerrit.wikimedia.org/r/1359748 [07:59:18] (03PS1) 10Muehlenhoff: Record LDAP access for sancheza [puppet] - 10https://gerrit.wikimedia.org/r/1359749 [08:01:18] (03CR) 10Jelto: [C:03+2] spiderpig: relative-trending deployment (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359226 (https://phabricator.wikimedia.org/T442341) (owner: 10JavierMonton) [08:01:32] (03CR) 10Marostegui: [C:03+2] db2252: Remove note. [puppet] - 10https://gerrit.wikimedia.org/r/1359748 (owner: 10Marostegui) [08:03:18] (03CR) 10JMeybohm: [C:03+1] shellbox-syntaxhighlight: Update to Envoy 1.40.1 for validation [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359430 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [08:03:24] (03CR) 10JMeybohm: [C:03+1] rest-gateway: Update to Envoy v1.40.1 in staging for validation [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359438 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [08:03:39] (03CR) 10JMeybohm: [C:03+1] mw-debug: Update to Envoy v1.40.1 for validation [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359439 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [08:06:12] (03CR) 10Brouberol: [C:03+1] "Nicely done, thank you!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359340 (https://phabricator.wikimedia.org/T433696) (owner: 10Aqu) [08:07:21] (03CR) 10Muehlenhoff: [C:03+2] Record LDAP access for sancheza [puppet] - 10https://gerrit.wikimedia.org/r/1359749 (owner: 10Muehlenhoff) [08:09:00] !log failover Ganeti master in eqiad to ganeti1048 [08:09:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:11:57] PROBLEM - ganeti-wconfd running on ganeti1046 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 111 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [08:12:03] (03CR) 10JMeybohm: [C:03+1] Revert "trafficserver: Remove mwdebug-next XWD" [puppet] - 10https://gerrit.wikimedia.org/r/1359316 (owner: 10Clément Goubert) [08:12:12] (03PS1) 10Bartosz Wójtowicz: ml-services: lower reference-need CPU from 22 to 14 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359751 [08:12:29] !log failover Ganeti master in codfw to ganeti2048 [08:12:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:15:15] PROBLEM - ganeti-wconfd running on ganeti2047 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 110 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [08:16:23] (03CR) 10Jelto: [C:03+2] cdn: exclude miscweb from ATSBackendErrorsHigh [alerts] - 10https://gerrit.wikimedia.org/r/1359216 (https://phabricator.wikimedia.org/T442366) (owner: 10Jelto) [08:16:57] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet [08:16:58] !log jmm@cumin2003 END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2034.codfw.wmnet [08:17:28] (03PS2) 10Bartosz Wójtowicz: ml-services: lower reference-need CPU request from 22 to 14 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359751 [08:17:43] !log installing clamav security updates [08:17:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:19:50] (03Merged) 10jenkins-bot: cdn: exclude miscweb from ATSBackendErrorsHigh [alerts] - 10https://gerrit.wikimedia.org/r/1359216 (https://phabricator.wikimedia.org/T442366) (owner: 10Jelto) [08:28:28] (03PS5) 10CWilliams: backup: adding initial offsite manifest [puppet] - 10https://gerrit.wikimedia.org/r/1359311 (https://phabricator.wikimedia.org/T441854) [08:29:58] (03PS1) 10Bartosz Wójtowicz: aptrepo: add ROCm 7.2.3 packages to wikimedia trixie mirror [puppet] - 10https://gerrit.wikimedia.org/r/1359752 (https://phabricator.wikimedia.org/T442138) [08:30:49] (03CR) 10CI reject: [V:04-1] aptrepo: add ROCm 7.2.3 packages to wikimedia trixie mirror [puppet] - 10https://gerrit.wikimedia.org/r/1359752 (https://phabricator.wikimedia.org/T442138) (owner: 10Bartosz Wójtowicz) [08:31:26] (03CR) 10CWilliams: [C:03+2] backup: adding initial offsite manifest [puppet] - 10https://gerrit.wikimedia.org/r/1359311 (https://phabricator.wikimedia.org/T441854) (owner: 10CWilliams) [08:31:47] (03CR) 10CWilliams: [C:03+1] backup: adding initial offsite manifest [puppet] - 10https://gerrit.wikimedia.org/r/1359311 (https://phabricator.wikimedia.org/T441854) (owner: 10CWilliams) [08:33:08] !log installing perl security updates [08:33:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:33:18] (03PS2) 10Bartosz Wójtowicz: aptrepo: add ROCm 7.2.3 packages to wikimedia trixie mirror [puppet] - 10https://gerrit.wikimedia.org/r/1359752 (https://phabricator.wikimedia.org/T442138) [08:33:45] (03CR) 10CWilliams: [C:03+2] backup: adding initial offsite manifest [puppet] - 10https://gerrit.wikimedia.org/r/1359311 (https://phabricator.wikimedia.org/T441854) (owner: 10CWilliams) [08:34:27] FIRING: ProbeDown: Service vrts1003:1443 has failed probes (http_ticket_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#vrts1003:1443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [08:35:51] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet [08:35:52] !log jmm@cumin2003 END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2034.codfw.wmnet [08:36:24] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2034.codfw.wmnet [08:36:56] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti2034.codfw.wmnet [08:37:57] (03CR) 10Marostegui: "Just a comment, nothing major. If anything it can be addressed at later stages." [puppet] - 10https://gerrit.wikimedia.org/r/1359311 (https://phabricator.wikimedia.org/T441854) (owner: 10CWilliams) [08:38:12] (03PS1) 10Ayounsi: Revert "eqsin: drain HE transport to eqiad" [homer/public] - 10https://gerrit.wikimedia.org/r/1359754 [08:39:27] RESOLVED: ProbeDown: Service vrts1003:1443 has failed probes (http_ticket_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#vrts1003:1443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [08:40:25] !log repooling cp2049 [08:40:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:41:41] (03CR) 10AikoChou: [C:03+1] ml-services: lower reference-need CPU request from 22 to 14 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359751 (owner: 10Bartosz Wójtowicz) [08:42:31] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: lower reference-need CPU request from 22 to 14 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359751 (owner: 10Bartosz Wójtowicz) [08:43:06] (03CR) 10Jelto: [C:03+2] extdist: add HTTP_USER_AGENT configuration [puppet] - 10https://gerrit.wikimedia.org/r/1358229 (https://phabricator.wikimedia.org/T442167) (owner: 10Hashar) [08:44:00] (03Merged) 10jenkins-bot: ml-services: lower reference-need CPU request from 22 to 14 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359751 (owner: 10Bartosz Wójtowicz) [08:44:47] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revision-models' for release 'main' . [08:44:52] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12483084 (10phaultfinder) [08:45:00] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2034.codfw.wmnet [08:45:09] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2034.codfw.wmnet [08:46:31] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . [08:48:17] PROBLEM - ganeti-wconfd running on ganeti2033 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 111 (gnt-masterd), command name ganeti-wconfd https://wikitech.wikimedia.org/wiki/Ganeti [08:48:37] !log failover Ganeti master in codfw02 to ganeti2034 [08:48:42] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user niharika29 [08:48:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:48:50] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user niharika29 [08:49:19] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user niharika29 [08:49:27] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user niharika29 [08:49:37] (03CR) 10Ayounsi: [C:04-1] "duplicate of merged I09e73cc98775c88cf370a19c77ff3955b99bf09e" [puppet] - 10https://gerrit.wikimedia.org/r/1310991 (https://phabricator.wikimedia.org/T327300) (owner: 10Cathal Mooney) [08:50:22] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-c4-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [08:50:45] (03CR) 10CWilliams: [C:03+2] backup: adding initial offsite manifest (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359311 (https://phabricator.wikimedia.org/T441854) (owner: 10CWilliams) [08:52:51] (03CR) 10Marostegui: backup: adding initial offsite manifest (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359311 (https://phabricator.wikimedia.org/T441854) (owner: 10CWilliams) [08:54:11] (03CR) 10Jelto: [V:03+1] "PCC SUCCESS (CORE_DIFF 3): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9563/co" [puppet] - 10https://gerrit.wikimedia.org/r/1359337 (owner: 10Jforrester) [08:55:03] (03CR) 10Jelto: [V:03+1 C:03+2] deployment_server::deploy_service::service_catalog: Add Wikifunctions [puppet] - 10https://gerrit.wikimedia.org/r/1359337 (owner: 10Jforrester) [08:56:04] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:56:57] (03PS1) 10Vgutierrez: hieradata: Depool lvs5005 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1359758 (https://phabricator.wikimedia.org/T401832) [08:58:01] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user niharika29 [08:58:10] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user niharika29 [08:59:35] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1359758 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [09:02:17] (03CR) 10Daniel Kertesz: [C:03+1] hieradata: Depool lvs5005 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1359758 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [09:06:28] (03CR) 10Cathal Mooney: [C:03+1] Revert "eqsin: drain HE transport to eqiad" [homer/public] - 10https://gerrit.wikimedia.org/r/1359754 (owner: 10Ayounsi) [09:07:18] (03CR) 10Ayounsi: [C:03+2] Revert "eqsin: drain HE transport to eqiad" [homer/public] - 10https://gerrit.wikimedia.org/r/1359754 (owner: 10Ayounsi) [09:08:19] (03Merged) 10jenkins-bot: Revert "eqsin: drain HE transport to eqiad" [homer/public] - 10https://gerrit.wikimedia.org/r/1359754 (owner: 10Ayounsi) [09:08:42] PROBLEM - OSPF status on cr4-ulsfo is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:09:00] PROBLEM - OSPF status on cr2-eqiad is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:09:02] PROBLEM - OSPF status on cr2-drmrs is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:09:12] PROBLEM - OSPF status on cr2-codfw is CRITICAL: OSPFv2: 3/8 UP : OSPFv3: 3/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:09:39] FIRING: CoreBGPDown: Core BGP session down between cr2-codfw and cr2-eqiad (208.80.153.214) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr2-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [09:10:10] FIRING: [9x] BFDdown: BFD session down between cr2-codfw and 208.80.153.214 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:10:28] !log javiermonton@deploy1003 Started scap deploy-service pageview-trending-relative-next: testing checks for Flink deployments. [09:10:28] !log javiermonton@deploy1003 scap deploy-service pageview-trending-relative-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying pageview-trending-relative-next [09:10:57] PROBLEM - Postfix SMTP on crm2001 is CRITICAL: CRITICAL - Certificate crm2001.codfw.wmnet expires in 15 day(s) (Fri 13 Nov 2026 09:10:00 AM GMT +0000). https://wikitech.wikimedia.org/wiki/Mail%23Troubleshooting [09:12:35] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2033.codfw.wmnet [09:14:26] !log javiermonton@deploy1003 Finished scap deploy-service pageview-trending-relative-next: testing checks for Flink deployments. (duration: 03m 58s) [09:14:39] FIRING: [8x] CoreBGPDown: Core BGP session down between cr2-codfw and cr3-eqsin (103.102.166.153) - group Confed_eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [09:15:10] FIRING: [10x] BFDdown: BFD session down between cr2-codfw and 208.80.153.214 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [09:16:39] (03PS1) 10Kevin Bazira: ml-services: update tts-generator service to fix parallel article generation failures [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359760 (https://phabricator.wikimedia.org/T441983) [09:16:51] (03PS1) 10Hashar: gerrit: add Java 21 on hosts [puppet] - 10https://gerrit.wikimedia.org/r/1359761 (https://phabricator.wikimedia.org/T392465) [09:16:53] (03PS1) 10Hashar: gerrit: switch replica and spare to Java 21 [puppet] - 10https://gerrit.wikimedia.org/r/1359762 (https://phabricator.wikimedia.org/T392465) [09:16:55] (03PS1) 10Hashar: gerrit: switch primary to Java 21 [puppet] - 10https://gerrit.wikimedia.org/r/1359763 (https://phabricator.wikimedia.org/T392465) [09:17:08] (03CR) 10Hashar: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1359761 (https://phabricator.wikimedia.org/T392465) (owner: 10Hashar) [09:17:10] (03CR) 10Hashar: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1359762 (https://phabricator.wikimedia.org/T392465) (owner: 10Hashar) [09:17:12] (03CR) 10Hashar: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1359763 (https://phabricator.wikimedia.org/T392465) (owner: 10Hashar) [09:17:56] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [09:18:17] PROBLEM - OSPF status on cr2-esams is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:18:21] (03CR) 10Kevin Bazira: [C:03+2] ml-services: update tts-generator service to fix parallel article generation failures [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359760 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [09:18:38] jmm@cumin2003 drain-node (PID 1419096) is awaiting input [09:19:50] (03Merged) 10jenkins-bot: ml-services: update tts-generator service to fix parallel article generation failures [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359760 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [09:22:00] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [09:22:05] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [09:22:10] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [09:22:11] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [09:22:15] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [09:22:17] PROBLEM - OSPF status on cr3-eqsin is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [09:22:28] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti2033.codfw.wmnet [09:22:31] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [09:22:47] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [09:23:12] !log installing jq security updates [09:23:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:24:54] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . [09:25:39] !log kevinbazira@deploy1003 helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [09:27:42] (03CR) 10Hashar: "PPC confirms:" [puppet] - 10https://gerrit.wikimedia.org/r/1359761 (https://phabricator.wikimedia.org/T392465) (owner: 10Hashar) [09:29:15] (03CR) 10Hashar: "PCC shows the systemd unit is changed to Java 21:" [puppet] - 10https://gerrit.wikimedia.org/r/1359762 (https://phabricator.wikimedia.org/T392465) (owner: 10Hashar) [09:29:43] (03PS1) 10Muehlenhoff: Add FIDO keys for tlepage [puppet] - 10https://gerrit.wikimedia.org/r/1359766 (https://phabricator.wikimedia.org/T440887) [09:30:30] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2033.codfw.wmnet [09:30:34] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2033.codfw.wmnet [09:34:32] (03PS1) 10Ladsgroup: Fix two bugs on PurgeScoreCache [extensions/ORES] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359767 (https://phabricator.wikimedia.org/T440358) [09:34:45] (03PS1) 10Ladsgroup: Fix two bugs on PurgeScoreCache [extensions/ORES] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359768 (https://phabricator.wikimedia.org/T440358) [09:34:56] (03CR) 10Ladsgroup: [C:03+2] Fix two bugs on PurgeScoreCache [extensions/ORES] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359767 (https://phabricator.wikimedia.org/T440358) (owner: 10Ladsgroup) [09:34:59] (03CR) 10Ladsgroup: [C:03+2] Fix two bugs on PurgeScoreCache [extensions/ORES] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359768 (https://phabricator.wikimedia.org/T440358) (owner: 10Ladsgroup) [09:36:47] (03Merged) 10jenkins-bot: Fix two bugs on PurgeScoreCache [extensions/ORES] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359767 (https://phabricator.wikimedia.org/T440358) (owner: 10Ladsgroup) [09:37:39] (03Merged) 10jenkins-bot: Fix two bugs on PurgeScoreCache [extensions/ORES] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359768 (https://phabricator.wikimedia.org/T440358) (owner: 10Ladsgroup) [09:39:57] !log ladsgroup@deploy1003 Started scap sync-world: Backport for Fix two bugs on PurgeScoreCache (T440358) ([[gerrit:1359767|1.47.0-wmf.24]], [[gerrit:1359768|1.48.0-wmf.1]]) [09:40:04] T440358: Lots of old data in ores_classification table - https://phabricator.wikimedia.org/T440358 [09:44:23] (03PS1) 10Jelto: phabricator: auto restart phd service [puppet] - 10https://gerrit.wikimedia.org/r/1359770 (https://phabricator.wikimedia.org/T442177) [09:44:25] !log ladsgroup@deploy1003 ladsgroup: Backport for Fix two bugs on PurgeScoreCache (T440358) ([[gerrit:1359767|1.47.0-wmf.24]], [[gerrit:1359768|1.48.0-wmf.1]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [09:45:34] (03CR) 10Brouberol: [C:03+1] "Reviewed at Arzhel's request." [puppet] - 10https://gerrit.wikimedia.org/r/1355384 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [09:45:36] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [09:46:31] (03CR) 10Jelto: [V:03+1] "PCC SUCCESS (NOOP 1 CORE_DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1359770 (https://phabricator.wikimedia.org/T442177) (owner: 10Jelto) [09:47:09] (03PS1) 10Jgiannelos: thumbor: Make async engine loading configurable [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359771 (https://phabricator.wikimedia.org/T442031) [09:47:52] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [09:47:57] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [09:48:02] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [09:48:03] 10ops-eqiad, 06SRE, 06DC-Ops, 07Essential-Work: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12483283 (10klausman) >>! In T438161#12476918, @elukey wrote: > @klausman IIRC you were working on BIOS/etc.. settings suggested by AMD for the MI3XX hosts.. Is there any cha... [09:48:10] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [09:49:55] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [09:49:59] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative-next: apply [09:50:05] !log installing nodejs security updates on bookworm [09:50:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:50:26] !log javiermonton@deploy1003 Started scap deploy-service pageview-trending-relative-next: Test Flink checks in spiderpig [09:50:26] !log javiermonton@deploy1003 scap deploy-service pageview-trending-relative-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying pageview-trending-relative-next [09:51:17] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for Fix two bugs on PurgeScoreCache (T440358) ([[gerrit:1359767|1.47.0-wmf.24]], [[gerrit:1359768|1.48.0-wmf.1]]) (duration: 11m 20s) [09:51:24] T440358: Lots of old data in ores_classification table - https://phabricator.wikimedia.org/T440358 [09:51:47] !log javiermonton@deploy1003 Finished scap deploy-service pageview-trending-relative-next: Test Flink checks in spiderpig (duration: 01m 21s) [09:52:10] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user amire80 [09:52:23] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user amire80 [09:53:09] (03CR) 10Slyngshede: [C:03+1] Add FIDO keys for tlepage [puppet] - 10https://gerrit.wikimedia.org/r/1359766 (https://phabricator.wikimedia.org/T440887) (owner: 10Muehlenhoff) [09:55:08] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user amire80 [09:55:21] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user amire80 [09:59:12] (03CR) 10Brouberol: "Thanks, will do." [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) (owner: 10Brouberol) [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T1000) [10:01:58] (03PS1) 10David Caro: maintain_dbusers: continue populating other accounts if one failed [puppet] - 10https://gerrit.wikimedia.org/r/1359773 (https://phabricator.wikimedia.org/T442683) [10:02:49] (03CR) 10CI reject: [V:04-1] maintain_dbusers: continue populating other accounts if one failed [puppet] - 10https://gerrit.wikimedia.org/r/1359773 (https://phabricator.wikimedia.org/T442683) (owner: 10David Caro) [10:06:15] (03CR) 10Dpogorzelski: [C:03+2] "I will merge this and if it's undesired poke me :)" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359259 (https://phabricator.wikimedia.org/T441912) (owner: 10Dpogorzelski) [10:07:19] !log dpogorzelski@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [10:07:29] (03PS2) 10David Caro: maintain_dbusers: continue populating other accounts if one failed [puppet] - 10https://gerrit.wikimedia.org/r/1359773 (https://phabricator.wikimedia.org/T442683) [10:07:51] !log dpogorzelski@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [10:08:06] !log dpogorzelski@deploy1003 helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [10:08:33] !log dpogorzelski@deploy1003 helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [10:08:51] !log dpogorzelski@deploy1003 helmfile [codfw] START helmfile.d/services/rest-gateway: apply [10:09:12] !log dpogorzelski@deploy1003 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [10:11:17] (03PS1) 10Aqu: Bump blunderbuss image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359774 (https://phabricator.wikimedia.org/T435292) [10:13:19] (03CR) 10Clément Goubert: [C:03+2] Revert "trafficserver: Remove mwdebug-next XWD" [puppet] - 10https://gerrit.wikimedia.org/r/1359316 (owner: 10Clément Goubert) [10:14:55] (03CR) 10Joal: [C:03+1] "LGTM!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359774 (https://phabricator.wikimedia.org/T435292) (owner: 10Aqu) [10:20:21] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO keys for tlepage [puppet] - 10https://gerrit.wikimedia.org/r/1359766 (https://phabricator.wikimedia.org/T440887) (owner: 10Muehlenhoff) [10:21:29] (03CR) 10Btullis: [C:03+2] Bump blunderbuss image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359774 (https://phabricator.wikimedia.org/T435292) (owner: 10Aqu) [10:21:36] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2047.codfw.wmnet [10:23:08] (03Merged) 10jenkins-bot: Bump blunderbuss image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359774 (https://phabricator.wikimedia.org/T435292) (owner: 10Aqu) [10:24:31] !log installing Perl security updates [10:24:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:24:42] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti2047.codfw.wmnet [10:26:46] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host apt2002.wikimedia.org [10:26:49] PROBLEM - Host ml-staging-etcd2002 is DOWN: PING CRITICAL - Packet loss = 100% [10:29:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12483509 (10phaultfinder) [10:30:02] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2047.codfw.wmnet [10:30:31] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2047.codfw.wmnet [10:31:17] RECOVERY - Host ml-staging-etcd2002 is UP: PING OK - Packet loss = 0%, RTA = 36.04 ms [10:32:25] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt2002.wikimedia.org [10:33:04] (03PS1) 10David Caro: maintain_dbusers: add a stat to track population errors [puppet] - 10https://gerrit.wikimedia.org/r/1359775 (https://phabricator.wikimedia.org/T442688) [10:33:08] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [10:34:19] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [10:35:59] !log jmm@cumin2003 START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti1046.eqiad.wmnet [10:39:47] (03CR) 10Vgutierrez: [C:03+2] hieradata: Depool lvs5005 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1359758 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [10:40:17] RECOVERY - OSPF status on cr2-esams is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:40:21] RECOVERY - OSPF status on cr3-eqsin is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:40:37] RECOVERY - OSPF status on cr4-ulsfo is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:40:45] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ganeti1046.eqiad.wmnet [10:40:59] RECOVERY - OSPF status on cr2-eqiad is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:41:00] (03PS1) 10JavierMonton: spiderpig: Add pageview-trending-relative [puppet] - 10https://gerrit.wikimedia.org/r/1359776 (https://phabricator.wikimedia.org/T442341) [10:41:03] RECOVERY - OSPF status on cr2-drmrs is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:41:11] RECOVERY - OSPF status on cr2-codfw is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [10:42:10] (03PS1) 10Ozge: profile::k8s::deployment_server: add config for llm-batch [puppet] - 10https://gerrit.wikimedia.org/r/1359777 (https://phabricator.wikimedia.org/T442150) [10:42:33] !log vgutierrez@cumin2003 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5005.*} and A:liberica [10:42:56] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5005.*} and A:liberica [10:42:57] (03CR) 10CI reject: [V:04-1] profile::k8s::deployment_server: add config for llm-batch [puppet] - 10https://gerrit.wikimedia.org/r/1359777 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [10:44:39] RESOLVED: [8x] CoreBGPDown: Core BGP session down between cr2-codfw and cr3-eqsin (103.102.166.153) - group Confed_eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [10:45:10] RESOLVED: [10x] BFDdown: BFD session down between cr2-codfw and 208.80.153.214 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [10:46:07] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti1046.eqiad.wmnet [10:46:13] !log jmm@cumin2003 END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1046.eqiad.wmnet [10:46:27] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user west1 [10:46:29] 10ops-eqiad, 06SRE, 06DC-Ops, 07Essential-Work: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12483580 (10cmooney) >>! In T438161#12478486, @elukey wrote: > @cmooney after re-reading T438161#12350717 I had a chat with John, and IIUC the 10g ports are available and "bo... [10:46:37] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user west1 [10:46:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:47:09] (03PS1) 10David Caro: maintain_dbusers: add alert when population fails [alerts] - 10https://gerrit.wikimedia.org/r/1359779 (https://phabricator.wikimedia.org/T442688) [10:49:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [10:49:10] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.reimage for host lvs5005.eqsin.wmnet with OS trixie [10:50:06] (03CR) 10CI reject: [V:04-1] maintain_dbusers: add alert when population fails [alerts] - 10https://gerrit.wikimedia.org/r/1359779 (https://phabricator.wikimedia.org/T442688) (owner: 10David Caro) [10:50:13] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.dhcp for host lvs5005.eqsin.wmnet (push config only) [10:50:22] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5005.eqsin.wmnet (push config only) [10:51:43] (03PS2) 10Ozge: profile::k8s::deployment_server: add config for llm-batch [puppet] - 10https://gerrit.wikimedia.org/r/1359777 (https://phabricator.wikimedia.org/T442150) [10:54:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [10:54:24] !incidents [10:54:25] 8666 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [10:54:25] 8665 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [10:54:25] 8664 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [10:54:25] 8663 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [10:54:25] 8662 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [10:54:26] 8661 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [10:55:29] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 6 hosts with reason: migrate row a/b network uplinkks from CRs to new Nokia spines [10:55:36] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12483604 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=cd2fa008-95d5-4a9d-8... [10:56:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:57:08] !log shut down cr1-eqiad links connecting asw2-a-eqiad to shift traffic to cr2-eqiad equivalent T441014 [10:57:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:57:15] T441014: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014 [10:59:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12483631 (10phaultfinder) [10:59:52] PROBLEM - Host ps1-a6-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [10:59:52] PROBLEM - Host ps1-b2-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [10:59:52] PROBLEM - Host ps1-a1-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [11:00:05] mvolz: I, the Bot under the Fountain, call upon thee, The Deployer, to do Services – Citoid / Zotero deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T1100). [11:00:08] PROBLEM - Host ps1-a3-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [11:00:08] PROBLEM - Host ps1-a4-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [11:00:08] PROBLEM - Host ps1-a2-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [11:00:08] PROBLEM - Host ps1-a7-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [11:00:08] PROBLEM - Host ps1-a8-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [11:00:33] (03PS1) 10Filippo Giunchedi: rbd2backy2: Detect missing volumes from rbd stderr [puppet] - 10https://gerrit.wikimedia.org/r/1359781 (https://phabricator.wikimedia.org/T442690) [11:00:40] PROBLEM - Host mr1-eqiad IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [11:00:48] RECOVERY - Host ps1-b6-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.91 ms [11:00:48] RECOVERY - Host ps1-b3-eqiad is UP: PING OK - Packet loss = 0%, RTA = 2.04 ms [11:00:48] RECOVERY - Host ps1-a1-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.10 ms [11:00:48] RECOVERY - Host ps1-a8-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.51 ms [11:00:48] RECOVERY - Host ps1-a4-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.55 ms [11:02:40] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user arinaigum [11:03:03] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user arinaigum [11:03:40] ^^ above likely slowliness of bgp detecting the down interfaces to the MR specifically, other BGP sessions are multi-hop and are going around the long way [11:03:47] should be fine anyway [11:04:26] (03PS2) 10David Caro: maintain_dbusers: add alert when population fails [alerts] - 10https://gerrit.wikimedia.org/r/1359779 (https://phabricator.wikimedia.org/T442688) [11:05:11] (03PS1) 10Atsuko: airflow-wikidata: upgrade to Airflow 3.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359783 (https://phabricator.wikimedia.org/T435486) [11:05:22] RESOLVED: CertAlmostExpired: gNMI TLS certificate for lsw1-c4-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [11:05:34] (03PS1) 10WMDE-Fisch: Enable sub-references globally [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359782 (https://phabricator.wikimedia.org/T440304) [11:05:42] RECOVERY - Host mr1-eqiad IPv6 is UP: PING OK - Packet loss = 0%, RTA = 0.94 ms [11:05:54] jouncebot: nowandnext [11:05:54] For the next 0 hour(s) and 54 minute(s): Services – Citoid / Zotero (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T1100) [11:05:54] In 1 hour(s) and 54 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T1300) [11:06:00] (03CR) 10Hnowlan: [C:03+2] prometheus: don't scrape gerrit when using in pontoon (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359258 (https://phabricator.wikimedia.org/T441498) (owner: 10Hnowlan) [11:06:43] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 02 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359782 (https://phabricator.wikimedia.org/T440304) (owner: 10WMDE-Fisch) [11:06:56] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user joelyrookewmde [11:07:05] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user joelyrookewmde [11:08:14] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user joelyrookewmde [11:08:22] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user joelyrookewmde [11:09:15] (03CR) 10Btullis: [C:03+1] "Nice." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359783 (https://phabricator.wikimedia.org/T435486) (owner: 10Atsuko) [11:12:05] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user resquito [11:12:14] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user resquito [11:13:17] (03PS1) 10Ozge: admin_ng: Add llm-batch namespace to LiftWing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359785 (https://phabricator.wikimedia.org/T442150) [11:14:45] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.dhcp for host lvs5005.eqsin.wmnet (remove config only) [11:14:57] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5005.eqsin.wmnet (remove config only) [11:15:41] (03CR) 10Atsuko: [C:03+2] airflow-wikidata: upgrade to Airflow 3.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359783 (https://phabricator.wikimedia.org/T435486) (owner: 10Atsuko) [11:17:04] (03Merged) 10jenkins-bot: airflow-wikidata: upgrade to Airflow 3.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359783 (https://phabricator.wikimedia.org/T435486) (owner: 10Atsuko) [11:18:28] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply [11:18:52] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on lvs5005.eqsin.wmnet with reason: host reimage [11:19:06] !log atsuko@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply [11:19:44] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12483741 (10cmooney) [11:20:01] (03PS1) 10Hnowlan: prometheus: use log level WARN on pushgateway [puppet] - 10https://gerrit.wikimedia.org/r/1359786 (https://phabricator.wikimedia.org/T441498) [11:20:32] (03CR) 10Filippo Giunchedi: [C:03+1] maintain_dbusers: continue populating other accounts if one failed [puppet] - 10https://gerrit.wikimedia.org/r/1359773 (https://phabricator.wikimedia.org/T442683) (owner: 10David Caro) [11:21:06] (03CR) 10Filippo Giunchedi: [C:03+1] maintain_dbusers: add a stat to track population errors [puppet] - 10https://gerrit.wikimedia.org/r/1359775 (https://phabricator.wikimedia.org/T442688) (owner: 10David Caro) [11:21:21] (03CR) 10Filippo Giunchedi: [C:03+1] maintain_dbusers: add alert when population fails [alerts] - 10https://gerrit.wikimedia.org/r/1359779 (https://phabricator.wikimedia.org/T442688) (owner: 10David Caro) [11:21:22] (03PS3) 10Slyngshede: CAS 8.0.3 [software/cas-overlay-template] - 10https://gerrit.wikimedia.org/r/1346564 (https://phabricator.wikimedia.org/T439509) [11:21:24] 06SRE, 06Collaboration-Services, 06SRE Observability: create a new place for prometheus/alertmanager checks not tied to physical machines - https://phabricator.wikimedia.org/T397264#12483754 (10hnowlan) [11:22:24] (03CR) 10David Caro: [C:03+2] maintain_dbusers: add a stat to track population errors [puppet] - 10https://gerrit.wikimedia.org/r/1359775 (https://phabricator.wikimedia.org/T442688) (owner: 10David Caro) [11:22:28] (03CR) 10David Caro: [C:03+2] maintain_dbusers: continue populating other accounts if one failed [puppet] - 10https://gerrit.wikimedia.org/r/1359773 (https://phabricator.wikimedia.org/T442683) (owner: 10David Caro) [11:22:57] !log cwilliams@cumin1004 START - Cookbook sre.backup.offsite [11:22:57] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.backup.offsite (exit_code=0) [11:23:28] (03CR) 10Dbrant: [C:03+2] mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359402 (owner: 10PipelineBot) [11:23:30] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs5005.eqsin.wmnet with reason: host reimage [11:24:23] (03CR) 10David Caro: [C:03+2] maintain_dbusers: add alert when population fails [alerts] - 10https://gerrit.wikimedia.org/r/1359779 (https://phabricator.wikimedia.org/T442688) (owner: 10David Caro) [11:24:56] (03Merged) 10jenkins-bot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359402 (owner: 10PipelineBot) [11:24:57] (03CR) 10Hnowlan: [C:03+2] wmnet: remove graphite, statsd [dns] - 10https://gerrit.wikimedia.org/r/1332687 (https://phabricator.wikimedia.org/T435340) (owner: 10Hnowlan) [11:25:33] !log hnowlan@dns1004 START - running authdns-update [11:26:59] (03Merged) 10jenkins-bot: maintain_dbusers: add alert when population fails [alerts] - 10https://gerrit.wikimedia.org/r/1359779 (https://phabricator.wikimedia.org/T442688) (owner: 10David Caro) [11:27:06] !log dbrant@deploy1003 helmfile [staging] START helmfile.d/services/mobileapps: apply [11:27:25] !log dbrant@deploy1003 helmfile [staging] DONE helmfile.d/services/mobileapps: apply [11:27:43] Ack, I forgot about daylight savings time [11:27:51] * Mvolz will start deploying now [11:27:56] !log dbrant@deploy1003 helmfile [eqiad] START helmfile.d/services/mobileapps: apply [11:28:34] !log hnowlan@dns1004 END - running authdns-update [11:28:40] !log dbrant@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [11:28:50] !log dbrant@deploy1003 helmfile [codfw] START helmfile.d/services/mobileapps: apply [11:29:16] (03PS1) 10Jgiannelos: experiment: Use wmf thumbor fork with async engine support [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1359787 [11:29:34] !log dbrant@deploy1003 helmfile [codfw] DONE helmfile.d/services/mobileapps: apply [11:32:34] (03PS1) 10Matthias Mullie: Squashed diff to master (minus i18n) [extensions/ReaderExperiments] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359789 [11:33:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [11:33:07] FIRING: [7x] ProbeDown: Service restbase2024:7231 has failed probes (http_restbase_discovery_wmnet_ip4) - https://wikitech.wikimedia.org/wiki/RESTBase - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:33:49] (03PS2) 10Jgiannelos: experiment: Use wmf thumbor fork with async engine support [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1359787 (https://phabricator.wikimedia.org/T442031) [11:34:14] (03PS1) 10Matthias Mullie: Squashed diff to master (minus i18n) [extensions/ReaderExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359790 [11:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:34:36] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, October 28 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [extensions/ReaderExperiments] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359789 (owner: 10Matthias Mullie) [11:34:36] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - restbase-https_7443: Servers restbase1042.eqiad.wmnet, restbase1031.eqiad.wmnet, restbase1036.eqiad.wmnet, restbase1037.eqiad.wmnet, restbase1035.eqiad.wmnet, restbase1040.eqiad.wmnet, restbase1045.eqiad.wmnet, restbase1041.eqiad.wmnet are marked down but pooled: restbase-backend_7233: Servers restbase1042.eqiad.wmnet, restbase1031.eqiad.wmnet, restb [11:34:36] eqiad.wmnet, restbase1037.eqiad.wmnet, restbase1035.eqiad.wmnet, restbase1040.eqiad.wmnet, restbase1045.eqiad.wmnet, restbase1041.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [11:34:57] FIRING: [2x] ProbeDown: Service restbase-https:7443 has failed probes (http_restbase-https_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#restbase-https:7443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:34:59] !log cwilliams@cumin1004 START - Cookbook sre.backup.offsite [11:34:59] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.backup.offsite (exit_code=0) [11:35:07] !ack [11:35:08] 8667 (ACKED) [2x] ProbeDown sre (ip4 restbase-https:7443 probes/service http_restbase-https_ip4) [11:35:10] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, October 28 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [extensions/ReaderExperiments] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359789 (owner: 10Matthias Mullie) [11:35:16] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - restbase-https_7443: Servers restbase1042.eqiad.wmnet, restbase1031.eqiad.wmnet, restbase1036.eqiad.wmnet, restbase1037.eqiad.wmnet, restbase1035.eqiad.wmnet, restbase1040.eqiad.wmnet, restbase1045.eqiad.wmnet, restbase1041.eqiad.wmnet are marked down but pooled: restbase-backend_7233: Servers restbase1042.eqiad.wmnet, restbase1031.eqiad.wmnet, restb [11:35:16] eqiad.wmnet, restbase1037.eqiad.wmnet, restbase1035.eqiad.wmnet, restbase1040.eqiad.wmnet, restbase1045.eqiad.wmnet, restbase1041.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [11:35:18] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - restbase-https_7443: Servers restbase2027.codfw.wmnet, restbase2024.codfw.wmnet, restbase2032.codfw.wmnet, restbase2031.codfw.wmnet, restbase2025.codfw.wmnet, restbase2037.codfw.wmnet, restbase2038.codfw.wmnet, restbase2030.codfw.wmnet are marked down but pooled: restbase-backend_7233: Servers restbase2027.codfw.wmnet, restbase2038.codfw.wmnet, restb [11:35:18] codfw.wmnet, restbase2032.codfw.wmnet, restbase2036.codfw.wmnet, restbase2033.codfw.wmnet, restbase2034.codfw.wmnet, restbase2037.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [11:35:41] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, October 28 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [extensions/ReaderExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359790 (owner: 10Matthias Mullie) [11:36:49] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359793 [11:37:04] (03CR) 10Mvolz: [C:03+2] citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359793 (owner: 10PipelineBot) [11:37:51] (03PS1) 10Hnowlan: Revert "wmnet: remove graphite, statsd" [dns] - 10https://gerrit.wikimedia.org/r/1359794 [11:38:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-web - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [11:38:07] FIRING: [60x] ProbeDown: Service restbase1031:7231 has failed probes (http_restbase_discovery_wmnet_ip4) - https://wikitech.wikimedia.org/wiki/RESTBase - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:38:31] (03Merged) 10jenkins-bot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359793 (owner: 10PipelineBot) [11:38:51] FIRING: [2x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [11:38:55] (03CR) 10Hnowlan: [C:03+2] Revert "wmnet: remove graphite, statsd" [dns] - 10https://gerrit.wikimedia.org/r/1359794 (owner: 10Hnowlan) [11:39:39] !log hnowlan@dns1004 START - running authdns-update [11:40:14] !log mvolz@deploy1003 Started scap deploy-service citoid: update [11:40:14] !log mvolz@deploy1003 scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid [11:42:46] !log hnowlan@dns1004 END - running authdns-update [11:46:16] !log mvolz@deploy1003 scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid [11:46:16] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs5005.eqsin.wmnet with OS trixie [11:46:48] !incidents [11:46:48] 8667 (ACKED) [2x] ProbeDown sre (ip4 restbase-https:7443 probes/service http_restbase-https_ip4) [11:46:48] 8666 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [11:46:49] 8665 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [11:46:49] 8664 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [11:46:49] 8663 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [11:46:49] 8662 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [11:46:49] 8661 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [11:47:00] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - restbase-https_7443: Servers restbase2027.codfw.wmnet, restbase2024.codfw.wmnet, restbase2032.codfw.wmnet, restbase2036.codfw.wmnet, restbase2034.codfw.wmnet, restbase2037.codfw.wmnet, restbase2030.codfw.wmnet, restbase2033.codfw.wmnet are marked down but pooled: restbase-backend_7233: Servers restbase2024.codfw.wmnet, restbase2032.codfw.wmnet, restb [11:47:00] codfw.wmnet, restbase2031.codfw.wmnet, restbase2037.codfw.wmnet, restbase2038.codfw.wmnet, restbase2030.codfw.wmnet, restbase2033.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [11:47:12] PROBLEM - Host an-worker1213 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:12] PROBLEM - Host aux-k8s-worker1008 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:26] PROBLEM - Host cirrussearch1111 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:38] PROBLEM - Host cirrussearch1073 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:38] PROBLEM - Host cirrussearch1072 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:38] PROBLEM - Host cirrussearch1084 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:39] PROBLEM - Host cirrussearch1068 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:40] PROBLEM - Host cirrussearch1070 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:41] PROBLEM - Host cirrussearch1071 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:42] PROBLEM - Host cirrussearch1069 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:43] PROBLEM - Host cirrussearch1112 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:43] PROBLEM - Host clouddb1022 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:45] PROBLEM - Host cloudelastic1007 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:45] PROBLEM - Host dse-k8s-worker1015 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:46] PROBLEM - Host stat1008 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:46] PROBLEM - Host ganeti1055 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:51] FIRING: ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from restbase.discovery.wmnet in drmrs #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&viewPanel=12&var-site=drmrs&var-cluster=text&var-origin=restbase.discovery.wmnet - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [11:47:52] PROBLEM - Host dns1004 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:52] PROBLEM - Host dse-k8s-worker1016 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:52] PROBLEM - Host an-conf1004 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:52] PROBLEM - Host zuul1006 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:52] PROBLEM - Host zuul1004 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:52] PROBLEM - Host zuul1007 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:56] PROBLEM - Host conf1007 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:56] PROBLEM - Host dse-k8s-worker1012 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:56] PROBLEM - Host config-master1001 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:58] PROBLEM - Host an-druid1003 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:58] PROBLEM - Host an-presto1016 is DOWN: PING CRITICAL - Packet loss = 100% [11:47:58] PROBLEM - Host an-worker1149 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:02] PROBLEM - Host netbox1003 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:03] PROBLEM - Host pc1021 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:03] PROBLEM - Host restbase1035 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:03] PROBLEM - Host dse-k8s-worker1001 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:04] PROBLEM - Host wikikube-worker1376 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:04] PROBLEM - Host wikikube-worker1048 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:04] PROBLEM - Host db1267 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:05] PROBLEM - Host wikikube-worker1244 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:05] PROBLEM - Host db1235 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:05] PROBLEM - Host dbproxy1022 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:07] PROBLEM - Host db1157 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:07] PROBLEM - Host db1154 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:08] PROBLEM - Host wikikube-worker1384 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:08] PROBLEM - Host cp1100 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:08] !ack [11:48:08] PROBLEM - Host db1260 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:09] 8668 (ACKED) ATSBackendErrorsHigh cache_text sre (restbase.discovery.wmnet drmrs) [11:48:09] PROBLEM - Host es1049 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:09] PROBLEM - Host cp1102 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:09] PROBLEM - Host wikikube-worker1246 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:10] PROBLEM - Host cp1103 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:10] PROBLEM - Host db1273 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:11] PROBLEM - Host db1257 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:11] PROBLEM - Host db1237 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:12] PROBLEM - Host db1255 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:12] PROBLEM - Host wdqs1011 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:13] PROBLEM - Host db1158 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:14] PROBLEM - Host db1250 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:14] PROBLEM - Host cp1101 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:14] PROBLEM - Host aux-k8s-etcd1003 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:15] PROBLEM - Host wikikube-ctrl1005 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:17] PROBLEM - Host db1160 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:17] PROBLEM - Host db1161 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:18] PROBLEM - Host db1159 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:19] !ack [11:48:19] PROBLEM - Host db1209 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:19] All incidents are already acked. [11:48:19] PROBLEM - Host db1185 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:20] PROBLEM - Host db1207 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:21] PROBLEM - Host db1186 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:21] PROBLEM - Host db1226 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:22] PROBLEM - Host db1271 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:22] PROBLEM - Host dse-k8s-worker1009 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:22] PROBLEM - Host dse-k8s-wdqs1001 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:22] PROBLEM - Host dse-k8s-worker1025 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:23] PROBLEM - Host es1041 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:28] PROBLEM - Host es1047 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:28] PROBLEM - Host es1055 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:35] PROBLEM - Host krb1004 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:35] PROBLEM - Host ms-be1083 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:36] !ack [11:48:36] PROBLEM - Host db1227 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:36] You're not allowed to perform this action. [11:48:36] PROBLEM - Host kafka-main1006 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:36] PROBLEM - Host kubestage1006 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:37] PROBLEM - Host an-worker1214 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:37] PROBLEM - Host db1210 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:38] PROBLEM - Host dse-k8s-etcd1001 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:38] PROBLEM - Host ganeti1031 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:38] PROBLEM - Host db1234 #page is DOWN: PING CRITICAL - Packet loss = 100% [11:48:39] PROBLEM - Host an-worker1233 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:40] PROBLEM - Host ms-fe1009 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:40] PROBLEM - Host ganeti1054 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:41] PROBLEM - Host logging-hd1001 is DOWN: PING CRITICAL - Packet loss = 100% [11:48:41] PROBLEM - Host mx-out1001 is DOWN: PING CRITICAL - Packet loss = 100% [11:49:02] oh no, icinga-wm got kicked /o\ [11:49:32] FIRING: [3x] ProbeDown: Service doc2003.codfw.wmnet:443 has failed probes (http_doc2003_codfw_wmnet_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:49:37] ack [11:49:49] !ack [11:49:50] 8679 (ACKED) Host es1041 (paged) [11:49:50] 8681 (ACKED) Host es1055 (paged) [11:49:51] 8683 (ACKED) Host db1210 (paged) [11:49:51] 8684 (ACKED) Host db1234 (paged) [11:49:51] 8685 (ACKED) Host pc1015 (paged) [11:49:51] 8686 (ACKED) Host es1054 (paged) [11:49:56] FIRING: WidespreadPuppetFailure: Puppet has failed in magru - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [11:50:08] I am in the middle of deploying and just deployed codfw, but haven't gotten to eqiad. wat do. [11:50:15] Mvolz: hold for a minute please [11:50:16] FIRING: ProbeDown: Service gerrit1003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://wikitech.wikimedia.org/wiki/TLS/Runbook#gerrit1003:29418 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:50:26] sorry folks those host down alerts are me - it should be fixed now I expect to see recoveries [11:50:29] FIRING: [10x] ProbeDown: Service doc1004.eqiad.wmnet:443 has failed probes (http_doc1004_eqiad_wmnet_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:50:41] (03PS1) 10Vgutierrez: Revert "hieradata: Depool lvs5005 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1359796 [11:50:43] FIRING: [2x] ProbeDown: Service gerrit1003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:50:55] !ack [11:50:56] 8687 (ACKED) Host db1268 (paged) [11:50:56] 8688 (ACKED) Host db1266 (paged) [11:50:56] 8689 (ACKED) Host db1272 (paged) [11:50:56] 8691 (ACKED) Host es1042 (paged) [11:50:56] (03PS2) 10Vgutierrez: Revert "hieradata: Depool lvs5005 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1359796 (https://phabricator.wikimedia.org/T401832) [11:50:56] 8692 (ACKED) Host pc1021 (paged) [11:50:57] 8693 (ACKED) PHPFPMTooBusy sre (mw-api-ext main codfw) [11:50:57] 8694 (ACKED) Host db1267 (paged) [11:50:58] 8695 (ACKED) Host db1235 (paged) [11:50:58] 8696 (ACKED) Host db1157 (paged) [11:50:59] 8697 (ACKED) Host db1260 (paged) [11:50:59] 8698 (ACKED) Host es1049 (paged) [11:51:00] 8699 (ACKED) Host db1273 (paged) [11:51:00] 8700 (ACKED) Host db1257 (paged) [11:51:01] 8701 (ACKED) Host db1255 (paged) [11:51:01] 8702 (ACKED) Host db1237 (paged) [11:51:02] FIRING: [2x] ProbeDown: Service rpki1001:3323 has failed probes (tcp_rpkivalidator_rtr_ip4) - https://wikitech.wikimedia.org/wiki/RPKI#RPKI_to_router_port - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:51:02] 8703 (ACKED) Host db1250 (paged) [11:51:02] 8704 (ACKED) Host db1158 (paged) [11:51:14] FIRING: [9x] ProbeDown: Service commons.wikimedia.org:443 has failed probes (http_commons_wikimedia_org_ip4) #page - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:51:22] who knows how to bring icinga-wm back? it got kicked for Excess Flood [11:51:29] Lucas_WMDE: that's not urgent for now [11:51:33] ok [11:51:40] (I thought we were out of the woods otherwise, sorry) [11:52:09] !log mwscript-k8s --follow --dblist=ores -- extensions/ORES/maintenance/PurgeScoreCache.php --model damaging --old (T440358) [11:52:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:52:47] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1359796 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [11:53:16] oh I just saw. Anything I can help with? [11:53:51] FIRING: [13x] ProbeDown: Service commons.wikimedia.org:443 has failed probes (http_commons_wikimedia_org_ip4) #page - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [11:54:31] (03CR) 10Daniel Kertesz: [C:03+1] Revert "hieradata: Depool lvs5005 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1359796 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [11:54:46] !resolve [11:54:47] 8667 (RESOLVED) [2x] ProbeDown sre (ip4 restbase-https:7443 probes/service http_restbase-https_ip4) [11:54:47] 8668 (RESOLVED) ATSBackendErrorsHigh cache_text sre (restbase.discovery.wmnet drmrs) [11:54:47] 8681 (RESOLVED) Host es1055 (paged) [11:54:47] 8685 (RESOLVED) Host pc1015 (paged) [11:54:47] 8686 (RESOLVED) Host es1054 (paged) [11:54:48] 8687 (RESOLVED) Host db1268 (paged) [11:54:48] 8688 (RESOLVED) Host db1266 (paged) [11:54:48] 8689 (RESOLVED) Host db1272 (paged) [11:54:49] 8691 (RESOLVED) Host es1042 (paged) [11:54:49] 8692 (RESOLVED) Host pc1021 (paged) [11:54:50] 8693 (RESOLVED) PHPFPMTooBusy sre (mw-api-ext main codfw) [11:54:50] 8694 (RESOLVED) Host db1267 (paged) [11:54:51] 8695 (RESOLVED) Host db1235 (paged) [11:54:51] 8696 (RESOLVED) Host db1157 (paged) [11:54:52] 8697 (RESOLVED) Host db1260 (paged) [11:54:52] 8698 (RESOLVED) Host es1049 (paged) [11:54:53] 8699 (RESOLVED) Host db1273 (paged) [11:54:53] 8700 (RESOLVED) Host db1257 (paged) [11:54:54] 8701 (RESOLVED) Host db1255 (paged) [11:54:54] 8702 (RESOLVED) Host db1237 (paged) [11:54:55] 8703 (RESOLVED) Host db1250 (paged) [11:54:55] 8704 (RESOLVED) Host db1158 (paged) [11:54:56] 8705 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [11:54:56] 8706 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [11:56:55] !log hnowlan@cumin1004 START - Cookbook sre.misc-clusters.roll-restart-restbase rolling restart_daemons on A:restbase [11:57:47] (03CR) 10Vgutierrez: [C:03+2] Revert "hieradata: Depool lvs5005 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1359796 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [11:59:01] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [11:59:07] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-int releases routed via main at eqiad: 5.469% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-int&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:59:13] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at codfw: 0.9877% idle #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:00:05] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:00:13] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:00:34] FIRING: VarnishUnavailable: varnish-text has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/Varnish#Diagnosing_Varnish_alerts - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DVarnishUnavailable [12:00:37] !log vgutierrez@cumin2003 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5005.*} and A:liberica [12:00:38] FIRING: HaproxyUnavailable: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [12:00:44] FIRING: [4x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [12:00:50] RECOVERY - haproxy failover on dbproxy1028 is OK: OK check_failover servers up 2 down 0: https://wikitech.wikimedia.org/wiki/HAProxy [12:01:00] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5005.*} and A:liberica [12:01:02] RECOVERY - haproxy failover on dbproxy1022 is OK: OK check_failover servers up 2 down 0: https://wikitech.wikimedia.org/wiki/HAProxy [12:01:06] who is deploying shellbox as root? [12:01:08] !ack [12:01:09] 8707 (ACKED) [2x] ProbeDown sre () [12:01:17] FIRING: SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:01:18] PROBLEM - PyBal IPVS diff check on lvs2013 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [12:01:24] PROBLEM - PyBal IPVS diff check on lvs2014 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [12:01:34] FIRING: [6x] ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from restbase.discovery.wmnet in codfw #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [12:01:38] FIRING: [296x] ProbeDown: Service alert1002:443 has failed probes (http_icinga_wikimedia_org_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:01:41] !ack [12:01:41] All incidents are already acked. [12:01:51] !incidents [12:01:52] 8707 (ACKED) [2x] ProbeDown sre () [12:01:52] 8706 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [12:01:52] 8705 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [12:01:53] 8704 (RESOLVED) Host db1158 (paged) [12:01:53] 8703 (RESOLVED) Host db1250 (paged) [12:01:53] 8702 (RESOLVED) Host db1237 (paged) [12:01:53] 8701 (RESOLVED) Host db1255 (paged) [12:01:54] 8700 (RESOLVED) Host db1257 (paged) [12:01:54] 8699 (RESOLVED) Host db1273 (paged) [12:01:55] 8698 (RESOLVED) Host es1049 (paged) [12:01:55] 8697 (RESOLVED) Host db1260 (paged) [12:01:56] 8696 (RESOLVED) Host db1157 (paged) [12:01:56] 8695 (RESOLVED) Host db1235 (paged) [12:01:57] 8694 (RESOLVED) Host db1267 (paged) [12:01:57] 8693 (RESOLVED) PHPFPMTooBusy sre (mw-api-ext main codfw) [12:01:58] 8692 (RESOLVED) Host pc1021 (paged) [12:01:58] 8691 (RESOLVED) Host es1042 (paged) [12:01:59] 8689 (RESOLVED) Host db1272 (paged) [12:01:59] 8688 (RESOLVED) Host db1266 (paged) [12:02:00] 8687 (RESOLVED) Host db1268 (paged) [12:02:00] 8686 (RESOLVED) Host es1054 (paged) [12:02:39] RESOLVED: [14x] ProbeDown: Service doc1004.eqiad.wmnet:443 has failed probes (http_doc1004_eqiad_wmnet_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:02:56] RESOLVED: [2x] ProbeDown: Service gerrit1003:29418 has failed probes (tcp_gerrit_ssh_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:03:13] RESOLVED: [2x] ProbeDown: Service rpki1001:3323 has failed probes (tcp_rpkivalidator_rtr_ip4) - https://wikitech.wikimedia.org/wiki/RPKI#RPKI_to_router_port - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:03:25] FIRING: [13x] ProbeDown: Service commons.wikimedia.org:443 has failed probes (http_commons_wikimedia_org_ip4) #page - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:04:05] !ack [12:04:06] All incidents are already acked. [12:07:14] PROBLEM - PyBal IPVS diff check on lvs1019 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [12:10:52] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [12:10:56] RESOLVED: [2x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at codfw: 0.9877% idle - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:11:04] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at codfw: 0.9877% idle #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-ext&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [12:11:10] PROBLEM - PyBal IPVS diff check on lvs1020 is CRITICAL: (CRITICAL: Mismatch between IPVS and PyBal https://wikitech.wikimedia.org/wiki/PyBal [12:11:34] RESOLVED: VarnishUnavailable: varnish-text has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/Varnish#Diagnosing_Varnish_alerts - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DVarnishUnavailable [12:11:38] RESOLVED: HaproxyUnavailable: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [12:11:44] RESOLVED: [4x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [12:11:52] FIRING: [3x] KubernetesRsyslogDown: rsyslog on wikikube-worker1146:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [12:12:01] FIRING: [3x] SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:12:22] FIRING: [2x] JobUnavailable: Reduced availability for job bacula in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:12:50] !log hnowlan@cumin1004 END (PASS) - Cookbook sre.misc-clusters.roll-restart-restbase (exit_code=0) rolling restart_daemons on A:restbase [12:13:09] (03PS1) 10Vgutierrez: hieradata: Depool lvs5004 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1359801 (https://phabricator.wikimedia.org/T401832) [12:13:43] RESOLVED: [6x] ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from restbase.discovery.wmnet in codfw #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [12:13:52] FIRING: [2x] JobUnavailable: Reduced availability for job bacula in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:13:56] (03PS1) 10Cathal Mooney: cr1-eqiad: move row a sub-interfaces from ae1 to et-0/1/0 [homer/public] - 10https://gerrit.wikimedia.org/r/1359802 (https://phabricator.wikimedia.org/T441014) [12:14:02] FIRING: [5x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [12:14:05] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1359801 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [12:14:11] RESOLVED: [3x] KubernetesRsyslogDown: rsyslog on wikikube-worker1146:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [12:14:20] FIRING: [3x] SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:14:36] FIRING: [6x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [12:15:14] FIRING: [7x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [12:15:51] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [12:17:01] (03CR) 10Fabfur: [C:03+1] hieradata: Depool lvs5004 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1359801 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [12:17:16] (03CR) 10Vgutierrez: [C:03+2] hieradata: Depool lvs5004 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1359801 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [12:17:25] (03CR) 10Daniel Kertesz: [C:03+1] hieradata: Depool lvs5004 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1359801 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [12:17:36] jouncebot: nowandnext [12:17:36] No deployments scheduled for the next 0 hour(s) and 42 minute(s) [12:17:36] In 0 hour(s) and 42 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T1300) [12:18:11] Is the incident preventing backports? [12:18:33] *incident assumed based on the pages [12:18:51] FIRING: [7x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [12:18:51] please hold [12:19:00] (03PS1) 10Dreamy Jazz: Add update maintenance script to drop watchlist rows for temp users [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359803 (https://phabricator.wikimedia.org/T439059) [12:19:06] Sure [12:20:25] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns for moved IPs on cr1-eqiad - cmooney@cumin1004" [12:20:44] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update reverse dns for moved IPs on cr1-eqiad - cmooney@cumin1004" [12:20:45] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [12:20:48] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, October 28 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359803 (https://phabricator.wikimedia.org/T439059) (owner: 10Dreamy Jazz) [12:21:52] !log trunk all row A vlans from asw2-a-eqiad ae1 to ssw1-a1-eqiad lag1 T441014 [12:21:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:21:59] T441014: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014 [12:22:25] RESOLVED: [2x] SystemdUnitFailed: netbox_report_librenms_run.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:22:26] !log hnowlan@cumin1004 START - Cookbook sre.dns.wipe-cache statsd.eqiad.wmnet on all recursors [12:22:30] !log hnowlan@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) statsd.eqiad.wmnet on all recursors [12:23:05] !log vgutierrez@cumin2003 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5004.*} and A:liberica [12:23:10] I still need to finish an interrupted deploy as well fyi Dreamy_Jazz [12:23:29] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5004.*} and A:liberica [12:23:51] FIRING: [8x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [12:24:15] I've added mine to the backport window [12:24:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in magru - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [12:24:57] FIRING: [2x] ProbeDown: Service restbase-https:7443 has failed probes (http_restbase-https_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#restbase-https:7443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:25:50] !log re-enabling et-0/1/0 sub-interfaces for row A vlans on cr1-eqiad (they are now trunked on asw2-a-eqiad side to ssw1-a1-eqiad so no repeat of last time expected) T441014 [12:25:51] Just a reminder that right after the backport window, the switchover starts (we haven't decided to call it off yet) so please don't overload [12:25:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:26:10] RECOVERY - PyBal IPVS diff check on lvs1020 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [12:26:13] !log vgutierrez@cumin2003 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5005.*} and A:liberica [12:26:36] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5005.*} and A:liberica [12:26:40] RECOVERY - PyBal IPVS diff check on lvs1019 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [12:27:09] Mine can wait till later if we run out of time [12:28:07] FIRING: [61x] ProbeDown: Service restbase1031:7231 has failed probes (http_restbase_discovery_wmnet_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:28:17] Finishing mine can't wait unfortunately. [12:28:51] FIRING: [8x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [12:28:51] Mvolz: it was a citoid deploy right? [12:28:54] you can proceed [12:28:59] !log hnowlan@cumin1004 START - Cookbook sre.misc-clusters.roll-restart-restbase rolling restart_daemons on A:restbase-codfw [12:29:01] awesome, thanks. [12:29:06] !log mvolz@deploy1003 scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid [12:29:37] (03CR) 10Jforrester: "I imagine that this is not your normal wheelhouse, but any chance this could be landed? It will help with the under-loved kartotherian's m" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1358263 (owner: 10Jforrester) [12:29:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12484007 (10phaultfinder) [12:29:49] (03CR) 10Cathal Mooney: [C:03+2] cr1-eqiad: move row a sub-interfaces from ae1 to et-0/1/0 [homer/public] - 10https://gerrit.wikimedia.org/r/1359802 (https://phabricator.wikimedia.org/T441014) (owner: 10Cathal Mooney) [12:30:32] (03CR) 10Ladsgroup: [C:03+2] experiment: Use wmf thumbor fork with async engine support [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1359787 (https://phabricator.wikimedia.org/T442031) (owner: 10Jgiannelos) [12:30:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [12:30:53] (03Merged) 10jenkins-bot: cr1-eqiad: move row a sub-interfaces from ae1 to et-0/1/0 [homer/public] - 10https://gerrit.wikimedia.org/r/1359802 (https://phabricator.wikimedia.org/T441014) (owner: 10Cathal Mooney) [12:31:00] !log mvolz@deploy1003 Finished scap deploy-service citoid: update (duration: 50m 46s) [12:31:21] all done, ttvm! [12:31:22] RECOVERY - PyBal IPVS diff check on lvs2014 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [12:31:58] !log revert VRRP changes on cr2-eqiad so cr1-eqiad takes over for some of eqiad row A subnets T441014 [12:32:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:32:05] T441014: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014 [12:32:50] !log hnowlan@cumin1004 START - Cookbook sre.misc-clusters.roll-restart-restbase rolling restart_daemons on A:restbase-eqiad [12:33:07] FIRING: [60x] ProbeDown: Service restbase1031:7231 has failed probes (http_restbase_discovery_wmnet_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:33:51] RESOLVED: [6x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [12:34:26] (03PS1) 10Fabian Kaelin: admin: Add FIDO SSH key for fab [puppet] - 10https://gerrit.wikimedia.org/r/1359804 (https://phabricator.wikimedia.org/T442665) [12:34:54] !log make cr1-eqiad take over VRRP for the remaining row A vlans [12:34:57] RESOLVED: [2x] ProbeDown: Service restbase-https:7443 has failed probes (http_restbase-https_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#restbase-https:7443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:34:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:35:01] (03CR) 10Ladsgroup: "Don't forget to bump to the most recent image of thumbor to pick Idad5456e53d54" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359771 (https://phabricator.wikimedia.org/T442031) (owner: 10Jgiannelos) [12:35:04] (03CR) 10Ladsgroup: [C:03+1] thumbor: Make async engine loading configurable [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359771 (https://phabricator.wikimedia.org/T442031) (owner: 10Jgiannelos) [12:35:18] (03CR) 10CI reject: [V:04-1] admin: Add FIDO SSH key for fab [puppet] - 10https://gerrit.wikimedia.org/r/1359804 (https://phabricator.wikimedia.org/T442665) (owner: 10Fabian Kaelin) [12:35:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [12:36:12] !log hnowlan@cumin1004 END (PASS) - Cookbook sre.misc-clusters.roll-restart-restbase (exit_code=0) rolling restart_daemons on A:restbase-codfw [12:36:16] (03PS1) 10KartikMistry: Update cxserver to 2026-10-28-122929-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359806 [12:36:18] RECOVERY - PyBal IPVS diff check on lvs2013 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal [12:38:07] FIRING: [62x] ProbeDown: Service restbase1031:7231 has failed probes (http_restbase_discovery_wmnet_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:38:32] (03CR) 10Kevin Bazira: [C:03+1] admin_ng: Add llm-batch namespace to LiftWing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359785 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [12:38:32] (03Merged) 10jenkins-bot: experiment: Use wmf thumbor fork with async engine support [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1359787 (https://phabricator.wikimedia.org/T442031) (owner: 10Jgiannelos) [12:39:08] (03PS1) 10Btullis: cache: Remove datasets-config from the CDN [puppet] - 10https://gerrit.wikimedia.org/r/1359808 (https://phabricator.wikimedia.org/T442701) [12:39:11] (03PS1) 10Btullis: deployment_server: Absent the datasets-config kubeconfigs [puppet] - 10https://gerrit.wikimedia.org/r/1359809 (https://phabricator.wikimedia.org/T442701) [12:39:14] (03PS1) 10Btullis: deployment_server: Remove the datasets-config kubeconfigs [puppet] - 10https://gerrit.wikimedia.org/r/1359810 (https://phabricator.wikimedia.org/T442701) [12:39:14] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [12:39:58] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [12:40:00] !log hnowlan@cumin1004 END (PASS) - Cookbook sre.misc-clusters.roll-restart-restbase (exit_code=0) rolling restart_daemons on A:restbase-eqiad [12:40:11] (03PS1) 10Btullis: Remove the datasets-config DNS records [dns] - 10https://gerrit.wikimedia.org/r/1359814 (https://phabricator.wikimedia.org/T442701) [12:40:37] (03CR) 10Kevin Bazira: [C:03+1] profile::k8s::deployment_server: add config for llm-batch [puppet] - 10https://gerrit.wikimedia.org/r/1359777 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [12:40:58] We're clear for the backport window I think cc hnowlan [12:41:04] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [12:41:20] (03PS1) 10Btullis: dse-k8s: Remove the datasets-config helmfiles and namespaces [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359815 (https://phabricator.wikimedia.org/T442701) [12:41:21] go ahead [12:41:22] (03PS1) 10Btullis: Remove the datasets-config chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359816 (https://phabricator.wikimedia.org/T442701) [12:41:29] (03CR) 10Dpogorzelski: [C:03+1] aptrepo: add ROCm 7.2.3 packages to wikimedia trixie mirror [puppet] - 10https://gerrit.wikimedia.org/r/1359752 (https://phabricator.wikimedia.org/T442138) (owner: 10Bartosz Wójtowicz) [12:41:44] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12484047 (10cmooney) [12:41:50] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [12:41:55] Dreamy_Jazz: if you want to do your backport now you can [12:42:02] Thanks [12:42:05] I will do so [12:42:13] that way only matthias' is left in the window and we'll be clear for the switchover [12:42:47] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359803 (https://phabricator.wikimedia.org/T439059) (owner: 10Dreamy Jazz) [12:43:07] RESOLVED: [61x] ProbeDown: Service restbase1031:7231 has failed probes (http_restbase_discovery_wmnet_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:45:02] (03CR) 10KartikMistry: [C:03+2] Update cxserver to 2026-10-28-122929-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359806 (owner: 10KartikMistry) [12:45:42] (03CR) 10Dpogorzelski: [C:03+2] aptrepo: add ROCm 7.2.3 packages to wikimedia trixie mirror [puppet] - 10https://gerrit.wikimedia.org/r/1359752 (https://phabricator.wikimedia.org/T442138) (owner: 10Bartosz Wójtowicz) [12:46:36] (03Merged) 10jenkins-bot: Update cxserver to 2026-10-28-122929-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359806 (owner: 10KartikMistry) [12:47:14] (03PS1) 10Jelto: Revert "etherpad: stop services" [puppet] - 10https://gerrit.wikimedia.org/r/1359817 (https://phabricator.wikimedia.org/T442232) [12:47:33] (03Merged) 10jenkins-bot: Add update maintenance script to drop watchlist rows for temp users [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359803 (https://phabricator.wikimedia.org/T439059) (owner: 10Dreamy Jazz) [12:47:59] !log dreamyjazz@deploy1003 Started scap sync-world: Backport for [[gerrit:1359803|Add update maintenance script to drop watchlist rows for temp users (T439059)]] (1.48.0-wmf.1) [12:48:07] T439059: Skip default watchlist additions during temp account creation - https://phabricator.wikimedia.org/T439059 [12:49:02] (03PS4) 10Slyngshede: CAS 8.0.3 [software/cas-overlay-template] - 10https://gerrit.wikimedia.org/r/1346564 (https://phabricator.wikimedia.org/T439509) [12:49:10] (03CR) 10Brouberol: [C:03+1] cache: Remove datasets-config from the CDN [puppet] - 10https://gerrit.wikimedia.org/r/1359808 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [12:49:19] (03CR) 10Brouberol: [C:03+1] Remove the datasets-config DNS records [dns] - 10https://gerrit.wikimedia.org/r/1359814 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [12:49:32] (03CR) 10Brouberol: [C:03+1] dse-k8s: Remove the datasets-config helmfiles and namespaces [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359815 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [12:49:40] (03CR) 10Brouberol: [C:03+1] Remove the datasets-config chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359816 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [12:49:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12484089 (10phaultfinder) [12:50:47] (03CR) 10Klausman: [V:03+2 C:03+2] admin_ng: Add llm-batch namespace to LiftWing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359785 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [12:51:14] !log kartik@deploy1003 helmfile [staging] START helmfile.d/services/cxserver: apply [12:51:25] !log disable ae1 on cr2-eqiad connecting to asw2-a-eqiad (traffic is flowing via cr1-eqiad) T441014 [12:51:32] !log kartik@deploy1003 helmfile [staging] DONE helmfile.d/services/cxserver: apply [12:51:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:51:33] T441014: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014 [12:52:11] !log dreamyjazz@deploy1003 dreamyjazz: Backport for [[gerrit:1359803|Add update maintenance script to drop watchlist rows for temp users (T439059)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:52:21] (03PS1) 10Hnowlan: service::node: remove statsd configuration [puppet] - 10https://gerrit.wikimedia.org/r/1359819 (https://phabricator.wikimedia.org/T435340) [12:52:56] (03CR) 10Joal: [C:03+1] cache: Remove datasets-config from the CDN [puppet] - 10https://gerrit.wikimedia.org/r/1359808 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [12:53:42] (03PS1) 10Jgiannelos: thumbor: Pin staging image to async experiment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359820 (https://phabricator.wikimedia.org/T442031) [12:53:46] (03CR) 10Joal: [C:03+1] Remove the datasets-config DNS records [dns] - 10https://gerrit.wikimedia.org/r/1359814 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [12:53:49] !log kartik@deploy1003 helmfile [codfw] START helmfile.d/services/cxserver: apply [12:54:04] (03CR) 10Joal: [C:03+1] dse-k8s: Remove the datasets-config helmfiles and namespaces [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359815 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [12:54:16] !log kartik@deploy1003 helmfile [codfw] DONE helmfile.d/services/cxserver: apply [12:54:24] PROBLEM - Host ps1-b2-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [12:54:24] PROBLEM - Host ps1-a1-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [12:54:24] PROBLEM - Host ps1-d6-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [12:54:24] PROBLEM - Host ps1-a6-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [12:54:24] PROBLEM - Host ps1-f1-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [12:54:25] PROBLEM - Host ps1-c3-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [12:54:25] PROBLEM - Host ps1-e10-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [12:54:26] PROBLEM - Host ps1-e9-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [12:54:26] PROBLEM - Host ps1-e5-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [12:54:27] PROBLEM - Host ps1-d4-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [12:54:27] PROBLEM - Host ps1-e11-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [12:54:28] PROBLEM - Host ps1-c5-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [12:54:28] PROBLEM - Host ps1-d7-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [12:55:20] RECOVERY - Host ps1-e8-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.90 ms [12:55:20] RECOVERY - Host ps1-f6-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.55 ms [12:55:20] RECOVERY - Host ps1-f2-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.41 ms [12:55:20] RECOVERY - Host ps1-f3-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.56 ms [12:55:20] RECOVERY - Host ps1-f7-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.40 ms [12:55:21] RECOVERY - Host ps1-f4-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.18 ms [12:55:21] RECOVERY - Host ps1-a4-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.25 ms [12:55:22] RECOVERY - Host ps1-a3-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.12 ms [12:55:22] RECOVERY - Host ps1-c4-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.22 ms [12:55:34] (03CR) 10Brouberol: [C:03+1] profile::k8s::deployment_server: add config for llm-batch [puppet] - 10https://gerrit.wikimedia.org/r/1359777 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [12:55:49] (03CR) 10Brouberol: [C:03+2] profile::k8s::deployment_server: add config for llm-batch [puppet] - 10https://gerrit.wikimedia.org/r/1359777 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [12:55:51] ^^ ugh... management router not playing ball for some reason [12:56:15] !log dreamyjazz@deploy1003 dreamyjazz: Continuing with deployment [12:56:20] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:56:37] (03CR) 10Jgiannelos: [C:03+2] thumbor: Make async engine loading configurable [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359771 (https://phabricator.wikimedia.org/T442031) (owner: 10Jgiannelos) [12:56:41] (03PS1) 10A-pizzata: EventStreamConfig: Add time to persist config for page_change [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359824 (https://phabricator.wikimedia.org/T440968) [12:56:47] (03CR) 10Jgiannelos: [C:03+2] thumbor: Pin staging image to async experiment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359820 (https://phabricator.wikimedia.org/T442031) (owner: 10Jgiannelos) [12:57:23] !log kartik@deploy1003 helmfile [eqiad] START helmfile.d/services/cxserver: apply [12:57:48] !log kartik@deploy1003 helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [12:58:08] !log Updated cxserver to 2026-10-28-122929-production [12:58:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:58:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [12:58:49] (03Merged) 10jenkins-bot: admin_ng: Add llm-batch namespace to LiftWing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359785 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [13:00:04] urbanecm and TheresNoTime: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for UTC afternoon backport window . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T1300). [13:00:05] matthiasmullie and Dreamy_Jazz: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:18] \o [13:00:22] Scap is still running for mine [13:01:58] !log dreamyjazz@deploy1003 Finished scap sync-world: Backport for [[gerrit:1359803|Add update maintenance script to drop watchlist rows for temp users (T439059)]] (1.48.0-wmf.1) (duration: 13m 59s) [13:02:03] Mine is done [13:02:06] T439059: Skip default watchlist additions during temp account creation - https://phabricator.wikimedia.org/T439059 [13:02:20] Is matthiasmullie here for the window? [13:02:42] o/ [13:02:43] (03Merged) 10jenkins-bot: thumbor: Make async engine loading configurable [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359771 (https://phabricator.wikimedia.org/T442031) (owner: 10Jgiannelos) [13:02:55] Dreamy_Jazz: yes, hre [13:02:57] here* [13:03:22] (03Merged) 10jenkins-bot: thumbor: Pin staging image to async experiment [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359820 (https://phabricator.wikimedia.org/T442031) (owner: 10Jgiannelos) [13:03:40] Dreamy_Jazz: you just finished yours, correct? [13:03:46] Yeah I have [13:03:49] Ah, nvm - just saw your previous note :D [13:03:51] (03Abandoned) 10Ladsgroup: [WIP] Stop blocking the loop on the failure-counter memcached write [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337313 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup-claude) [13:04:04] Aight, I'm beginning [13:04:09] (03Abandoned) 10Ladsgroup: [WIP] Collect timing headers on the context instead of the request handler [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337315 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup-claude) [13:04:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/ReaderExperiments] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359789 (owner: 10Matthias Mullie) [13:04:12] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/ReaderExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359790 (owner: 10Matthias Mullie) [13:04:20] (03Abandoned) 10Ladsgroup: [WIP] Run engine.load() on the engine thread pool too [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1337317 (https://phabricator.wikimedia.org/T431767) (owner: 10Ladsgroup-claude) [13:04:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12484195 (10phaultfinder) [13:04:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12484196 (10phaultfinder) [13:05:32] (03Merged) 10jenkins-bot: Squashed diff to master (minus i18n) [extensions/ReaderExperiments] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359789 (owner: 10Matthias Mullie) [13:06:26] (03Merged) 10jenkins-bot: Squashed diff to master (minus i18n) [extensions/ReaderExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359790 (owner: 10Matthias Mullie) [13:06:58] !log mlitn@deploy1003 Started scap sync-world: Backport for Squashed diff to master (minus i18n) ([[gerrit:1359789|1.47.0-wmf.24]], [[gerrit:1359790|1.48.0-wmf.1]]) [13:07:09] !log klausman@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. [13:08:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:08:54] !log klausman@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. [13:09:08] (03CR) 10Btullis: [C:03+2] cache: Remove datasets-config from the CDN [puppet] - 10https://gerrit.wikimedia.org/r/1359808 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [13:09:15] !log Running `foreachwikiindblist group0.dblist cleanupWatchlistTemporaryAccountRows.php --wiki=testwiki` for T439059 [13:09:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:09:22] T439059: Skip default watchlist additions during temp account creation - https://phabricator.wikimedia.org/T439059 [13:09:29] !log Running `foreachwikiindblist group0.dblist cleanupWatchlistTemporaryAccountRows.php` for T439059 [13:09:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:11:29] !log mlitn@deploy1003 mlitn: Backport for Squashed diff to master (minus i18n) ([[gerrit:1359789|1.47.0-wmf.24]], [[gerrit:1359790|1.48.0-wmf.1]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:12:25] !log mlitn@deploy1003 mlitn: Continuing with deployment [13:12:39] 10ops-eqiad, 06SRE, 06DC-Ops, 06Wikidata Platform Team, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Q4:rack/setup/install dse-k8s-wdqs100[1-3] (formerly wdqs103[6-8]) - https://phabricator.wikimedia.org/T423314#12484221 (10bking) 05Open→03Resolved I've corrected the records above and run the `... [13:12:50] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker2337.codfw.wmnet [13:14:32] (03PS1) 10Bartosz Wójtowicz: aptrepo: mirror only the ROCm 7.2.3 packages the vLLM image uses [puppet] - 10https://gerrit.wikimedia.org/r/1359825 (https://phabricator.wikimedia.org/T442138) [13:15:30] !log blake@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host wikikube-worker2337.codfw.wmnet [13:16:29] !log cwilliams@cumin1004 START - Cookbook sre.backup.offsite [13:16:29] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.backup.offsite (exit_code=0) [13:17:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:17:05] (03PS1) 10Ottomata: deployment_server::deploy_service::service_catalog: Add eventgate [puppet] - 10https://gerrit.wikimedia.org/r/1359827 (https://phabricator.wikimedia.org/T442177) [13:17:07] (03PS1) 10Ottomata: deployment_server::deploy_service::service_catalog: Add eventgate-all [puppet] - 10https://gerrit.wikimedia.org/r/1359828 (https://phabricator.wikimedia.org/T442177) [13:17:46] (03CR) 10Dpogorzelski: [C:03+2] aptrepo: mirror only the ROCm 7.2.3 packages the vLLM image uses [puppet] - 10https://gerrit.wikimedia.org/r/1359825 (https://phabricator.wikimedia.org/T442138) (owner: 10Bartosz Wójtowicz) [13:17:54] (03CR) 10CI reject: [V:04-1] deployment_server::deploy_service::service_catalog: Add eventgate [puppet] - 10https://gerrit.wikimedia.org/r/1359827 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [13:18:03] !log mlitn@deploy1003 Finished scap sync-world: Backport for Squashed diff to master (minus i18n) ([[gerrit:1359789|1.47.0-wmf.24]], [[gerrit:1359790|1.48.0-wmf.1]]) (duration: 11m 05s) [13:18:16] (03CR) 10CI reject: [V:04-1] deployment_server::deploy_service::service_catalog: Add eventgate-all [puppet] - 10https://gerrit.wikimedia.org/r/1359828 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [13:18:33] Done; I don't think there's any other deploys in this window? [13:18:50] Not that I saw [13:18:54] (03CR) 10Jelto: [V:03+1] "PCC SUCCESS (CORE_DIFF 1 NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/" [puppet] - 10https://gerrit.wikimedia.org/r/1359817 (https://phabricator.wikimedia.org/T442232) (owner: 10Jelto) [13:19:24] Probably can end the backport window here [13:19:40] !log Afternoon UTC backport window done [13:19:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:20:19] claime: Backport window is done if you wanted to take scap lock etc. [13:20:41] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [13:20:51] !log klausman@deploy1003 helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. [13:21:05] !log cwilliams@cumin1004 START - Cookbook sre.backup.offsite [13:21:05] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.backup.offsite (exit_code=0) [13:21:13] !log cwilliams@cumin1004 START - Cookbook sre.backup.offsite [13:21:13] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.backup.offsite (exit_code=0) [13:21:33] Dreamy_Jazz: thanks, slyngs you can lock whenever you want. [13:21:49] (03CR) 10Jelto: [V:03+1 C:03+2] Revert "etherpad: stop services" [puppet] - 10https://gerrit.wikimedia.org/r/1359817 (https://phabricator.wikimedia.org/T442232) (owner: 10Jelto) [13:22:36] !log klausman@deploy1003 helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. [13:22:46] !log slyngshede@cumin1004 START - Cookbook sre.switchdc.mediawiki.00-lock-scap for datacenter switchover from eqiad to codfw [13:22:48] !log root@deploy1003 Locking from deployment [ALL REPOSITORIES]: Datacenter switchover from eqiad to codfw - [13:22:48] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.switchdc.mediawiki.00-lock-scap (exit_code=0) for datacenter switchover from eqiad to codfw [13:22:57] claime: Done [13:24:35] !log move asw2-a-eqiad ae2 ports into ae1 and enable the far-end on ssw1-a8-eqiad where the fibres have been moved T441014 [13:24:36] (03CR) 10Ssingh: [C:04-1] "Please do not use Claude or any AI agent to generate SSH keys as doing so can compromise the integrity of the system and is a violation of" [puppet] - 10https://gerrit.wikimedia.org/r/1359804 (https://phabricator.wikimedia.org/T442665) (owner: 10Fabian Kaelin) [13:24:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:24:43] T441014: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014 [13:25:26] PROBLEM - Host ps1-c2-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:25:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:25:47] 10ops-eqiad, 06SRE, 06DC-Ops, 07Essential-Work: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12484293 (10elukey) We have two outstanding issues: 1) The nvme disks are not recognized by d-i In T438161#12352303 we successfully installed Trixie, but then recent reimag... [13:26:02] PROBLEM - Host ps1-f6-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:12] PROBLEM - Host ps1-a2-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:18] PROBLEM - Host ps1-d6-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:18] PROBLEM - Host ps1-e5-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:18] PROBLEM - Host ps1-e16-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:18] PROBLEM - Host ps1-b2-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:20] PROBLEM - Host ps1-c3-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:20] PROBLEM - Host ps1-c6-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:20] PROBLEM - Host ps1-e10-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:24] PROBLEM - Host ps1-f5-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:24] PROBLEM - Host ps1-a7-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:24] PROBLEM - Host ps1-b6-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:24] PROBLEM - Host ps1-e15-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:24] PROBLEM - Host ps1-e2-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:25] PROBLEM - Host ps1-c7-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:26] PROBLEM - Host ps1-a3-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:27] PROBLEM - Host ps1-d7-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:28] PROBLEM - Host ps1-e6-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:38] PROBLEM - Host ps1-a6-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:38] PROBLEM - Host ps1-d3-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:26:38] PROBLEM - Host ps1-b3-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:27:02] PROBLEM - Host ps1-d2-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:27:02] PROBLEM - Host ps1-e1-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:27:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:27:10] PROBLEM - Host ps1-b7-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:27:12] PROBLEM - Host ps1-f1-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:27:22] PROBLEM - Host ps1-f2-eqiad is DOWN: PING CRITICAL - Packet loss = 100% [13:27:46] PROBLEM - ps1-b4-eqiad-infeed-load-tower-A-phase-Y on ps1-b4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:27:54] PROBLEM - ps1-f3-eqiad-infeed-load-tower-B-phase-Z on ps1-f3-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:27:56] PROBLEM - ps1-e9-eqiad-infeed-load-tower-A-phase-Z on ps1-e9-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:27:56] PROBLEM - ps1-c4-eqiad-infeed-load-tower-A-phase-Z on ps1-c4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:27:56] PROBLEM - ps1-f7-eqiad-infeed-load-tower-B-phase-Z on ps1-f7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:27:56] PROBLEM - ps1-b4-eqiad-infeed-load-tower-A-phase-X on ps1-b4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:27:56] PROBLEM - ps1-f4-eqiad-infeed-load-tower-A-phase-Z on ps1-f4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:27:57] PROBLEM - ps1-d8-eqiad-infeed-load-tower-B-phase-X on ps1-d8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:27:58] RECOVERY - ps1-f3-eqiad-infeed-load-tower-B-phase-Z on ps1-f3-eqiad is OK: SNMP OK - ps1-f3-eqiad-infeed-load-tower-B-phase-Z 618 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:00] RECOVERY - ps1-b4-eqiad-infeed-load-tower-A-phase-X on ps1-b4-eqiad is OK: SNMP OK - ps1-b4-eqiad-infeed-load-tower-A-phase-X 484 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:00] RECOVERY - ps1-c4-eqiad-infeed-load-tower-A-phase-Z on ps1-c4-eqiad is OK: SNMP OK - ps1-c4-eqiad-infeed-load-tower-A-phase-Z 512 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:01] RECOVERY - ps1-e9-eqiad-infeed-load-tower-A-phase-Z on ps1-e9-eqiad is OK: SNMP OK - ps1-e9-eqiad-infeed-load-tower-A-phase-Z 204 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:02] PROBLEM - ps1-b5-eqiad-infeed-load-tower-B-phase-X on ps1-b5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:02] PROBLEM - ps1-a5-eqiad-infeed-load-tower-A-phase-Y on ps1-a5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:04] PROBLEM - ps1-e7-eqiad-infeed-load-tower-B-phase-X on ps1-e7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:06] PROBLEM - ps1-e12-eqiad-infeed-load-tower-B-phase-X on ps1-e12-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:06] PROBLEM - ps1-f4-eqiad-infeed-load-tower-B-phase-X on ps1-f4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:07] PROBLEM - ps1-a5-eqiad-infeed-load-tower-B-phase-Y on ps1-a5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:08] RECOVERY - ps1-a5-eqiad-infeed-load-tower-A-phase-Y on ps1-a5-eqiad is OK: SNMP OK - ps1-a5-eqiad-infeed-load-tower-A-phase-Y 459 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:08] RECOVERY - ps1-e7-eqiad-infeed-load-tower-B-phase-X on ps1-e7-eqiad is OK: SNMP OK - ps1-e7-eqiad-infeed-load-tower-B-phase-X 636 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:10] RECOVERY - ps1-e12-eqiad-infeed-load-tower-B-phase-X on ps1-e12-eqiad is OK: SNMP OK - ps1-e12-eqiad-infeed-load-tower-B-phase-X 219 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:12] PROBLEM - ps1-b4-eqiad-infeed-load-tower-B-phase-Z on ps1-b4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:12] PROBLEM - ps1-b1-eqiad-infeed-load-tower-B-phase-Z on ps1-b1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:12] PROBLEM - ps1-b1-eqiad-infeed-load-tower-A-phase-Z on ps1-b1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:12] PROBLEM - ps1-e12-eqiad-infeed-load-tower-A-phase-X on ps1-e12-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:12] PROBLEM - ps1-e4-eqiad-infeed-load-tower-A-phase-X on ps1-e4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:13] RECOVERY - ps1-a5-eqiad-infeed-load-tower-B-phase-Y on ps1-a5-eqiad is OK: SNMP OK - ps1-a5-eqiad-infeed-load-tower-B-phase-Y 415 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:16] RECOVERY - ps1-e4-eqiad-infeed-load-tower-A-phase-X on ps1-e4-eqiad is OK: SNMP OK - ps1-e4-eqiad-infeed-load-tower-A-phase-X 554 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:17] RECOVERY - ps1-b1-eqiad-infeed-load-tower-B-phase-Z on ps1-b1-eqiad is OK: SNMP OK - ps1-b1-eqiad-infeed-load-tower-B-phase-Z 254 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:22] PROBLEM - ps1-f4-eqiad-infeed-load-tower-A-phase-Y on ps1-f4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:27] !log klausman@deploy1003 helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [13:28:28] RECOVERY - ps1-f4-eqiad-infeed-load-tower-A-phase-Y on ps1-f4-eqiad is OK: SNMP OK - ps1-f4-eqiad-infeed-load-tower-A-phase-Y 474 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:46] PROBLEM - ps1-b1-eqiad-infeed-load-tower-A-phase-X on ps1-b1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:50] RECOVERY - ps1-b1-eqiad-infeed-load-tower-A-phase-X on ps1-b1-eqiad is OK: SNMP OK - ps1-b1-eqiad-infeed-load-tower-A-phase-X 368 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:51] RECOVERY - ps1-b4-eqiad-infeed-load-tower-A-phase-Y on ps1-b4-eqiad is OK: SNMP OK - ps1-b4-eqiad-infeed-load-tower-A-phase-Y 469 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:28:56] PROBLEM - ps1-e11-eqiad-infeed-load-tower-A-phase-Z on ps1-e11-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:00] RECOVERY - ps1-f7-eqiad-infeed-load-tower-B-phase-Z on ps1-f7-eqiad is OK: SNMP OK - ps1-f7-eqiad-infeed-load-tower-B-phase-Z 521 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:00] RECOVERY - ps1-e11-eqiad-infeed-load-tower-A-phase-Z on ps1-e11-eqiad is OK: SNMP OK - ps1-e11-eqiad-infeed-load-tower-A-phase-Z 281 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:01] RECOVERY - ps1-f4-eqiad-infeed-load-tower-A-phase-Z on ps1-f4-eqiad is OK: SNMP OK - ps1-f4-eqiad-infeed-load-tower-A-phase-Z 496 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:02] PROBLEM - ps1-b5-eqiad-infeed-load-tower-B-phase-Y on ps1-b5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:02] PROBLEM - ps1-b4-eqiad-infeed-load-tower-B-phase-X on ps1-b4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:02] PROBLEM - ps1-a8-eqiad-infeed-load-tower-A-phase-Z on ps1-a8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:07] RECOVERY - ps1-b5-eqiad-infeed-load-tower-B-phase-Y on ps1-b5-eqiad is OK: SNMP OK - ps1-b5-eqiad-infeed-load-tower-B-phase-Y 483 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:07] PROBLEM - ps1-a8-eqiad-infeed-load-tower-B-phase-Y on ps1-a8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:07] RECOVERY - ps1-b5-eqiad-infeed-load-tower-B-phase-X on ps1-b5-eqiad is OK: SNMP OK - ps1-b5-eqiad-infeed-load-tower-B-phase-X 369 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:12] RECOVERY - ps1-f4-eqiad-infeed-load-tower-B-phase-X on ps1-f4-eqiad is OK: SNMP OK - ps1-f4-eqiad-infeed-load-tower-B-phase-X 571 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:12] RECOVERY - ps1-a8-eqiad-infeed-load-tower-B-phase-Y on ps1-a8-eqiad is OK: SNMP OK - ps1-a8-eqiad-infeed-load-tower-B-phase-Y 491 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:16] RECOVERY - ps1-e12-eqiad-infeed-load-tower-A-phase-X on ps1-e12-eqiad is OK: SNMP OK - ps1-e12-eqiad-infeed-load-tower-A-phase-X 209 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:23] (03PS2) 10Ottomata: deployment_server::deploy_service::service_catalog: Add eventgate [puppet] - 10https://gerrit.wikimedia.org/r/1359827 (https://phabricator.wikimedia.org/T442177) [13:29:23] (03PS2) 10Ottomata: deployment_server::deploy_service::service_catalog: Add eventgate-all [puppet] - 10https://gerrit.wikimedia.org/r/1359828 (https://phabricator.wikimedia.org/T442177) [13:29:44] PROBLEM - ps1-e12-eqiad-infeed-load-tower-A-phase-Y on ps1-e12-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:45] (03CR) 10Bking: [C:03+1] "Copied votes on follow-up patch sets have been updated:" [puppet] - 10https://gerrit.wikimedia.org/r/1359827 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [13:29:48] PROBLEM - ps1-e12-eqiad-infeed-load-tower-B-phase-Y on ps1-e12-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:54] PROBLEM - ps1-b5-eqiad-infeed-load-tower-A-phase-X on ps1-b5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:56] !log klausman@deploy1003 helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [13:29:56] PROBLEM - ps1-b5-eqiad-infeed-load-tower-A-phase-Y on ps1-b5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:29:56] PROBLEM - ps1-c8-eqiad-infeed-load-tower-A-phase-Z on ps1-c8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:00] RECOVERY - ps1-d8-eqiad-infeed-load-tower-B-phase-X on ps1-d8-eqiad is OK: SNMP OK - ps1-d8-eqiad-infeed-load-tower-B-phase-X 313 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:02] PROBLEM - ps1-e9-eqiad-infeed-load-tower-B-phase-X on ps1-e9-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:02] PROBLEM - ps1-d4-eqiad-infeed-load-tower-B-phase-Z on ps1-d4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:02] PROBLEM - ps1-a4-eqiad-infeed-load-tower-B-phase-X on ps1-a4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:06] PROBLEM - ps1-c1-eqiad-infeed-load-tower-B-phase-X on ps1-c1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:07] PROBLEM - ps1-e3-eqiad-infeed-load-tower-A-phase-Y on ps1-e3-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:07] RECOVERY - ps1-d4-eqiad-infeed-load-tower-B-phase-Z on ps1-d4-eqiad is OK: SNMP OK - ps1-d4-eqiad-infeed-load-tower-B-phase-Z 518 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:07] RECOVERY - ps1-e9-eqiad-infeed-load-tower-B-phase-X on ps1-e9-eqiad is OK: SNMP OK - ps1-e9-eqiad-infeed-load-tower-B-phase-X 227 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:12] PROBLEM - ps1-f4-eqiad-infeed-load-tower-A-phase-X on ps1-f4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:19] (03CR) 10Jelto: "this is not properly cleaned up when ensure is changed from `stopped` to `running`" [puppet] - 10https://gerrit.wikimedia.org/r/1358284 (https://phabricator.wikimedia.org/T442232) (owner: 10Dzahn) [13:30:39] (03CR) 10jenkins-bot: deployment_server::deploy_service::service_catalog: Add eventgate [puppet] - 10https://gerrit.wikimedia.org/r/1359827 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [13:30:48] PROBLEM - ps1-f8-eqiad-infeed-load-tower-A-phase-Y on ps1-f8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:50] RECOVERY - ps1-e12-eqiad-infeed-load-tower-A-phase-Y on ps1-e12-eqiad is OK: SNMP OK - ps1-e12-eqiad-infeed-load-tower-A-phase-Y 208 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:52] RECOVERY - ps1-e12-eqiad-infeed-load-tower-B-phase-Y on ps1-e12-eqiad is OK: SNMP OK - ps1-e12-eqiad-infeed-load-tower-B-phase-Y 209 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:52] RECOVERY - ps1-f8-eqiad-infeed-load-tower-A-phase-Y on ps1-f8-eqiad is OK: SNMP OK - ps1-f8-eqiad-infeed-load-tower-A-phase-Y 434 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:53] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:30:54] PROBLEM - ps1-f7-eqiad-infeed-load-tower-A-phase-Z on ps1-f7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:54] PROBLEM - ps1-e12-eqiad-infeed-load-tower-A-phase-Z on ps1-e12-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:56] PROBLEM - ps1-f3-eqiad-infeed-load-tower-A-phase-Z on ps1-f3-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:30:57] (03CR) 10Bking: [C:03+1] deployment_server::deploy_service::service_catalog: Add eventgate-all [puppet] - 10https://gerrit.wikimedia.org/r/1359828 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [13:31:00] RECOVERY - ps1-e12-eqiad-infeed-load-tower-A-phase-Z on ps1-e12-eqiad is OK: SNMP OK - ps1-e12-eqiad-infeed-load-tower-A-phase-Z 209 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:00] RECOVERY - ps1-f3-eqiad-infeed-load-tower-A-phase-Z on ps1-f3-eqiad is OK: SNMP OK - ps1-f3-eqiad-infeed-load-tower-A-phase-Z 585 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:01] RECOVERY - ps1-c8-eqiad-infeed-load-tower-A-phase-Z on ps1-c8-eqiad is OK: SNMP OK - ps1-c8-eqiad-infeed-load-tower-A-phase-Z 424 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:06] PROBLEM - ps1-d1-eqiad-infeed-load-tower-A-phase-X on ps1-d1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:06] PROBLEM - ps1-e12-eqiad-infeed-load-tower-B-phase-Z on ps1-e12-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:07] PROBLEM - ps1-b1-eqiad-infeed-load-tower-B-phase-Y on ps1-b1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:12] RECOVERY - ps1-d1-eqiad-infeed-load-tower-A-phase-X on ps1-d1-eqiad is OK: SNMP OK - ps1-d1-eqiad-infeed-load-tower-A-phase-X 512 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:12] RECOVERY - ps1-e12-eqiad-infeed-load-tower-B-phase-Z on ps1-e12-eqiad is OK: SNMP OK - ps1-e12-eqiad-infeed-load-tower-B-phase-Z 201 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:12] RECOVERY - ps1-e3-eqiad-infeed-load-tower-A-phase-Y on ps1-e3-eqiad is OK: SNMP OK - ps1-e3-eqiad-infeed-load-tower-A-phase-Y 573 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:12] PROBLEM - ps1-f7-eqiad-infeed-load-tower-A-phase-X on ps1-f7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:12] PROBLEM - ps1-e4-eqiad-infeed-load-tower-A-phase-Y on ps1-e4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:13] PROBLEM - ps1-f8-eqiad-infeed-load-tower-B-phase-Z on ps1-f8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:13] PROBLEM - ps1-c1-eqiad-infeed-load-tower-B-phase-Y on ps1-c1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:14] RECOVERY - ps1-b1-eqiad-infeed-load-tower-B-phase-Y on ps1-b1-eqiad is OK: SNMP OK - ps1-b1-eqiad-infeed-load-tower-B-phase-Y 234 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:16] RECOVERY - ps1-e4-eqiad-infeed-load-tower-A-phase-Y on ps1-e4-eqiad is OK: SNMP OK - ps1-e4-eqiad-infeed-load-tower-A-phase-Y 649 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:16] RECOVERY - ps1-f8-eqiad-infeed-load-tower-B-phase-Z on ps1-f8-eqiad is OK: SNMP OK - ps1-f8-eqiad-infeed-load-tower-B-phase-Z 504 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:17] RECOVERY - ps1-c1-eqiad-infeed-load-tower-B-phase-Y on ps1-c1-eqiad is OK: SNMP OK - ps1-c1-eqiad-infeed-load-tower-B-phase-Y 0 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:22] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-b7-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:31:48] PROBLEM - ps1-f3-eqiad-infeed-load-tower-A-phase-Y on ps1-f3-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:56] PROBLEM - ps1-f3-eqiad-infeed-load-tower-B-phase-Y on ps1-f3-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:31:58] RECOVERY - ps1-f7-eqiad-infeed-load-tower-A-phase-Z on ps1-f7-eqiad is OK: SNMP OK - ps1-f7-eqiad-infeed-load-tower-A-phase-Z 497 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:32:00] RECOVERY - ps1-f3-eqiad-infeed-load-tower-B-phase-Y on ps1-f3-eqiad is OK: SNMP OK - ps1-f3-eqiad-infeed-load-tower-B-phase-Y 472 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:32:00] RECOVERY - ps1-b5-eqiad-infeed-load-tower-A-phase-Y on ps1-b5-eqiad is OK: SNMP OK - ps1-b5-eqiad-infeed-load-tower-A-phase-Y 462 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:32:06] PROBLEM - ps1-f3-eqiad-infeed-load-tower-B-phase-X on ps1-f3-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:32:07] RECOVERY - ps1-b4-eqiad-infeed-load-tower-B-phase-X on ps1-b4-eqiad is OK: SNMP OK - ps1-b4-eqiad-infeed-load-tower-B-phase-X 494 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:32:08] PROBLEM - ps1-a5-eqiad-infeed-load-tower-B-phase-Y on ps1-a5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:32:08] RECOVERY - ps1-a4-eqiad-infeed-load-tower-B-phase-X on ps1-a4-eqiad is OK: SNMP OK - ps1-a4-eqiad-infeed-load-tower-B-phase-X 555 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:32:12] PROBLEM - ps1-e3-eqiad-infeed-load-tower-B-phase-Z on ps1-e3-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:32:12] PROBLEM - ps1-d1-eqiad-infeed-load-tower-B-phase-Z on ps1-d1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:32:12] RECOVERY - ps1-f3-eqiad-infeed-load-tower-B-phase-X on ps1-f3-eqiad is OK: SNMP OK - ps1-f3-eqiad-infeed-load-tower-B-phase-X 571 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:32:12] RECOVERY - ps1-a5-eqiad-infeed-load-tower-B-phase-Y on ps1-a5-eqiad is OK: SNMP OK - ps1-a5-eqiad-infeed-load-tower-B-phase-Y 417 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:32:16] RECOVERY - ps1-b1-eqiad-infeed-load-tower-A-phase-Z on ps1-b1-eqiad is OK: SNMP OK - ps1-b1-eqiad-infeed-load-tower-A-phase-Z 270 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:32:20] PROBLEM - ps1-b1-eqiad-infeed-load-tower-A-phase-Y on ps1-b1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:32:22] PROBLEM - Juniper alarms on fasw2-e16b-eqiad is CRITICAL: JNX_ALARMS CRITICAL - No response from remote host 10.65.4.250 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [13:32:48] PROBLEM - ps1-d1-eqiad-infeed-load-tower-B-phase-Y on ps1-d1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:32:48] PROBLEM - ps1-d8-eqiad-infeed-load-tower-B-phase-Y on ps1-d8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:33:06] PROBLEM - ps1-e9-eqiad-infeed-load-tower-B-phase-Z on ps1-e9-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:33:12] PROBLEM - ps1-e8-eqiad-infeed-load-tower-B-phase-Y on ps1-e8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:33:12] RECOVERY - ps1-c1-eqiad-infeed-load-tower-B-phase-X on ps1-c1-eqiad is OK: SNMP OK - ps1-c1-eqiad-infeed-load-tower-B-phase-X 47 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:33:14] RECOVERY - Juniper alarms on fasw2-e16b-eqiad is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [13:33:16] RECOVERY - ps1-b4-eqiad-infeed-load-tower-B-phase-Z on ps1-b4-eqiad is OK: SNMP OK - ps1-b4-eqiad-infeed-load-tower-B-phase-Z 291 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:33:16] RECOVERY - ps1-d1-eqiad-infeed-load-tower-B-phase-Z on ps1-d1-eqiad is OK: SNMP OK - ps1-d1-eqiad-infeed-load-tower-B-phase-Z 539 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:33:17] RECOVERY - ps1-f4-eqiad-infeed-load-tower-A-phase-X on ps1-f4-eqiad is OK: SNMP OK - ps1-f4-eqiad-infeed-load-tower-A-phase-X 519 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:33:20] PROBLEM - ps1-f8-eqiad-infeed-load-tower-B-phase-X on ps1-f8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:33:44] PROBLEM - ps1-a8-eqiad-infeed-load-tower-A-phase-X on ps1-a8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:33:46] PROBLEM - ps1-a1-eqiad-infeed-load-tower-B-phase-Y on ps1-a1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:33:50] RECOVERY - ps1-a8-eqiad-infeed-load-tower-A-phase-X on ps1-a8-eqiad is OK: SNMP OK - ps1-a8-eqiad-infeed-load-tower-A-phase-X 263 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:33:52] RECOVERY - ps1-f3-eqiad-infeed-load-tower-A-phase-Y on ps1-f3-eqiad is OK: SNMP OK - ps1-f3-eqiad-infeed-load-tower-A-phase-Y 441 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:33:52] RECOVERY - ps1-d1-eqiad-infeed-load-tower-B-phase-Y on ps1-d1-eqiad is OK: SNMP OK - ps1-d1-eqiad-infeed-load-tower-B-phase-Y 501 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:33:56] PROBLEM - ps1-d1-eqiad-infeed-load-tower-A-phase-Z on ps1-d1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:33:56] PROBLEM - ps1-c5-eqiad-infeed-load-tower-A-phase-Z on ps1-c5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:00] RECOVERY - ps1-b5-eqiad-infeed-load-tower-A-phase-X on ps1-b5-eqiad is OK: SNMP OK - ps1-b5-eqiad-infeed-load-tower-A-phase-X 355 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:00] PROBLEM - ps1-a1-eqiad-infeed-load-tower-A-phase-Y on ps1-a1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:00] RECOVERY - ps1-d1-eqiad-infeed-load-tower-A-phase-Z on ps1-d1-eqiad is OK: SNMP OK - ps1-d1-eqiad-infeed-load-tower-A-phase-Z 547 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:00] RECOVERY - ps1-c5-eqiad-infeed-load-tower-A-phase-Z on ps1-c5-eqiad is OK: SNMP OK - ps1-c5-eqiad-infeed-load-tower-A-phase-Z 507 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:00] I really should have planned the mgmt network uplink side of this better, apologies [13:34:02] PROBLEM - ps1-e8-eqiad-infeed-load-tower-B-phase-X on ps1-e8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:02] PROBLEM - ps1-b5-eqiad-infeed-load-tower-B-phase-X on ps1-b5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:04] PROBLEM - ps1-d1-eqiad-infeed-load-tower-A-phase-Y on ps1-d1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:04] PROBLEM - ps1-e7-eqiad-infeed-load-tower-B-phase-X on ps1-e7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:04] RECOVERY - ps1-a1-eqiad-infeed-load-tower-A-phase-Y on ps1-a1-eqiad is OK: SNMP OK - ps1-a1-eqiad-infeed-load-tower-A-phase-Y 241 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:06] RECOVERY - ps1-e8-eqiad-infeed-load-tower-B-phase-X on ps1-e8-eqiad is OK: SNMP OK - ps1-e8-eqiad-infeed-load-tower-B-phase-X 592 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:06] PROBLEM - ps1-e9-eqiad-infeed-load-tower-B-phase-Y on ps1-e9-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:06] PROBLEM - ps1-a5-eqiad-infeed-load-tower-A-phase-Z on ps1-a5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:07] PROBLEM - ps1-e4-eqiad-infeed-load-tower-B-phase-X on ps1-e4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:12] RECOVERY - ps1-e9-eqiad-infeed-load-tower-B-phase-Y on ps1-e9-eqiad is OK: SNMP OK - ps1-e9-eqiad-infeed-load-tower-B-phase-Y 222 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:12] PROBLEM - ps1-b1-eqiad-infeed-load-tower-B-phase-Z on ps1-b1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:12] PROBLEM - ps1-c4-eqiad-infeed-load-tower-A-phase-Y on ps1-c4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:14] RECOVERY - ps1-e9-eqiad-infeed-load-tower-B-phase-Z on ps1-e9-eqiad is OK: SNMP OK - ps1-e9-eqiad-infeed-load-tower-B-phase-Z 236 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:14] RECOVERY - ps1-e4-eqiad-infeed-load-tower-B-phase-X on ps1-e4-eqiad is OK: SNMP OK - ps1-e4-eqiad-infeed-load-tower-B-phase-X 553 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:14] RECOVERY - ps1-a5-eqiad-infeed-load-tower-A-phase-Z on ps1-a5-eqiad is OK: SNMP OK - ps1-a5-eqiad-infeed-load-tower-A-phase-Z 467 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:16] RECOVERY - ps1-b1-eqiad-infeed-load-tower-B-phase-Z on ps1-b1-eqiad is OK: SNMP OK - ps1-b1-eqiad-infeed-load-tower-B-phase-Z 268 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:16] RECOVERY - ps1-c4-eqiad-infeed-load-tower-A-phase-Y on ps1-c4-eqiad is OK: SNMP OK - ps1-c4-eqiad-infeed-load-tower-A-phase-Y 571 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:16] RECOVERY - ps1-f7-eqiad-infeed-load-tower-A-phase-X on ps1-f7-eqiad is OK: SNMP OK - ps1-f7-eqiad-infeed-load-tower-A-phase-X 458 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:23] topranks: is it only mgmt network? [13:34:26] RECOVERY - ps1-b1-eqiad-infeed-load-tower-A-phase-Y on ps1-b1-eqiad is OK: SNMP OK - ps1-b1-eqiad-infeed-load-tower-A-phase-Y 144 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:34] claime: yes 100% [13:34:36] PROBLEM - Juniper alarms on fasw2-e15a-eqiad is CRITICAL: JNX_ALARMS CRITICAL - No response from remote host 10.65.4.92 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [13:34:42] Trying to figure out if we need to hold the switchover [13:34:42] (03CR) 10Ottomata: "I'm not sure about this one! It sounds quite useful, because sometimes we need to roll out changes to all eventgate-* deployments (like f" [puppet] - 10https://gerrit.wikimedia.org/r/1359828 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [13:34:44] PROBLEM - ps1-e12-eqiad-infeed-load-tower-A-phase-Y on ps1-e12-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:48] PROBLEM - ps1-e12-eqiad-infeed-load-tower-B-phase-Y on ps1-e12-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:48] PROBLEM - ps1-f7-eqiad-infeed-load-tower-A-phase-Y on ps1-f7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:50] RECOVERY - ps1-e12-eqiad-infeed-load-tower-A-phase-Y on ps1-e12-eqiad is OK: SNMP OK - ps1-e12-eqiad-infeed-load-tower-A-phase-Y 212 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:52] RECOVERY - ps1-e12-eqiad-infeed-load-tower-B-phase-Y on ps1-e12-eqiad is OK: SNMP OK - ps1-e12-eqiad-infeed-load-tower-B-phase-Y 206 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:52] RECOVERY - ps1-f7-eqiad-infeed-load-tower-A-phase-Y on ps1-f7-eqiad is OK: SNMP OK - ps1-f7-eqiad-infeed-load-tower-A-phase-Y 459 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:34:56] PROBLEM - ps1-e9-eqiad-infeed-load-tower-A-phase-Z on ps1-e9-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:02] PROBLEM - ps1-a4-eqiad-infeed-load-tower-A-phase-Z on ps1-a4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:04] PROBLEM - ps1-f7-eqiad-infeed-load-tower-B-phase-Y on ps1-f7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:06] RECOVERY - ps1-b5-eqiad-infeed-load-tower-B-phase-X on ps1-b5-eqiad is OK: SNMP OK - ps1-b5-eqiad-infeed-load-tower-B-phase-X 362 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:06] RECOVERY - ps1-a8-eqiad-infeed-load-tower-A-phase-Z on ps1-a8-eqiad is OK: SNMP OK - ps1-a8-eqiad-infeed-load-tower-A-phase-Z 337 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:08] RECOVERY - ps1-f7-eqiad-infeed-load-tower-B-phase-Y on ps1-f7-eqiad is OK: SNMP OK - ps1-f7-eqiad-infeed-load-tower-B-phase-Y 463 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:08] RECOVERY - ps1-e7-eqiad-infeed-load-tower-B-phase-X on ps1-e7-eqiad is OK: SNMP OK - ps1-e7-eqiad-infeed-load-tower-B-phase-X 595 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:12] PROBLEM - ps1-e4-eqiad-infeed-load-tower-A-phase-Y on ps1-e4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:16] RECOVERY - ps1-e3-eqiad-infeed-load-tower-B-phase-Z on ps1-e3-eqiad is OK: SNMP OK - ps1-e3-eqiad-infeed-load-tower-B-phase-Z 549 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:16] RECOVERY - ps1-e4-eqiad-infeed-load-tower-A-phase-Y on ps1-e4-eqiad is OK: SNMP OK - ps1-e4-eqiad-infeed-load-tower-A-phase-Y 655 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:17] RECOVERY - ps1-e8-eqiad-infeed-load-tower-B-phase-Y on ps1-e8-eqiad is OK: SNMP OK - ps1-e8-eqiad-infeed-load-tower-B-phase-Y 392 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:22] PROBLEM - ps1-f4-eqiad-infeed-load-tower-A-phase-Y on ps1-f4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:36] RECOVERY - ps1-f8-eqiad-infeed-load-tower-B-phase-X on ps1-f8-eqiad is OK: SNMP OK - ps1-f8-eqiad-infeed-load-tower-B-phase-X 410 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:38] RECOVERY - ps1-f4-eqiad-infeed-load-tower-A-phase-Y on ps1-f4-eqiad is OK: SNMP OK - ps1-f4-eqiad-infeed-load-tower-A-phase-Y 463 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:40] topranks: (pinging because flood) trying to figure out if we need to hold the switchover, if it’s only mgmt we’re fine right [13:35:46] PROBLEM - ps1-b4-eqiad-infeed-load-tower-A-phase-Z on ps1-b4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:48] PROBLEM - ps1-e11-eqiad-infeed-load-tower-B-phase-Y on ps1-e11-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:52] RECOVERY - ps1-e11-eqiad-infeed-load-tower-B-phase-Y on ps1-e11-eqiad is OK: SNMP OK - ps1-e11-eqiad-infeed-load-tower-B-phase-Y 275 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:52] RECOVERY - ps1-d8-eqiad-infeed-load-tower-B-phase-Y on ps1-d8-eqiad is OK: SNMP OK - ps1-d8-eqiad-infeed-load-tower-B-phase-Y 274 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:54] PROBLEM - ps1-c4-eqiad-infeed-load-tower-B-phase-Y on ps1-c4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:54] PROBLEM - ps1-e8-eqiad-infeed-load-tower-B-phase-Z on ps1-e8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:56] PROBLEM - ps1-d8-eqiad-infeed-load-tower-B-phase-X on ps1-d8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:56] PROBLEM - ps1-f8-eqiad-infeed-load-tower-A-phase-Z on ps1-f8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:58] RECOVERY - ps1-e8-eqiad-infeed-load-tower-B-phase-Z on ps1-e8-eqiad is OK: SNMP OK - ps1-e8-eqiad-infeed-load-tower-B-phase-Z 421 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:35:58] RECOVERY - ps1-c4-eqiad-infeed-load-tower-B-phase-Y on ps1-c4-eqiad is OK: SNMP OK - ps1-c4-eqiad-infeed-load-tower-B-phase-Y 608 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:00] RECOVERY - ps1-e9-eqiad-infeed-load-tower-A-phase-Z on ps1-e9-eqiad is OK: SNMP OK - ps1-e9-eqiad-infeed-load-tower-A-phase-Z 206 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:00] RECOVERY - ps1-f8-eqiad-infeed-load-tower-A-phase-Z on ps1-f8-eqiad is OK: SNMP OK - ps1-f8-eqiad-infeed-load-tower-A-phase-Z 495 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:04] PROBLEM - ps1-a4-eqiad-infeed-load-tower-B-phase-Y on ps1-a4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:06] PROBLEM - ps1-e12-eqiad-infeed-load-tower-B-phase-Z on ps1-e12-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:06] PROBLEM - ps1-b8-eqiad-infeed-load-tower-B-phase-X on ps1-b8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:07] PROBLEM - ps1-b8-eqiad-infeed-load-tower-A-phase-Y on ps1-b8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:08] RECOVERY - ps1-d1-eqiad-infeed-load-tower-A-phase-Y on ps1-d1-eqiad is OK: SNMP OK - ps1-d1-eqiad-infeed-load-tower-A-phase-Y 535 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:12] PROBLEM - ps1-e8-eqiad-infeed-load-tower-A-phase-X on ps1-e8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:12] PROBLEM - ps1-c4-eqiad-infeed-load-tower-A-phase-X on ps1-c4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:12] PROBLEM - ps1-b1-eqiad-infeed-load-tower-A-phase-Z on ps1-b1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:12] RECOVERY - ps1-b8-eqiad-infeed-load-tower-B-phase-X on ps1-b8-eqiad is OK: SNMP OK - ps1-b8-eqiad-infeed-load-tower-B-phase-X 352 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:12] RECOVERY - ps1-b8-eqiad-infeed-load-tower-A-phase-Y on ps1-b8-eqiad is OK: SNMP OK - ps1-b8-eqiad-infeed-load-tower-A-phase-Y 488 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:18] RECOVERY - ps1-c4-eqiad-infeed-load-tower-A-phase-X on ps1-c4-eqiad is OK: SNMP OK - ps1-c4-eqiad-infeed-load-tower-A-phase-X 570 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:18] RECOVERY - ps1-b1-eqiad-infeed-load-tower-A-phase-Z on ps1-b1-eqiad is OK: SNMP OK - ps1-b1-eqiad-infeed-load-tower-A-phase-Z 270 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:20] PROBLEM - ps1-c1-eqiad-infeed-load-tower-B-phase-Z on ps1-c1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:20] PROBLEM - ps1-c5-eqiad-infeed-load-tower-A-phase-X on ps1-c5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:22] FIRING: [7x] CertAlmostExpired: gNMI TLS certificate for fasw2-e16b-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:36:26] RECOVERY - ps1-c1-eqiad-infeed-load-tower-B-phase-Z on ps1-c1-eqiad is OK: SNMP OK - ps1-c1-eqiad-infeed-load-tower-B-phase-Z 26 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:26] RECOVERY - Juniper alarms on fasw2-e15a-eqiad is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [13:36:36] (03PS1) 10Jforrester: restbase: Drop cxserver from envoy::enabled_listeners, unused [puppet] - 10https://gerrit.wikimedia.org/r/1359836 (https://phabricator.wikimedia.org/T133001) [13:36:50] RECOVERY - ps1-b4-eqiad-infeed-load-tower-A-phase-Z on ps1-b4-eqiad is OK: SNMP OK - ps1-b4-eqiad-infeed-load-tower-A-phase-Z 259 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:56] PROBLEM - ps1-f4-eqiad-infeed-load-tower-A-phase-Z on ps1-f4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:36:56] PROBLEM - ps1-e3-eqiad-infeed-load-tower-A-phase-Z on ps1-e3-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:37:02] PROBLEM - ps1-c5-eqiad-infeed-load-tower-B-phase-Z on ps1-c5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:37:04] PROBLEM - ps1-d4-eqiad-infeed-load-tower-A-phase-Y on ps1-d4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:37:04] PROBLEM - ps1-b8-eqiad-infeed-load-tower-B-phase-Y on ps1-b8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:37:06] PROBLEM - ps1-a1-eqiad-infeed-load-tower-B-phase-X on ps1-a1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:37:07] RECOVERY - ps1-c5-eqiad-infeed-load-tower-B-phase-Z on ps1-c5-eqiad is OK: SNMP OK - ps1-c5-eqiad-infeed-load-tower-B-phase-Z 514 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:37:08] PROBLEM - ps1-a5-eqiad-infeed-load-tower-B-phase-Y on ps1-a5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:37:08] PROBLEM - ps1-e9-eqiad-infeed-load-tower-A-phase-Y on ps1-e9-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:37:12] PROBLEM - ps1-f3-eqiad-infeed-load-tower-A-phase-X on ps1-f3-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:37:12] RECOVERY - ps1-a5-eqiad-infeed-load-tower-B-phase-Y on ps1-a5-eqiad is OK: SNMP OK - ps1-a5-eqiad-infeed-load-tower-B-phase-Y 418 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:37:30] 06SRE, 10Citoid, 10CXServer, 10RESTBase, and 3 others: Decom legacy ex-parsoidcache cxserver, citoid, and restbase service hostnames - https://phabricator.wikimedia.org/T133001#12484346 (10Jdforrester-WMF) > cxserver.wikimedia.org - RB conversion still TODO For public traffic, ATS now sends cxserver.wikim... [13:37:35] 10ops-codfw, 06DC-Ops, 06ServiceOps: hw troubleshooting: power-on for wikikube-worker2337.codfw.wmnet - https://phabricator.wikimedia.org/T442714 (10Blake) 03NEW [13:37:48] PROBLEM - ps1-c8-eqiad-infeed-load-tower-B-phase-Y on ps1-c8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:37:48] PROBLEM - ps1-f3-eqiad-infeed-load-tower-A-phase-Y on ps1-f3-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:37:52] RECOVERY - ps1-f3-eqiad-infeed-load-tower-A-phase-Y on ps1-f3-eqiad is OK: SNMP OK - ps1-f3-eqiad-infeed-load-tower-A-phase-Y 454 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:37:52] RECOVERY - ps1-c8-eqiad-infeed-load-tower-B-phase-Y on ps1-c8-eqiad is OK: SNMP OK - ps1-c8-eqiad-infeed-load-tower-B-phase-Y 622 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:00] RECOVERY - ps1-d8-eqiad-infeed-load-tower-B-phase-X on ps1-d8-eqiad is OK: SNMP OK - ps1-d8-eqiad-infeed-load-tower-B-phase-X 301 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:04] PROBLEM - ps1-a8-eqiad-infeed-load-tower-A-phase-Y on ps1-a8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:06] PROBLEM - ps1-d8-eqiad-infeed-load-tower-A-phase-X on ps1-d8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:06] PROBLEM - ps1-c1-eqiad-infeed-load-tower-A-phase-X on ps1-c1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:08] RECOVERY - ps1-a4-eqiad-infeed-load-tower-A-phase-Z on ps1-a4-eqiad is OK: SNMP OK - ps1-a4-eqiad-infeed-load-tower-A-phase-Z 493 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:10] RECOVERY - ps1-d8-eqiad-infeed-load-tower-A-phase-X on ps1-d8-eqiad is OK: SNMP OK - ps1-d8-eqiad-infeed-load-tower-A-phase-X 296 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:12] RECOVERY - ps1-e12-eqiad-infeed-load-tower-B-phase-Z on ps1-e12-eqiad is OK: SNMP OK - ps1-e12-eqiad-infeed-load-tower-B-phase-Z 203 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:12] RECOVERY - ps1-e9-eqiad-infeed-load-tower-A-phase-Y on ps1-e9-eqiad is OK: SNMP OK - ps1-e9-eqiad-infeed-load-tower-A-phase-Y 225 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:18] RECOVERY - ps1-f3-eqiad-infeed-load-tower-A-phase-X on ps1-f3-eqiad is OK: SNMP OK - ps1-f3-eqiad-infeed-load-tower-A-phase-X 553 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:18] RECOVERY - ps1-e8-eqiad-infeed-load-tower-A-phase-X on ps1-e8-eqiad is OK: SNMP OK - ps1-e8-eqiad-infeed-load-tower-A-phase-X 602 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:22] PROBLEM - ps1-e3-eqiad-infeed-load-tower-A-phase-X on ps1-e3-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:22] PROBLEM - ps1-e7-eqiad-infeed-load-tower-A-phase-Y on ps1-e7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:28] RECOVERY - ps1-e7-eqiad-infeed-load-tower-A-phase-Y on ps1-e7-eqiad is OK: SNMP OK - ps1-e7-eqiad-infeed-load-tower-A-phase-Y 539 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:44] PROBLEM - ps1-a8-eqiad-infeed-load-tower-A-phase-X on ps1-a8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:46] PROBLEM - ps1-b1-eqiad-infeed-load-tower-A-phase-X on ps1-b1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:46] PROBLEM - ps1-b4-eqiad-infeed-load-tower-A-phase-Y on ps1-b4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:46] PROBLEM - ps1-a5-eqiad-infeed-load-tower-A-phase-X on ps1-a5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:46] PROBLEM - ps1-a4-eqiad-infeed-load-tower-B-phase-Z on ps1-a4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:46] PROBLEM - ps1-d4-eqiad-infeed-load-tower-B-phase-X on ps1-d4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:47] PROBLEM - ps1-a4-eqiad-infeed-load-tower-A-phase-X on ps1-a4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:50] RECOVERY - ps1-a1-eqiad-infeed-load-tower-B-phase-Y on ps1-a1-eqiad is OK: SNMP OK - ps1-a1-eqiad-infeed-load-tower-B-phase-Y 345 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:50] RECOVERY - ps1-b1-eqiad-infeed-load-tower-A-phase-X on ps1-b1-eqiad is OK: SNMP OK - ps1-b1-eqiad-infeed-load-tower-A-phase-X 358 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:50] RECOVERY - ps1-a4-eqiad-infeed-load-tower-A-phase-X on ps1-a4-eqiad is OK: SNMP OK - ps1-a4-eqiad-infeed-load-tower-A-phase-X 568 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:50] RECOVERY - ps1-a4-eqiad-infeed-load-tower-B-phase-Z on ps1-a4-eqiad is OK: SNMP OK - ps1-a4-eqiad-infeed-load-tower-B-phase-Z 501 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:50] RECOVERY - ps1-a5-eqiad-infeed-load-tower-A-phase-X on ps1-a5-eqiad is OK: SNMP OK - ps1-a5-eqiad-infeed-load-tower-A-phase-X 469 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:38:56] PROBLEM - ps1-e4-eqiad-infeed-load-tower-A-phase-Z on ps1-e4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:00] PROBLEM - ps1-a1-eqiad-infeed-load-tower-A-phase-Y on ps1-a1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:00] RECOVERY - ps1-e3-eqiad-infeed-load-tower-A-phase-Z on ps1-e3-eqiad is OK: SNMP OK - ps1-e3-eqiad-infeed-load-tower-A-phase-Z 547 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:00] RECOVERY - ps1-f4-eqiad-infeed-load-tower-A-phase-Z on ps1-f4-eqiad is OK: SNMP OK - ps1-f4-eqiad-infeed-load-tower-A-phase-Z 495 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:00] RECOVERY - ps1-e4-eqiad-infeed-load-tower-A-phase-Z on ps1-e4-eqiad is OK: SNMP OK - ps1-e4-eqiad-infeed-load-tower-A-phase-Z 614 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:02] PROBLEM - ps1-b4-eqiad-infeed-load-tower-B-phase-X on ps1-b4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:02] PROBLEM - ps1-d4-eqiad-infeed-load-tower-B-phase-Z on ps1-d4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:04] PROBLEM - ps1-a1-eqiad-infeed-load-tower-A-phase-Z on ps1-a1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:06] PROBLEM - ps1-c4-eqiad-infeed-load-tower-B-phase-X on ps1-c4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:06] PROBLEM - ps1-f3-eqiad-infeed-load-tower-B-phase-X on ps1-f3-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:07] PROBLEM - ps1-e12-eqiad-infeed-load-tower-B-phase-X on ps1-e12-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:07] PROBLEM - ps1-e7-eqiad-infeed-load-tower-A-phase-X on ps1-e7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:08] PROBLEM - ps1-c5-eqiad-infeed-load-tower-A-phase-Y on ps1-c5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:08] RECOVERY - ps1-d4-eqiad-infeed-load-tower-B-phase-Z on ps1-d4-eqiad is OK: SNMP OK - ps1-d4-eqiad-infeed-load-tower-B-phase-Z 471 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:08] RECOVERY - ps1-a4-eqiad-infeed-load-tower-B-phase-Y on ps1-a4-eqiad is OK: SNMP OK - ps1-a4-eqiad-infeed-load-tower-B-phase-Y 586 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:08] RECOVERY - ps1-d4-eqiad-infeed-load-tower-A-phase-Y on ps1-d4-eqiad is OK: SNMP OK - ps1-d4-eqiad-infeed-load-tower-A-phase-Y 354 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:10] RECOVERY - ps1-a1-eqiad-infeed-load-tower-A-phase-Z on ps1-a1-eqiad is OK: SNMP OK - ps1-a1-eqiad-infeed-load-tower-A-phase-Z 468 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:12] PROBLEM - ps1-a8-eqiad-infeed-load-tower-B-phase-Z on ps1-a8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:12] PROBLEM - ps1-d1-eqiad-infeed-load-tower-B-phase-Z on ps1-d1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:12] PROBLEM - ps1-f8-eqiad-infeed-load-tower-B-phase-Z on ps1-f8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:12] PROBLEM - ps1-f8-eqiad-infeed-load-tower-B-phase-Y on ps1-f8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:12] PROBLEM - ps1-e8-eqiad-infeed-load-tower-B-phase-Y on ps1-e8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:14] RECOVERY - ps1-c4-eqiad-infeed-load-tower-B-phase-X on ps1-c4-eqiad is OK: SNMP OK - ps1-c4-eqiad-infeed-load-tower-B-phase-X 511 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:14] RECOVERY - ps1-c1-eqiad-infeed-load-tower-A-phase-X on ps1-c1-eqiad is OK: SNMP OK - ps1-c1-eqiad-infeed-load-tower-A-phase-X 53 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:16] RECOVERY - ps1-d1-eqiad-infeed-load-tower-B-phase-Z on ps1-d1-eqiad is OK: SNMP OK - ps1-d1-eqiad-infeed-load-tower-B-phase-Z 531 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:16] RECOVERY - ps1-e8-eqiad-infeed-load-tower-B-phase-Y on ps1-e8-eqiad is OK: SNMP OK - ps1-e8-eqiad-infeed-load-tower-B-phase-Y 394 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:24] PROBLEM - Juniper alarms on fasw2-e16b-eqiad is CRITICAL: JNX_ALARMS CRITICAL - No response from remote host 10.65.4.250 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [13:39:26] RECOVERY - ps1-c5-eqiad-infeed-load-tower-A-phase-X on ps1-c5-eqiad is OK: SNMP OK - ps1-c5-eqiad-infeed-load-tower-A-phase-X 504 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:36] PROBLEM - Juniper alarms on fasw2-e15a-eqiad is CRITICAL: JNX_ALARMS CRITICAL - No response from remote host 10.65.4.92 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [13:39:48] PROBLEM - ps1-d1-eqiad-infeed-load-tower-B-phase-Y on ps1-d1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:48] PROBLEM - ps1-e11-eqiad-infeed-load-tower-B-phase-Y on ps1-e11-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:50] RECOVERY - ps1-a8-eqiad-infeed-load-tower-A-phase-X on ps1-a8-eqiad is OK: SNMP OK - ps1-a8-eqiad-infeed-load-tower-A-phase-X 265 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:50] RECOVERY - ps1-b4-eqiad-infeed-load-tower-A-phase-Y on ps1-b4-eqiad is OK: SNMP OK - ps1-b4-eqiad-infeed-load-tower-A-phase-Y 475 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:39:52] RECOVERY - ps1-d1-eqiad-infeed-load-tower-B-phase-Y on ps1-d1-eqiad is OK: SNMP OK - ps1-d1-eqiad-infeed-load-tower-B-phase-Y 511 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:02] PROBLEM - ps1-e11-eqiad-infeed-load-tower-B-phase-X on ps1-e11-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:04] PROBLEM - ps1-d1-eqiad-infeed-load-tower-A-phase-Y on ps1-d1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:04] PROBLEM - ps1-f7-eqiad-infeed-load-tower-B-phase-Y on ps1-f7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:04] RECOVERY - ps1-a1-eqiad-infeed-load-tower-A-phase-Y on ps1-a1-eqiad is OK: SNMP OK - ps1-a1-eqiad-infeed-load-tower-A-phase-Y 241 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:08] RECOVERY - ps1-b4-eqiad-infeed-load-tower-B-phase-X on ps1-b4-eqiad is OK: SNMP OK - ps1-b4-eqiad-infeed-load-tower-B-phase-X 477 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:08] PROBLEM - ps1-e4-eqiad-infeed-load-tower-B-phase-Z on ps1-e4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:08] PROBLEM - ps1-b5-eqiad-infeed-load-tower-B-phase-Z on ps1-b5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:08] PROBLEM - ps1-f7-eqiad-infeed-load-tower-B-phase-X on ps1-f7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:10] RECOVERY - ps1-d1-eqiad-infeed-load-tower-A-phase-Y on ps1-d1-eqiad is OK: SNMP OK - ps1-d1-eqiad-infeed-load-tower-A-phase-Y 521 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:10] RECOVERY - ps1-a8-eqiad-infeed-load-tower-A-phase-Y on ps1-a8-eqiad is OK: SNMP OK - ps1-a8-eqiad-infeed-load-tower-A-phase-Y 473 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:10] RECOVERY - ps1-f7-eqiad-infeed-load-tower-B-phase-Y on ps1-f7-eqiad is OK: SNMP OK - ps1-f7-eqiad-infeed-load-tower-B-phase-Y 493 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:10] RECOVERY - ps1-b8-eqiad-infeed-load-tower-B-phase-Y on ps1-b8-eqiad is OK: SNMP OK - ps1-b8-eqiad-infeed-load-tower-B-phase-Y 497 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:12] PROBLEM - ps1-e4-eqiad-infeed-load-tower-A-phase-X on ps1-e4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:12] PROBLEM - ps1-b1-eqiad-infeed-load-tower-A-phase-Z on ps1-b1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:12] PROBLEM - ps1-e12-eqiad-infeed-load-tower-A-phase-X on ps1-e12-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:12] RECOVERY - ps1-f7-eqiad-infeed-load-tower-B-phase-X on ps1-f7-eqiad is OK: SNMP OK - ps1-f7-eqiad-infeed-load-tower-B-phase-X 460 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:14] RECOVERY - Juniper alarms on fasw2-e16b-eqiad is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [13:40:14] RECOVERY - ps1-f3-eqiad-infeed-load-tower-B-phase-X on ps1-f3-eqiad is OK: SNMP OK - ps1-f3-eqiad-infeed-load-tower-B-phase-X 600 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:14] RECOVERY - ps1-e7-eqiad-infeed-load-tower-A-phase-X on ps1-e7-eqiad is OK: SNMP OK - ps1-e7-eqiad-infeed-load-tower-A-phase-X 591 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:14] RECOVERY - ps1-c5-eqiad-infeed-load-tower-A-phase-Y on ps1-c5-eqiad is OK: SNMP OK - ps1-c5-eqiad-infeed-load-tower-A-phase-Y 478 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:16] RECOVERY - ps1-a8-eqiad-infeed-load-tower-B-phase-Z on ps1-a8-eqiad is OK: SNMP OK - ps1-a8-eqiad-infeed-load-tower-B-phase-Z 398 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:16] RECOVERY - ps1-f8-eqiad-infeed-load-tower-B-phase-Y on ps1-f8-eqiad is OK: SNMP OK - ps1-f8-eqiad-infeed-load-tower-B-phase-Y 453 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:16] RECOVERY - ps1-f8-eqiad-infeed-load-tower-B-phase-Z on ps1-f8-eqiad is OK: SNMP OK - ps1-f8-eqiad-infeed-load-tower-B-phase-Z 469 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:26] RECOVERY - Juniper alarms on fasw2-e15a-eqiad is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [13:40:28] RECOVERY - ps1-e3-eqiad-infeed-load-tower-A-phase-X on ps1-e3-eqiad is OK: SNMP OK - ps1-e3-eqiad-infeed-load-tower-A-phase-X 559 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:40:50] RECOVERY - ps1-d4-eqiad-infeed-load-tower-B-phase-X on ps1-d4-eqiad is OK: SNMP OK - ps1-d4-eqiad-infeed-load-tower-B-phase-X 396 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:41:02] PROBLEM - ps1-a4-eqiad-infeed-load-tower-B-phase-X on ps1-a4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:41:08] RECOVERY - ps1-e11-eqiad-infeed-load-tower-B-phase-X on ps1-e11-eqiad is OK: SNMP OK - ps1-e11-eqiad-infeed-load-tower-B-phase-X 260 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:41:08] RECOVERY - ps1-a4-eqiad-infeed-load-tower-B-phase-X on ps1-a4-eqiad is OK: SNMP OK - ps1-a4-eqiad-infeed-load-tower-B-phase-X 582 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:41:08] PROBLEM - ps1-e4-eqiad-infeed-load-tower-B-phase-X on ps1-e4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:41:12] PROBLEM - ps1-e9-eqiad-infeed-load-tower-A-phase-X on ps1-e9-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:41:13] RECOVERY - ps1-b5-eqiad-infeed-load-tower-B-phase-Z on ps1-b5-eqiad is OK: SNMP OK - ps1-b5-eqiad-infeed-load-tower-B-phase-Z 498 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:41:14] RECOVERY - ps1-e4-eqiad-infeed-load-tower-B-phase-X on ps1-e4-eqiad is OK: SNMP OK - ps1-e4-eqiad-infeed-load-tower-B-phase-X 579 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:41:14] RECOVERY - ps1-e4-eqiad-infeed-load-tower-B-phase-Z on ps1-e4-eqiad is OK: SNMP OK - ps1-e4-eqiad-infeed-load-tower-B-phase-Z 618 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:41:14] RECOVERY - ps1-e12-eqiad-infeed-load-tower-B-phase-X on ps1-e12-eqiad is OK: SNMP OK - ps1-e12-eqiad-infeed-load-tower-B-phase-X 211 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:41:16] RECOVERY - ps1-e4-eqiad-infeed-load-tower-A-phase-X on ps1-e4-eqiad is OK: SNMP OK - ps1-e4-eqiad-infeed-load-tower-A-phase-X 560 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:41:18] RECOVERY - ps1-e12-eqiad-infeed-load-tower-A-phase-X on ps1-e12-eqiad is OK: SNMP OK - ps1-e12-eqiad-infeed-load-tower-A-phase-X 201 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:41:18] RECOVERY - ps1-b1-eqiad-infeed-load-tower-A-phase-Z on ps1-b1-eqiad is OK: SNMP OK - ps1-b1-eqiad-infeed-load-tower-A-phase-Z 266 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:41:44] RESOLVED: [4x] CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:41:54] PROBLEM - ps1-d4-eqiad-infeed-load-tower-A-phase-Z on ps1-d4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:41:56] PROBLEM - ps1-e11-eqiad-infeed-load-tower-B-phase-Z on ps1-e11-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:41:56] PROBLEM - ps1-f3-eqiad-infeed-load-tower-A-phase-Z on ps1-f3-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:42:00] RECOVERY - ps1-d4-eqiad-infeed-load-tower-A-phase-Z on ps1-d4-eqiad is OK: SNMP OK - ps1-d4-eqiad-infeed-load-tower-A-phase-Z 443 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:42:00] RECOVERY - ps1-f3-eqiad-infeed-load-tower-A-phase-Z on ps1-f3-eqiad is OK: SNMP OK - ps1-f3-eqiad-infeed-load-tower-A-phase-Z 614 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:42:12] PROBLEM - ps1-c1-eqiad-infeed-load-tower-A-phase-Y on ps1-c1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:42:14] RECOVERY - ps1-a1-eqiad-infeed-load-tower-B-phase-X on ps1-a1-eqiad is OK: SNMP OK - ps1-a1-eqiad-infeed-load-tower-B-phase-X 188 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:42:16] RECOVERY - ps1-c1-eqiad-infeed-load-tower-A-phase-Y on ps1-c1-eqiad is OK: SNMP OK - ps1-c1-eqiad-infeed-load-tower-A-phase-Y 0 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:42:54] PROBLEM - ps1-c4-eqiad-infeed-load-tower-B-phase-Y on ps1-c4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:42:56] PROBLEM - ps1-d8-eqiad-infeed-load-tower-A-phase-Z on ps1-d8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:42:56] PROBLEM - ps1-f7-eqiad-infeed-load-tower-B-phase-Z on ps1-f7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:42:56] PROBLEM - ps1-f3-eqiad-infeed-load-tower-B-phase-Y on ps1-f3-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:42:56] PROBLEM - ps1-c4-eqiad-infeed-load-tower-A-phase-Z on ps1-c4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:42:56] PROBLEM - ps1-b5-eqiad-infeed-load-tower-A-phase-Y on ps1-b5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:43:00] RECOVERY - ps1-b5-eqiad-infeed-load-tower-A-phase-Y on ps1-b5-eqiad is OK: SNMP OK - ps1-b5-eqiad-infeed-load-tower-A-phase-Y 458 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:43:00] RECOVERY - ps1-f7-eqiad-infeed-load-tower-B-phase-Z on ps1-f7-eqiad is OK: SNMP OK - ps1-f7-eqiad-infeed-load-tower-B-phase-Z 496 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:43:00] RECOVERY - ps1-c4-eqiad-infeed-load-tower-A-phase-Z on ps1-c4-eqiad is OK: SNMP OK - ps1-c4-eqiad-infeed-load-tower-A-phase-Z 540 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:43:00] RECOVERY - ps1-d8-eqiad-infeed-load-tower-A-phase-Z on ps1-d8-eqiad is OK: SNMP OK - ps1-d8-eqiad-infeed-load-tower-A-phase-Z 415 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:43:04] PROBLEM - ps1-d4-eqiad-infeed-load-tower-A-phase-Y on ps1-d4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:43:08] RECOVERY - ps1-d4-eqiad-infeed-load-tower-A-phase-Y on ps1-d4-eqiad is OK: SNMP OK - ps1-d4-eqiad-infeed-load-tower-A-phase-Y 355 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:43:12] PROBLEM - ps1-f7-eqiad-infeed-load-tower-A-phase-X on ps1-f7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:43:18] RECOVERY - ps1-e9-eqiad-infeed-load-tower-A-phase-X on ps1-e9-eqiad is OK: SNMP OK - ps1-e9-eqiad-infeed-load-tower-A-phase-X 223 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:43:20] PROBLEM - ps1-c5-eqiad-infeed-load-tower-A-phase-X on ps1-c5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:43:22] PROBLEM - ps1-e7-eqiad-infeed-load-tower-A-phase-Y on ps1-e7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:43:22] FIRING: CertAlmostExpired: gNMI TLS certificate for fasw2-e15a-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:43:24] (03PS1) 10Elukey: osm: add a hack to send-tile-expiration-events.sh [puppet] - 10https://gerrit.wikimedia.org/r/1359841 (https://phabricator.wikimedia.org/T442715) [13:43:26] RECOVERY - ps1-c5-eqiad-infeed-load-tower-A-phase-X on ps1-c5-eqiad is OK: SNMP OK - ps1-c5-eqiad-infeed-load-tower-A-phase-X 491 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:43:52] RECOVERY - ps1-e11-eqiad-infeed-load-tower-B-phase-Y on ps1-e11-eqiad is OK: SNMP OK - ps1-e11-eqiad-infeed-load-tower-B-phase-Y 325 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:43:58] RECOVERY - ps1-c4-eqiad-infeed-load-tower-B-phase-Y on ps1-c4-eqiad is OK: SNMP OK - ps1-c4-eqiad-infeed-load-tower-B-phase-Y 589 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:04] PROBLEM - ps1-a4-eqiad-infeed-load-tower-B-phase-Y on ps1-a4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:08] PROBLEM - ps1-d8-eqiad-infeed-load-tower-B-phase-Z on ps1-d8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:08] PROBLEM - ps1-c4-eqiad-infeed-load-tower-B-phase-Z on ps1-c4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:08] RECOVERY - ps1-a4-eqiad-infeed-load-tower-B-phase-Y on ps1-a4-eqiad is OK: SNMP OK - ps1-a4-eqiad-infeed-load-tower-B-phase-Y 588 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:12] PROBLEM - ps1-f8-eqiad-infeed-load-tower-B-phase-Z on ps1-f8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:12] PROBLEM - ps1-b1-eqiad-infeed-load-tower-B-phase-Z on ps1-b1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:16] RECOVERY - ps1-b1-eqiad-infeed-load-tower-B-phase-Z on ps1-b1-eqiad is OK: SNMP OK - ps1-b1-eqiad-infeed-load-tower-B-phase-Z 262 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:18] RECOVERY - ps1-f7-eqiad-infeed-load-tower-A-phase-X on ps1-f7-eqiad is OK: SNMP OK - ps1-f7-eqiad-infeed-load-tower-A-phase-X 462 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:22] PROBLEM - ps1-c8-eqiad-infeed-load-tower-A-phase-X on ps1-c8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:22] PROBLEM - ps1-e3-eqiad-infeed-load-tower-A-phase-X on ps1-e3-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:22] PROBLEM - ps1-e11-eqiad-infeed-load-tower-A-phase-X on ps1-e11-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:26] RECOVERY - ps1-c8-eqiad-infeed-load-tower-A-phase-X on ps1-c8-eqiad is OK: SNMP OK - ps1-c8-eqiad-infeed-load-tower-A-phase-X 503 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:28] RECOVERY - ps1-e7-eqiad-infeed-load-tower-A-phase-Y on ps1-e7-eqiad is OK: SNMP OK - ps1-e7-eqiad-infeed-load-tower-A-phase-Y 538 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:46] PROBLEM - ps1-a1-eqiad-infeed-load-tower-B-phase-Z on ps1-a1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:46] PROBLEM - ps1-b1-eqiad-infeed-load-tower-A-phase-X on ps1-b1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:48] PROBLEM - ps1-f7-eqiad-infeed-load-tower-A-phase-Y on ps1-f7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:54] PROBLEM - ps1-e8-eqiad-infeed-load-tower-B-phase-Z on ps1-e8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:54] PROBLEM - ps1-f4-eqiad-infeed-load-tower-B-phase-Y on ps1-f4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:58] RECOVERY - ps1-f4-eqiad-infeed-load-tower-B-phase-Y on ps1-f4-eqiad is OK: SNMP OK - ps1-f4-eqiad-infeed-load-tower-B-phase-Y 462 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:44:58] RECOVERY - ps1-e8-eqiad-infeed-load-tower-B-phase-Z on ps1-e8-eqiad is OK: SNMP OK - ps1-e8-eqiad-infeed-load-tower-B-phase-Z 419 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:00] RECOVERY - ps1-f3-eqiad-infeed-load-tower-B-phase-Y on ps1-f3-eqiad is OK: SNMP OK - ps1-f3-eqiad-infeed-load-tower-B-phase-Y 430 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:02] PROBLEM - ps1-a8-eqiad-infeed-load-tower-A-phase-Z on ps1-a8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:02] PROBLEM - ps1-b4-eqiad-infeed-load-tower-B-phase-X on ps1-b4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:04] PROBLEM - ps1-a1-eqiad-infeed-load-tower-A-phase-Z on ps1-a1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:06] RECOVERY - ps1-a8-eqiad-infeed-load-tower-A-phase-Z on ps1-a8-eqiad is OK: SNMP OK - ps1-a8-eqiad-infeed-load-tower-A-phase-Z 334 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:08] RECOVERY - ps1-b4-eqiad-infeed-load-tower-B-phase-X on ps1-b4-eqiad is OK: SNMP OK - ps1-b4-eqiad-infeed-load-tower-B-phase-X 471 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:08] PROBLEM - ps1-c1-eqiad-infeed-load-tower-B-phase-X on ps1-c1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:10] RECOVERY - ps1-a1-eqiad-infeed-load-tower-A-phase-Z on ps1-a1-eqiad is OK: SNMP OK - ps1-a1-eqiad-infeed-load-tower-A-phase-Z 464 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:12] PROBLEM - ps1-e4-eqiad-infeed-load-tower-A-phase-Y on ps1-e4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:12] PROBLEM - ps1-e8-eqiad-infeed-load-tower-A-phase-X on ps1-e8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:14] RECOVERY - ps1-d8-eqiad-infeed-load-tower-B-phase-Z on ps1-d8-eqiad is OK: SNMP OK - ps1-d8-eqiad-infeed-load-tower-B-phase-Z 303 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:16] RECOVERY - ps1-f8-eqiad-infeed-load-tower-B-phase-Z on ps1-f8-eqiad is OK: SNMP OK - ps1-f8-eqiad-infeed-load-tower-B-phase-Z 460 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:22] PROBLEM - ps1-e7-eqiad-infeed-load-tower-B-phase-Y on ps1-e7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:28] RECOVERY - ps1-e11-eqiad-infeed-load-tower-A-phase-X on ps1-e11-eqiad is OK: SNMP OK - ps1-e11-eqiad-infeed-load-tower-A-phase-X 280 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:44] PROBLEM - ps1-e12-eqiad-infeed-load-tower-A-phase-Y on ps1-e12-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:48] PROBLEM - ps1-f8-eqiad-infeed-load-tower-A-phase-Y on ps1-f8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:50] RECOVERY - ps1-a1-eqiad-infeed-load-tower-B-phase-Z on ps1-a1-eqiad is OK: SNMP OK - ps1-a1-eqiad-infeed-load-tower-B-phase-Z 399 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:50] RECOVERY - ps1-b1-eqiad-infeed-load-tower-A-phase-X on ps1-b1-eqiad is OK: SNMP OK - ps1-b1-eqiad-infeed-load-tower-A-phase-X 346 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:52] RECOVERY - ps1-f8-eqiad-infeed-load-tower-A-phase-Y on ps1-f8-eqiad is OK: SNMP OK - ps1-f8-eqiad-infeed-load-tower-A-phase-Y 442 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:52] RECOVERY - ps1-f7-eqiad-infeed-load-tower-A-phase-Y on ps1-f7-eqiad is OK: SNMP OK - ps1-f7-eqiad-infeed-load-tower-A-phase-Y 457 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:45:54] PROBLEM - ps1-b5-eqiad-infeed-load-tower-A-phase-X on ps1-b5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:01] RECOVERY - ps1-e11-eqiad-infeed-load-tower-B-phase-Z on ps1-e11-eqiad is OK: SNMP OK - ps1-e11-eqiad-infeed-load-tower-B-phase-Z 263 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:04] PROBLEM - ps1-b8-eqiad-infeed-load-tower-B-phase-Y on ps1-b8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:08] PROBLEM - ps1-b8-eqiad-infeed-load-tower-A-phase-Y on ps1-b8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:08] PROBLEM - ps1-a1-eqiad-infeed-load-tower-B-phase-X on ps1-a1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:08] PROBLEM - ps1-e7-eqiad-infeed-load-tower-A-phase-X on ps1-e7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:10] RECOVERY - ps1-b8-eqiad-infeed-load-tower-B-phase-Y on ps1-b8-eqiad is OK: SNMP OK - ps1-b8-eqiad-infeed-load-tower-B-phase-Y 494 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:12] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:46:12] PROBLEM - ps1-c1-eqiad-infeed-load-tower-A-phase-Y on ps1-c1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:12] PROBLEM - ps1-d1-eqiad-infeed-load-tower-B-phase-Z on ps1-d1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:12] RECOVERY - ps1-b8-eqiad-infeed-load-tower-A-phase-Y on ps1-b8-eqiad is OK: SNMP OK - ps1-b8-eqiad-infeed-load-tower-A-phase-Y 492 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:14] RECOVERY - ps1-a1-eqiad-infeed-load-tower-B-phase-X on ps1-a1-eqiad is OK: SNMP OK - ps1-a1-eqiad-infeed-load-tower-B-phase-X 187 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:16] RECOVERY - ps1-c1-eqiad-infeed-load-tower-A-phase-Y on ps1-c1-eqiad is OK: SNMP OK - ps1-c1-eqiad-infeed-load-tower-A-phase-Y 0 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:20] PROBLEM - ps1-c1-eqiad-infeed-load-tower-B-phase-Z on ps1-c1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:22] PROBLEM - ps1-f8-eqiad-infeed-load-tower-A-phase-X on ps1-f8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:28] RECOVERY - ps1-e7-eqiad-infeed-load-tower-B-phase-Y on ps1-e7-eqiad is OK: SNMP OK - ps1-e7-eqiad-infeed-load-tower-B-phase-Y 540 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:46] PROBLEM - ps1-a4-eqiad-infeed-load-tower-B-phase-Z on ps1-a4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:50] RECOVERY - ps1-a4-eqiad-infeed-load-tower-B-phase-Z on ps1-a4-eqiad is OK: SNMP OK - ps1-a4-eqiad-infeed-load-tower-B-phase-Z 491 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:56] PROBLEM - ps1-d8-eqiad-infeed-load-tower-A-phase-Z on ps1-d8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:56] PROBLEM - ps1-c4-eqiad-infeed-load-tower-A-phase-Z on ps1-c4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:46:56] PROBLEM - ps1-e3-eqiad-infeed-load-tower-A-phase-Z on ps1-e3-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:00] RECOVERY - ps1-b5-eqiad-infeed-load-tower-A-phase-X on ps1-b5-eqiad is OK: SNMP OK - ps1-b5-eqiad-infeed-load-tower-A-phase-X 360 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:00] RECOVERY - ps1-e3-eqiad-infeed-load-tower-A-phase-Z on ps1-e3-eqiad is OK: SNMP OK - ps1-e3-eqiad-infeed-load-tower-A-phase-Z 556 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:00] RECOVERY - ps1-d8-eqiad-infeed-load-tower-A-phase-Z on ps1-d8-eqiad is OK: SNMP OK - ps1-d8-eqiad-infeed-load-tower-A-phase-Z 411 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:02] PROBLEM - ps1-a4-eqiad-infeed-load-tower-B-phase-X on ps1-a4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:08] PROBLEM - ps1-f7-eqiad-infeed-load-tower-B-phase-X on ps1-f7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:08] RECOVERY - ps1-a4-eqiad-infeed-load-tower-B-phase-X on ps1-a4-eqiad is OK: SNMP OK - ps1-a4-eqiad-infeed-load-tower-B-phase-X 563 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:08] PROBLEM - ps1-e9-eqiad-infeed-load-tower-B-phase-Z on ps1-e9-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:08] PROBLEM - ps1-c5-eqiad-infeed-load-tower-A-phase-Y on ps1-c5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:12] PROBLEM - ps1-c4-eqiad-infeed-load-tower-A-phase-Y on ps1-c4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:12] PROBLEM - ps1-a5-eqiad-infeed-load-tower-B-phase-Z on ps1-a5-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:12] PROBLEM - ps1-b4-eqiad-infeed-load-tower-B-phase-Y on ps1-b4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:14] RECOVERY - ps1-c1-eqiad-infeed-load-tower-B-phase-X on ps1-c1-eqiad is OK: SNMP OK - ps1-c1-eqiad-infeed-load-tower-B-phase-X 48 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:14] RECOVERY - ps1-e7-eqiad-infeed-load-tower-A-phase-X on ps1-e7-eqiad is OK: SNMP OK - ps1-e7-eqiad-infeed-load-tower-A-phase-X 590 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:14] RECOVERY - ps1-e9-eqiad-infeed-load-tower-B-phase-Z on ps1-e9-eqiad is OK: SNMP OK - ps1-e9-eqiad-infeed-load-tower-B-phase-Z 229 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:16] RECOVERY - ps1-d1-eqiad-infeed-load-tower-B-phase-Z on ps1-d1-eqiad is OK: SNMP OK - ps1-d1-eqiad-infeed-load-tower-B-phase-Z 535 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:16] RECOVERY - ps1-c4-eqiad-infeed-load-tower-A-phase-Y on ps1-c4-eqiad is OK: SNMP OK - ps1-c4-eqiad-infeed-load-tower-A-phase-Y 571 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:18] RECOVERY - ps1-e8-eqiad-infeed-load-tower-A-phase-X on ps1-e8-eqiad is OK: SNMP OK - ps1-e8-eqiad-infeed-load-tower-A-phase-X 569 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:28] RECOVERY - ps1-e3-eqiad-infeed-load-tower-A-phase-X on ps1-e3-eqiad is OK: SNMP OK - ps1-e3-eqiad-infeed-load-tower-A-phase-X 558 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:44] (03CR) 10Bking: [C:03+2] Deploy new edit/v4/per_editor endpoint to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359309 (https://phabricator.wikimedia.org/T434136) (owner: 10Snwachukwu) [13:47:46] PROBLEM - ps1-b8-eqiad-infeed-load-tower-A-phase-Z on ps1-b8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:48] PROBLEM - ps1-e11-eqiad-infeed-load-tower-B-phase-Y on ps1-e11-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:50] RECOVERY - ps1-b8-eqiad-infeed-load-tower-A-phase-Z on ps1-b8-eqiad is OK: SNMP OK - ps1-b8-eqiad-infeed-load-tower-A-phase-Z 356 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:54] PROBLEM - ps1-e12-eqiad-infeed-load-tower-A-phase-Z on ps1-e12-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:47:56] PROBLEM - ps1-b4-eqiad-infeed-load-tower-A-phase-X on ps1-b4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:48:00] RECOVERY - ps1-b4-eqiad-infeed-load-tower-A-phase-X on ps1-b4-eqiad is OK: SNMP OK - ps1-b4-eqiad-infeed-load-tower-A-phase-X 498 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:48:02] PROBLEM - ps1-a4-eqiad-infeed-load-tower-A-phase-Z on ps1-a4-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:48:08] PROBLEM - ps1-d1-eqiad-infeed-load-tower-A-phase-X on ps1-d1-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:48:12] PROBLEM - ps1-f7-eqiad-infeed-load-tower-A-phase-X on ps1-f7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:48:14] RECOVERY - ps1-c5-eqiad-infeed-load-tower-A-phase-Y on ps1-c5-eqiad is OK: SNMP OK - ps1-c5-eqiad-infeed-load-tower-A-phase-Y 482 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:48:14] RECOVERY - ps1-c4-eqiad-infeed-load-tower-B-phase-Z on ps1-c4-eqiad is OK: SNMP OK - ps1-c4-eqiad-infeed-load-tower-B-phase-Z 475 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:48:16] RECOVERY - ps1-a5-eqiad-infeed-load-tower-B-phase-Z on ps1-a5-eqiad is OK: SNMP OK - ps1-a5-eqiad-infeed-load-tower-B-phase-Z 434 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:48:16] RECOVERY - ps1-e4-eqiad-infeed-load-tower-A-phase-Y on ps1-e4-eqiad is OK: SNMP OK - ps1-e4-eqiad-infeed-load-tower-A-phase-Y 631 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:48:18] RECOVERY - ps1-f7-eqiad-infeed-load-tower-A-phase-X on ps1-f7-eqiad is OK: SNMP OK - ps1-f7-eqiad-infeed-load-tower-A-phase-X 467 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:48:22] PROBLEM - ps1-e7-eqiad-infeed-load-tower-A-phase-Y on ps1-e7-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:48:23] FIRING: [9x] CertAlmostExpired: gNMI TLS certificate for cloudsw1-d5-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:48:28] RECOVERY - ps1-e7-eqiad-infeed-load-tower-A-phase-Y on ps1-e7-eqiad is OK: SNMP OK - ps1-e7-eqiad-infeed-load-tower-A-phase-Y 524 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:48:36] PROBLEM - Juniper alarms on fasw2-e15a-eqiad is CRITICAL: JNX_ALARMS CRITICAL - No response from remote host 10.65.4.92 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [13:48:47] (03PS1) 10Ozge: ml-services: Add llm-batch service config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359842 (https://phabricator.wikimedia.org/T442150) [13:48:52] RECOVERY - ps1-e11-eqiad-infeed-load-tower-B-phase-Y on ps1-e11-eqiad is OK: SNMP OK - ps1-e11-eqiad-infeed-load-tower-B-phase-Y 317 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:48:52] RECOVERY - ps1-f7-eqiad-infeed-load-tower-B-phase-X on ps1-f7-eqiad is OK: SNMP OK - ps1-f7-eqiad-infeed-load-tower-B-phase-X 452 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:48:52] RECOVERY - ps1-d1-eqiad-infeed-load-tower-A-phase-X on ps1-d1-eqiad is OK: SNMP OK - ps1-d1-eqiad-infeed-load-tower-A-phase-X 499 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:48:54] RECOVERY - Host ps1-a2-eqiad is UP: PING WARNING - Packet loss = 33%, RTA = 1.42 ms [13:48:54] RECOVERY - Host ps1-b3-eqiad is UP: PING WARNING - Packet loss = 33%, RTA = 2.37 ms [13:48:54] RECOVERY - Host ps1-e10-eqiad is UP: PING WARNING - Packet loss = 33%, RTA = 2.17 ms [13:48:54] RECOVERY - Host ps1-e5-eqiad is UP: PING WARNING - Packet loss = 33%, RTA = 1.04 ms [13:48:54] RECOVERY - Host ps1-c3-eqiad is UP: PING WARNING - Packet loss = 33%, RTA = 1.42 ms [13:48:55] RECOVERY - Host ps1-d2-eqiad is UP: PING WARNING - Packet loss = 33%, RTA = 1.50 ms [13:48:55] RECOVERY - Host ps1-d3-eqiad is UP: PING WARNING - Packet loss = 33%, RTA = 1.15 ms [13:48:56] PROBLEM - ps1-e8-eqiad-infeed-load-tower-B-phase-Z on ps1-e8-eqiad is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:48:56] RECOVERY - Host ps1-e2-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.20 ms [13:48:57] RECOVERY - Host ps1-d7-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.23 ms [13:48:57] RECOVERY - Host ps1-d6-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.12 ms [13:48:58] RECOVERY - Host ps1-f1-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.35 ms [13:48:58] RECOVERY - Host ps1-a7-eqiad is UP: PING OK - Packet loss = 0%, RTA = 1.39 ms [13:49:26] RECOVERY - Juniper alarms on fasw2-e15a-eqiad is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [13:50:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:50:47] (03PS1) 10Jelto: etherpad: manage static apache with proper ensure flag [puppet] - 10https://gerrit.wikimedia.org/r/1359846 (https://phabricator.wikimedia.org/T442232) [13:53:06] (03CR) 10Jgiannelos: [C:03+1] osm: add a hack to send-tile-expiration-events.sh [puppet] - 10https://gerrit.wikimedia.org/r/1359841 (https://phabricator.wikimedia.org/T442715) (owner: 10Elukey) [13:53:22] RESOLVED: [3x] CertAlmostExpired: gNMI TLS certificate for cloudsw1-d5-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [13:55:41] (03CR) 10TChin: [C:03+1] deployment_server::deploy_service::service_catalog: Add eventgate [puppet] - 10https://gerrit.wikimedia.org/r/1359827 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [13:57:11] (03CR) 10Klausman: [V:03+2 C:03+2] ml-services: Add llm-batch service config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359842 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [13:57:51] (03PS2) 10Jforrester: wikifunctions: Expand check-wf-services.py to allow specifying a cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359336 [13:57:52] (03PS2) 10Jforrester: mathoid: Bump from 2026-05-12-175031-production to 2026-10-27-201041-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359397 [13:57:52] (03PS1) 10Jforrester: wikifunctions: Upgrade evaluators from 2026-10-14-185937 to 2026-10-28-122530 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359854 (https://phabricator.wikimedia.org/T433639) [13:58:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:58:05] (03PS1) 10Jforrester: wikifunctions: Upgrade orchestrator from 2026-10-14-161253 to 2026-10-28-013937 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359855 (https://phabricator.wikimedia.org/T357858) [13:58:25] (03PS1) 10Jgreen: Switch default fundraising bastion to codfw [dns] - 10https://gerrit.wikimedia.org/r/1359856 [13:58:37] (03Merged) 10jenkins-bot: ml-services: Add llm-batch service config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359842 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [14:00:05] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T1400) [14:00:21] (03PS2) 10Jelto: etherpad: manage static apache with proper ensure flag [puppet] - 10https://gerrit.wikimedia.org/r/1359846 (https://phabricator.wikimedia.org/T442232) [14:00:50] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . [14:00:52] (03CR) 10Jforrester: [C:03+2] wikifunctions: Expand check-wf-services.py to allow specifying a cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359336 (owner: 10Jforrester) [14:00:55] (03CR) 10Jforrester: [C:03+2] mathoid: Bump from 2026-05-12-175031-production to 2026-10-27-201041-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359397 (owner: 10Jforrester) [14:01:36] We have entered the datacenter switch-over maintenance window. Read-only mode is scheduled for 15:00 UTC [14:02:08] slyngs: Ah, should I not deploy the service fixes? [14:02:22] (03Merged) 10jenkins-bot: wikifunctions: Expand check-wf-services.py to allow specifying a cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359336 (owner: 10Jforrester) [14:02:23] Can you avoid it ? [14:02:27] (03Merged) 10jenkins-bot: mathoid: Bump from 2026-05-12-175031-production to 2026-10-27-201041-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359397 (owner: 10Jforrester) [14:02:44] (03CR) 10CI reject: [V:04-1] etherpad: manage static apache with proper ensure flag [puppet] - 10https://gerrit.wikimedia.org/r/1359846 (https://phabricator.wikimedia.org/T442232) (owner: 10Jelto) [14:02:53] slyngs: We can do the main deploy later today, but I’d really rather get the Mathoid fix out now. [14:03:21] Does it take less than an hour :-) [14:03:29] slyngs: Much much less. :-) [14:03:31] (03PS3) 10Jelto: etherpad: manage static apache with proper ensure flag [puppet] - 10https://gerrit.wikimedia.org/r/1359846 (https://phabricator.wikimedia.org/T442232) [14:03:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12484553 (10phaultfinder) [14:04:01] scap is locked though [14:04:11] Ack, this is services/helmfile. [14:04:15] James_F: send it [14:04:20] Okay, perfect, do your thing [14:04:30] !log jforrester@deploy1003 helmfile [staging] START helmfile.d/services/mathoid: apply [14:04:40] !log tchin@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply [14:04:41] (03PS4) 10Jelto: etherpad: manage static apache with proper ensure flag [puppet] - 10https://gerrit.wikimedia.org/r/1359846 (https://phabricator.wikimedia.org/T442232) [14:04:48] !log jforrester@deploy1003 helmfile [staging] DONE helmfile.d/services/mathoid: apply [14:04:56] !log jforrester@deploy1003 helmfile [codfw] START helmfile.d/services/mathoid: apply [14:05:02] does a scap lock affect the new scap service deployment thing? [14:05:05] (03PS2) 10Ottomata: [eventgate] Bump to v1.35.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359381 (https://phabricator.wikimedia.org/T442177) [14:05:11] !log tchin@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply [14:05:15] yes taavi [14:05:27] !log jforrester@deploy1003 helmfile [codfw] DONE helmfile.d/services/mathoid: apply [14:05:35] (03CR) 10Phuedx: [C:03+1] [eventgate] Bump to v1.35.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359381 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [14:05:45] !log jforrester@deploy1003 helmfile [eqiad] START helmfile.d/services/mathoid: apply [14:06:06] !log homer "lsw1-e3-codfw" commit "remove wikikube-worker2337 (T442714)" [14:06:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:06:13] T442714: hw troubleshooting: power-on for wikikube-worker2337.codfw.wmnet - https://phabricator.wikimedia.org/T442714 [14:06:13] !log jforrester@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mathoid: apply [14:06:22] Done. [14:07:59] (03CR) 10Jelto: "@dzahn@wikimedia.org we need something like that in the code for the offline page." [puppet] - 10https://gerrit.wikimedia.org/r/1359846 (https://phabricator.wikimedia.org/T442232) (owner: 10Jelto) [14:08:18] (03CR) 10Kamila Součková: alias deployment.codfw.wmnet to deploy2003 (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1359395 (https://phabricator.wikimedia.org/T426222) (owner: 10Kamila Součková) [14:09:52] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [14:10:21] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1359770 (https://phabricator.wikimedia.org/T442177) (owner: 10Jelto) [14:11:22] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1359761 (https://phabricator.wikimedia.org/T392465) (owner: 10Hashar) [14:11:34] (03CR) 10Btullis: [C:03+2] Remove the datasets-config DNS records [dns] - 10https://gerrit.wikimedia.org/r/1359814 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [14:11:35] (03CR) 10Jgreen: [C:03+2] Switch default fundraising bastion to codfw [dns] - 10https://gerrit.wikimedia.org/r/1359856 (owner: 10Jgreen) [14:11:49] (03CR) 10Muehlenhoff: "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1359762 (https://phabricator.wikimedia.org/T392465) (owner: 10Hashar) [14:11:53] !log btullis@dns1004 START - running authdns-update [14:12:09] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1359763 (https://phabricator.wikimedia.org/T392465) (owner: 10Hashar) [14:12:22] !log jgreen@dns2004 START - running authdns-update [14:13:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:13:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [14:14:37] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row A on cr2-eqiad - cmooney@cumin1004" [14:14:48] (03PS2) 10Jforrester: deployment_server::deploy_service::service_catalog: Run Wikifunctions checks [puppet] - 10https://gerrit.wikimedia.org/r/1359338 [14:14:53] (03PS3) 10Jforrester: deployment_server::deploy_service::service_catalog: Run Wikifunctions checks [puppet] - 10https://gerrit.wikimedia.org/r/1359338 [14:15:01] !log jgreen@dns2004 START - running authdns-update [14:15:08] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row A on cr2-eqiad - cmooney@cumin1004" [14:15:08] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [14:15:39] (03CR) 10Jelto: [C:03+1] "lgtm, thank you" [puppet] - 10https://gerrit.wikimedia.org/r/1359404 (https://phabricator.wikimedia.org/T439046) (owner: 10Majavah) [14:15:39] !log btullis@dns1004 START - running authdns-update [14:16:19] (03PS2) 10Klausman: hiera/k8s: Add pseudo-secrets stanza for llm-batch namespace on LW [labs/private] - 10https://gerrit.wikimedia.org/r/1359864 (https://phabricator.wikimedia.org/T442150) [14:16:29] (03CR) 10Klausman: [V:03+2 C:03+2] hiera/k8s: Add pseudo-secrets stanza for llm-batch namespace on LW [labs/private] - 10https://gerrit.wikimedia.org/r/1359864 (https://phabricator.wikimedia.org/T442150) (owner: 10Klausman) [14:16:30] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns5004 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 4248e59bac9894916bb6285fd7c1af428ebfd916, dns.git is 40804a6a885669d13c3a0e00551988712c602ea2) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [14:18:12] !log btullis@dns1004 END - running authdns-update [14:18:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [14:18:52] (03CR) 10Btullis: [C:03+2] dse-k8s: Remove the datasets-config helmfiles and namespaces [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359815 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [14:20:49] (03CR) 10Jelto: [C:03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1359338 (owner: 10Jforrester) [14:21:30] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns5004 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [14:21:40] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.reimage for host lvs5004.eqsin.wmnet with OS trixie [14:22:22] (03PS1) 10Cathal Mooney: cr1-eqiad: move row a sub-interfaces from ae1 to et-0/1/0 [homer/public] - 10https://gerrit.wikimedia.org/r/1359866 (https://phabricator.wikimedia.org/T441014) [14:22:41] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.dhcp for host lvs5004.eqsin.wmnet (push config only) [14:22:50] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5004.eqsin.wmnet (push config only) [14:24:59] (03Merged) 10jenkins-bot: dse-k8s: Remove the datasets-config helmfiles and namespaces [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359815 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [14:26:59] (03CR) 10Btullis: [C:03+1] deployment_server::deploy_service::service_catalog: Add eventgate [puppet] - 10https://gerrit.wikimedia.org/r/1359827 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [14:27:58] !log slyngshede@cumin1004 START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw [14:29:03] !log tchin@deploy1003 helmfile [staging] START helmfile.d/services/eventstreams: apply [14:29:30] !log tchin@deploy1003 helmfile [staging] DONE helmfile.d/services/eventstreams: apply [14:29:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12484708 (10phaultfinder) [14:30:05] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T1400) [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T1430) [14:30:41] !log ebysans@deploy1003 helmfile [staging] START helmfile.d/services/edit-analytics: apply [14:30:54] !log ebysans@deploy1003 helmfile [staging] DONE helmfile.d/services/edit-analytics: apply [14:30:55] !log tchin@deploy1003 helmfile [codfw] START helmfile.d/services/eventstreams: apply [14:31:34] !log tchin@deploy1003 helmfile [codfw] DONE helmfile.d/services/eventstreams: apply [14:33:41] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . [14:33:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [14:34:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12484753 (10phaultfinder) [14:37:07] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . [14:37:32] (03CR) 10Muehlenhoff: [C:03+1] "LGTM, one nit inline" [puppet] - 10https://gerrit.wikimedia.org/r/1359841 (https://phabricator.wikimedia.org/T442715) (owner: 10Elukey) [14:38:00] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.dhcp for host lvs5004.eqsin.wmnet (remove config only) [14:38:12] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5004.eqsin.wmnet (remove config only) [14:38:13] !log vgutierrez@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs5004.eqsin.wmnet with OS trixie [14:38:28] !log cmooney@cumin1004 DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 3:00:00 on asw2-b2-eqiad,cr[1-2]-eqiad,ssw1-a[1,8]-eqiad with reason: extending so down port alerts dont fire [14:39:23] (03PS1) 10Dpogorzelski: liftwing-studio: bump litellm to 1.100.4 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359869 [14:39:35] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: bump litellm to 1.100.4 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359869 (owner: 10Dpogorzelski) [14:39:54] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.reimage for host lvs5004.eqsin.wmnet with OS trixie [14:40:00] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on asw2-b-eqiad,cr[1-2]-eqiad,ssw1-a[1,8]-eqiad with reason: extending so down port alerts dont fire [14:40:05] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.dhcp for host lvs5004.eqsin.wmnet (push config only) [14:40:12] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12484789 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=bb0696e8-7843-44c6... [14:40:14] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5004.eqsin.wmnet (push config only) [14:40:34] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] liftwing-studio: bump litellm to 1.100.4 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359869 (owner: 10Dpogorzelski) [14:41:05] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: apply [14:41:52] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: apply [14:42:56] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . [14:43:46] !log slyngshede@cumin1004 END (ERROR) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=97) for datacenter switchover from eqiad to codfw [14:43:55] !log slyngshede@cumin1004 START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from eqiad to codfw [14:45:51] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://wikifeeds.svc.eqiad.wmnet:4101 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [14:49:10] (03CR) 10Btullis: deployment_server::deploy_service::service_catalog: Add eventgate-all (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359828 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [14:49:26] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) for datacenter switchover from eqiad to codfw [14:49:43] !log slyngshede@cumin1004 START - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from eqiad to codfw [14:49:58] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks (exit_code=0) for datacenter switchover from eqiad to codfw [14:50:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [14:50:51] RESOLVED: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://wikifeeds.svc.eqiad.wmnet:4101 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [14:52:29] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . [14:54:41] !log slyngshede@cumin1004 START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from eqiad to codfw [14:55:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [14:55:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [14:56:36] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) for datacenter switchover from eqiad to codfw [15:00:18] (03CR) 10Ahmon Dancy: Add a systemd timer to periodically drop the old MediaWiki images (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1359287 (https://phabricator.wikimedia.org/T437297) (owner: 10Elukey) [15:00:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [15:02:41] !log slyngshede@cumin1004 START - Cookbook sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from eqiad to codfw [15:02:41] !log slyngshede@cumin1004 MediaWiki read-only period starts at: 2026-10-28 15:02:41.444987 [15:02:48] slyngshede@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [15:02:59] i wonder why :D ah the classic [15:03:05] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) for datacenter switchover from eqiad to codfw [15:03:06] slyngshede@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [15:03:11] !log slyngshede@cumin1004 START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly for datacenter switchover from eqiad to codfw [15:03:13] slyngshede@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [15:03:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [15:03:41] 10SRE-Access-Requests: Add FIDO-backed SSH keys for alexsanford - https://phabricator.wikimedia.org/T442725 (10ASanford-WMF) 03NEW [15:04:06] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) for datacenter switchover from eqiad to codfw [15:04:07] slyngshede@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [15:04:12] !log slyngshede@cumin1004 START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki for datacenter switchover from eqiad to codfw [15:04:14] slyngshede@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [15:04:45] (03PS1) 10Snwachukwu: Edit-analytics: Deploy new edit/v4/per_editor endpoint to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359872 (https://phabricator.wikimedia.org/T434136) [15:05:05] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) for datacenter switchover from eqiad to codfw [15:05:07] slyngshede@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [15:05:13] !log slyngshede@cumin1004 START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite for datacenter switchover from eqiad to codfw [15:05:14] slyngshede@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [15:05:17] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) for datacenter switchover from eqiad to codfw [15:05:18] slyngshede@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [15:05:23] !log slyngshede@cumin1004 START - Cookbook sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from eqiad to codfw [15:05:25] slyngshede@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [15:05:31] !log slyngshede@cumin1004 MediaWiki read-only period ends at: 2026-10-28 15:05:31.302917 [15:05:31] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) for datacenter switchover from eqiad to codfw [15:05:34] (03PS1) 10Alex.sanford: Update SSH keys for alexsanford [puppet] - 10https://gerrit.wikimedia.org/r/1359873 (https://phabricator.wikimedia.org/T442725) [15:06:12] !log slyngshede@cumin1004 START - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from eqiad to codfw [15:06:13] !log root@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-jobrunner: sync [15:06:30] !log root@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-jobrunner: sync [15:06:31] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner (exit_code=0) for datacenter switchover from eqiad to codfw [15:06:42] !log slyngshede@cumin1004 START - Cookbook sre.switchdc.mediawiki.08-start-maintenance for datacenter switchover from eqiad to codfw [15:06:43] !log root@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-cron: apply [15:07:09] 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for alexsanford - https://phabricator.wikimedia.org/T442725#12484936 (10ASanford-WMF) [15:07:15] FIRING: [4x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [15:07:36] !log root@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply [15:08:01] total RO time of 02:49.857930 [15:08:24] !log root@deploy1003 helmfile [codfw] START helmfile.d/services/mw-cron: apply [15:08:29] (03CR) 10Brouberol: [C:03+1] "Looks good!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359872 (https://phabricator.wikimedia.org/T434136) (owner: 10Snwachukwu) [15:08:45] (03PS1) 10Btullis: dse-k8s-wdqs: Add dse-k8s-wdqs1004 and dse-k8s-wdqs2005 [puppet] - 10https://gerrit.wikimedia.org/r/1359874 (https://phabricator.wikimedia.org/T440436) [15:09:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12484946 (10phaultfinder) [15:09:53] !log root@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-cron: apply [15:10:13] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) for datacenter switchover from eqiad to codfw [15:10:29] (03CR) 10Btullis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1359874 (https://phabricator.wikimedia.org/T440436) (owner: 10Btullis) [15:10:30] !log slyngshede@cumin1004 START - Cookbook sre.switchdc.mediawiki.09-restore-ttl for datacenter switchover from eqiad to codfw [15:10:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [15:10:52] (03CR) 10Slyngshede: [C:03+2] wmnet: update CNAME records for DB masters to codfw [dns] - 10https://gerrit.wikimedia.org/r/1357620 (https://phabricator.wikimedia.org/T439214) (owner: 10Slyngshede) [15:10:57] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.switchdc.mediawiki.09-restore-ttl (exit_code=0) for datacenter switchover from eqiad to codfw [15:11:20] !log slyngshede@dns1004 START - running authdns-update [15:11:43] (03CR) 10Snwachukwu: [C:03+2] Edit-analytics: Deploy new edit/v4/per_editor endpoint to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359872 (https://phabricator.wikimedia.org/T434136) (owner: 10Snwachukwu) [15:12:08] (03CR) 10Elukey: Add a systemd timer to periodically drop the old MediaWiki images (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1359287 (https://phabricator.wikimedia.org/T437297) (owner: 10Elukey) [15:12:15] RESOLVED: [4x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [15:12:34] (03CR) 10Snwachukwu: [V:03+2 C:03+2] Edit-analytics: Deploy new edit/v4/per_editor endpoint to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359872 (https://phabricator.wikimedia.org/T434136) (owner: 10Snwachukwu) [15:13:55] !log slyngshede@dns1004 END - running authdns-update [15:14:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12484975 (10phaultfinder) [15:14:43] (03CR) 10Ahmon Dancy: Add a systemd timer to periodically drop the old MediaWiki images (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359287 (https://phabricator.wikimedia.org/T437297) (owner: 10Elukey) [15:16:03] !log slyngshede@cumin1004 START - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters for datacenter switchover from eqiad to codfw [15:16:39] (03CR) 10Slyngshede: [C:03+2] geo-maps: update map default to list codfw first [dns] - 10https://gerrit.wikimedia.org/r/1319816 (https://phabricator.wikimedia.org/T439674) (owner: 10Slyngshede) [15:16:47] !log jgiannelos@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: apply [15:16:48] !log slyngshede@dns1004 START - running authdns-update [15:17:01] !log jgiannelos@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [15:17:47] (03CR) 10Brouberol: [C:03+1] "LG!" [puppet] - 10https://gerrit.wikimedia.org/r/1359874 (https://phabricator.wikimedia.org/T440436) (owner: 10Btullis) [15:18:26] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12484989 (10cmooney) [15:19:00] FIRING: [2x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-wdqs1004 has a BGP session which is not in the 'established' state. [15:19:21] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: power-on failure for druid-internal1004.eqiad.wmnet - https://phabricator.wikimedia.org/T441866#12484991 (10VRiley-WMF) They have generated a new ticket for us. The new ticket is Case #CS-00218577 [15:19:38] !log slyngshede@dns1004 END - running authdns-update [15:20:40] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [15:21:00] (03PS7) 10Jdlrobson: Enable Vector ReadingList CTA on group 1 wikipedias+mw+wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359380 (https://phabricator.wikimedia.org/T441705) [15:21:20] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [15:21:30] !log ebysans@deploy1003 helmfile [eqiad] START helmfile.d/services/edit-analytics: apply [15:21:41] !log tchin@deploy1003 helmfile [codfw] START helmfile.d/services/eventstreams: apply [15:21:43] !log ebysans@deploy1003 helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply [15:22:07] !log ebysans@deploy1003 helmfile [codfw] START helmfile.d/services/edit-analytics: apply [15:22:22] !log ebysans@deploy1003 helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply [15:22:35] !log tchin@deploy1003 helmfile [codfw] DONE helmfile.d/services/eventstreams: apply [15:22:46] !log tchin@deploy1003 helmfile [codfw] START helmfile.d/services/eventstreams: apply [15:23:07] !log tchin@deploy1003 helmfile [codfw] DONE helmfile.d/services/eventstreams: apply [15:23:12] (03PS4) 10Krinkle: MathML+MathJax rollout to phase 6 (remaining large Wikipedias) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311061 (https://phabricator.wikimedia.org/T271001) [15:24:55] (03PS1) 10Ahmon Dancy: registryctl: accept relative values for --older-than [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1359879 (https://phabricator.wikimedia.org/T437297) [15:27:17] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.dhcp for host lvs5004.eqsin.wmnet (remove config only) [15:27:29] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs5004.eqsin.wmnet (remove config only) [15:27:43] (03CR) 10Btullis: [C:03+2] Remove the datasets-config chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359816 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [15:28:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [15:28:32] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters (exit_code=0) for datacenter switchover from eqiad to codfw [15:28:44] 06SRE, 06Fundraising-Backlog, 10FY26-27 ST6.4 - Logged-in performance, 10MediaWiki-extensions-CentralNotice, and 3 others: Special:HideBanners is not really cacheable - https://phabricator.wikimedia.org/T256447#12485032 (10Krinkle) [15:28:54] !log slyngshede@cumin1004 START - Cookbook sre.switchdc.mediawiki.09-unlock-scap for datacenter switchover from eqiad to codfw [15:28:56] !log root@deploy1003 Forcefully removed global lock: Datacenter switchover from eqiad to codfw - [15:28:56] !log root@deploy1003 Unlocked for deployment [ALL REPOSITORIES]: Datacenter switchover from eqiad to codfw - (duration: 126m 08s) [15:28:56] !log slyngshede@cumin1004 END (PASS) - Cookbook sre.switchdc.mediawiki.09-unlock-scap (exit_code=0) for datacenter switchover from eqiad to codfw [15:29:05] (03CR) 10TrainBranchBot: [C:03+2] "Approved by slyngshede@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319818 (https://phabricator.wikimedia.org/T439674) (owner: 10Slyngshede) [15:29:16] (03Merged) 10jenkins-bot: Remove the datasets-config chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359816 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [15:29:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12485038 (10phaultfinder) [15:29:53] (03Merged) 10jenkins-bot: debug.json: order codfw (primary) DC backends first [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1319818 (https://phabricator.wikimedia.org/T439674) (owner: 10Slyngshede) [15:30:08] (03CR) 10JMeybohm: [C:04-1] gateway-api-crds: add chart with the Gateway API CRDs (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357212 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [15:30:21] !log slyngshede@deploy1003 Started scap sync-world: Backport for [[gerrit:1319818|debug.json: order codfw (primary) DC backends first (T439674)]] [15:30:26] T439674: 🎃 Southward Datacenter Switchover (October 2026) - https://phabricator.wikimedia.org/T439674 [15:31:08] !log jgiannelos@deploy1003 helmfile [staging] START helmfile.d/services/thumbor: sync [15:31:12] !log jgiannelos@deploy1003 helmfile [staging] DONE helmfile.d/services/thumbor: sync [15:31:25] (03CR) 10Btullis: [C:03+2] dse-k8s-wdqs: Add dse-k8s-wdqs1004 and dse-k8s-wdqs2005 [puppet] - 10https://gerrit.wikimedia.org/r/1359874 (https://phabricator.wikimedia.org/T440436) (owner: 10Btullis) [15:31:56] !log vgutierrez@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on lvs5004.eqsin.wmnet with reason: host reimage [15:34:00] FIRING: [4x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-wdqs1004 has a BGP session which is not in the 'established' state. [15:34:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [15:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:34:48] !log slyngshede@deploy1003 slyngshede: Backport for [[gerrit:1319818|debug.json: order codfw (primary) DC backends first (T439674)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:35:04] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs5004.eqsin.wmnet with reason: host reimage [15:35:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [15:35:58] !log slyngshede@deploy1003 slyngshede: Continuing with deployment [15:36:44] (03CR) 10Fabian Kaelin: "The agent only prepared the CR; I created the key myself (sandboxed agent can't even access ~/.ssh)." [puppet] - 10https://gerrit.wikimedia.org/r/1359804 (https://phabricator.wikimedia.org/T442665) (owner: 10Fabian Kaelin) [15:39:13] (03PS1) 10SBassett: DO NOT MERGE: Test Change Set [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359887 [15:39:17] (03CR) 10Daniel Kinzler: "I support the intent, but I can't say anything about the implementation." [puppet] - 10https://gerrit.wikimedia.org/r/1354903 (https://phabricator.wikimedia.org/T438209) (owner: 10Aghirelli) [15:39:20] (03PS4) 10Aleksandar Mastilovic: blunderbuss: Allow egress to the analytics-test Hadoop cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358276 (https://phabricator.wikimedia.org/T442190) [15:39:27] (03CR) 10SBassett: [C:04-2] "DO NOT MERGE" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359887 (owner: 10SBassett) [15:39:59] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, October 28 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359887 (owner: 10SBassett) [15:40:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [15:41:41] !log slyngshede@deploy1003 Finished scap sync-world: Backport for [[gerrit:1319818|debug.json: order codfw (primary) DC backends first (T439674)]] (duration: 11m 20s) [15:41:47] T439674: 🎃 Southward Datacenter Switchover (October 2026) - https://phabricator.wikimedia.org/T439674 [15:41:58] (03PS5) 10Aleksandar Mastilovic: blunderbuss: Allow egress to the analytics-test Hadoop cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358276 (https://phabricator.wikimedia.org/T442190) [15:42:05] (03CR) 10Aleksandar Mastilovic: blunderbuss: Allow egress to the analytics-test Hadoop cluster (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358276 (https://phabricator.wikimedia.org/T442190) (owner: 10Aleksandar Mastilovic) [15:44:12] (03PS2) 10Elukey: osm: add a hack to send-tile-expiration-events.sh [puppet] - 10https://gerrit.wikimedia.org/r/1359841 (https://phabricator.wikimedia.org/T442715) [15:44:13] DC switch over has completed [15:44:21] <3 [15:44:23] <3 [15:45:03] (03CR) 10Brouberol: [C:03+1] blunderbuss: Allow egress to the analytics-test Hadoop cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358276 (https://phabricator.wikimedia.org/T442190) (owner: 10Aleksandar Mastilovic) [15:45:08] 🛫 🛬 [15:45:13] (03CR) 10Majavah: [C:03+2] gitlab_runner: Use ferm::join_hosts to resolve host names [puppet] - 10https://gerrit.wikimedia.org/r/1359404 (https://phabricator.wikimedia.org/T439046) (owner: 10Majavah) [15:48:12] (03CR) 10JHathaway: [C:03+2] mx: use production, rather than aggregate network list [puppet] - 10https://gerrit.wikimedia.org/r/1358850 (https://phabricator.wikimedia.org/T442063) (owner: 10JHathaway) [15:49:51] (03CR) 10Elukey: [C:03+2] osm: add a hack to send-tile-expiration-events.sh [puppet] - 10https://gerrit.wikimedia.org/r/1359841 (https://phabricator.wikimedia.org/T442715) (owner: 10Elukey) [15:50:18] 10SRE-swift-storage, 10Cloud-VPS (Debian Bullseye Deprecation): Migrate swift away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435783#12485164 (10komla) 05Open→03Resolved I've marked this as closed. Thanks! [15:51:41] (03CR) 10Brouberol: [C:03+2] blunderbuss: Allow egress to the analytics-test Hadoop cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358276 (https://phabricator.wikimedia.org/T442190) (owner: 10Aleksandar Mastilovic) [15:52:21] (03CR) 10Volans: [C:03+1] "LGTM, one question inline" [puppet] - 10https://gerrit.wikimedia.org/r/1359781 (https://phabricator.wikimedia.org/T442690) (owner: 10Filippo Giunchedi) [15:52:59] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply [15:54:21] !log amastilovic@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply [15:56:42] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs5004.eqsin.wmnet with OS trixie [15:59:00] RESOLVED: [4x] NodeBGPSessionStatusNotEstablished: Kubernetes node dse-k8s-wdqs1004 has a BGP session which is not in the 'established' state. [15:59:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [16:00:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [16:03:42] FIRING: JobUnavailable: Reduced availability for job bacula in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:06:48] !log brouberol@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [16:07:36] !log installing apache2 security updates [16:07:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:07:44] !log brouberol@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [16:08:18] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12485244 (10Milimetric) [16:11:04] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs2005.codfw.wmnet [16:12:08] (03PS1) 10Vgutierrez: Revert "hieradata: Depool lvs5004 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1359895 [16:12:23] (03PS2) 10Vgutierrez: Revert "hieradata: Depool lvs5004 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1359895 (https://phabricator.wikimedia.org/T401832) [16:13:42] FIRING: [3x] JobUnavailable: Reduced availability for job bacula in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:13:48] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1359895 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [16:14:08] (03CR) 10Ssingh: [C:03+1] Revert "hieradata: Depool lvs5004 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1359895 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [16:15:34] (03PS1) 10Ozge: ml-services: Temporarily set minReplicas to 1 for llm-batch services [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359896 (https://phabricator.wikimedia.org/T442150) [16:16:26] (03CR) 10Vgutierrez: [C:03+2] Revert "hieradata: Depool lvs5004 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1359895 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [16:16:46] (03CR) 10Kevin Bazira: [C:03+1] ml-services: Temporarily set minReplicas to 1 for llm-batch services [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359896 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [16:16:48] (03CR) 10Dduvall: "@dzahn@wikimedia.org do you think we're safe to move forward with this?" [puppet] - 10https://gerrit.wikimedia.org/r/1356455 (https://phabricator.wikimedia.org/T440670) (owner: 10Dduvall) [16:17:23] (03CR) 10Ozge: [V:03+2 C:03+2] ml-services: Temporarily set minReplicas to 1 for llm-batch services [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359896 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [16:17:45] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs2005.codfw.wmnet [16:18:42] FIRING: [3x] JobUnavailable: Reduced availability for job bacula in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:19:12] (03Merged) 10jenkins-bot: ml-services: Temporarily set minReplicas to 1 for llm-batch services [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359896 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [16:19:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12485383 (10phaultfinder) [16:19:44] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-wdqs1004.eqiad.wmnet [16:19:47] !log vgutierrez@cumin2003 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs5004.*} and A:liberica [16:20:10] !log vgutierrez@cumin2003 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs5004.*} and A:liberica [16:20:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [16:21:16] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . [16:23:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [16:24:13] !log jasmine@cumin2003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-worker-eqiad [16:24:41] (03PS2) 10Aqu: airflow: fold raw PVC releases into chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359340 (https://phabricator.wikimedia.org/T433696) [16:24:48] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1043].eqiad.wmnet [16:26:19] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-wdqs1004.eqiad.wmnet [16:26:31] (03PS3) 10Aqu: airflow: fold raw PVC releases into chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359340 (https://phabricator.wikimedia.org/T433696) [16:27:41] (03Abandoned) 10Aqu: airflow: add CephFS PVCs dev-metrics-git-pipeline [deployment-charts] - 10https://gerrit.wikimedia.org/r/1348762 (https://phabricator.wikimedia.org/T433696) (owner: 10Aqu) [16:32:40] (03PS1) 10Ozge: ml-services: set minReplicas back to 0 for llm-batch services [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359903 (https://phabricator.wikimedia.org/T442150) [16:33:27] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1043].eqiad.wmnet [16:33:40] (03CR) 10Ozge: [V:03+2 C:03+2] ml-services: set minReplicas back to 0 for llm-batch services [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359903 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [16:34:21] 10SRE-swift-storage: skwiki image table contains non existing image - https://phabricator.wikimedia.org/T72896#12485479 (10Ladsgroup) 05Open→03Resolved Got fixed some time in the past 12 years [16:34:53] (03Merged) 10jenkins-bot: ml-services: set minReplicas back to 0 for llm-batch services [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359903 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [16:35:13] 06SRE, 10SRE-Access-Requests, 06Data-Engineering: Requesting Kerberos identity for rsilvola - https://phabricator.wikimedia.org/T442695#12485483 (10Milimetric) This is a follow-up to {T411624}, moving to our radar and tagging SRE. [16:36:18] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359904 [16:36:21] (03PS1) 10Dbrant: push-notifications: update to 2026-10-28-153049 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359905 [16:37:00] !log ozge@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' . [16:38:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [16:38:34] (03CR) 10Kamila Součková: [C:03+2] alias deployment.codfw.wmnet to deploy2003 [dns] - 10https://gerrit.wikimedia.org/r/1359395 (https://phabricator.wikimedia.org/T426222) (owner: 10Kamila Součková) [16:39:32] !log kamila@dns7001 START - running authdns-update [16:39:38] (03CR) 10Brouberol: [C:03+2] airflow: fold raw PVC releases into chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359340 (https://phabricator.wikimedia.org/T433696) (owner: 10Aqu) [16:39:48] FIRING: PuppetFailure: Puppet has failed on build2004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [16:41:15] !log kamila@cumin1004 START - Cookbook sre.hosts.decommission for hosts deploy2002.codfw.wmnet [16:41:57] (03PS1) 10Clément Goubert: php/8.3: update to 8.3.36-1 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1359909 [16:42:00] !log kamila@dns7001 END - running authdns-update [16:42:19] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1043].eqiad.wmnet [16:42:29] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1006-1007,1015-1016,1021,1034-1043].eqiad.wmnet [16:43:01] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1044-1057,1064].eqiad.wmnet [16:43:19] FIRING: [11x] KubernetesCalicoDown: wikikube-worker1006.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:43:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [16:43:46] (03PS1) 10Brouberol: airflow: bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359911 (https://phabricator.wikimedia.org/T433696) [16:44:56] (03CR) 10Elukey: [C:03+1] "Really nice!" [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1359879 (https://phabricator.wikimedia.org/T437297) (owner: 10Ahmon Dancy) [16:45:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [16:45:37] (03CR) 10Aqu: [C:03+1] "Thanks" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359911 (https://phabricator.wikimedia.org/T433696) (owner: 10Brouberol) [16:46:19] !log kamila@cumin1004 START - Cookbook sre.dns.netbox [16:46:24] (03PS6) 10Dpogorzelski: gateway-api-crds: add chart with the Gateway API CRDs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357212 (https://phabricator.wikimedia.org/T436650) [16:46:24] (03PS2) 10Dpogorzelski: admin_ng: install the Gateway API CRDs on ml-staging-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358200 (https://phabricator.wikimedia.org/T436650) [16:46:24] (03PS2) 10Dpogorzelski: istio/ml-serve: add an overlay enabling the Inference Extension [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358201 (https://phabricator.wikimedia.org/T436650) [16:46:24] (03PS2) 10Dpogorzelski: kserve-llmisvc-resources: install the Inference Extension CRDs on ml-staging-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358202 (https://phabricator.wikimedia.org/T436650) [16:46:37] (03CR) 10Brouberol: [C:03+2] airflow: bump version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359911 (https://phabricator.wikimedia.org/T433696) (owner: 10Brouberol) [16:47:16] (03CR) 10RLazarus: [V:03+2 C:03+1] "`" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1359909 (owner: 10Clément Goubert) [16:49:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12485579 (10phaultfinder) [16:49:55] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12485590 (10phaultfinder) [16:50:05] (03CR) 10Dpogorzelski: gateway-api-crds: add chart with the Gateway API CRDs (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357212 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [16:50:15] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply [16:50:23] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply [16:50:55] !log kamila@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: deploy2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - kamila@cumin1004" [16:51:28] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply [16:51:36] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply [16:52:18] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply [16:52:30] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply [16:53:01] !log btullis@puppetserver1001 conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-wdqs1004.eqiad.wmnet [16:53:08] !log btullis@puppetserver1001 conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-wdqs1004.eqiad.wmnet [16:53:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [16:53:29] !log btullis@puppetserver1001 conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-wdqs2005.codfw.wmnet [16:53:35] !log btullis@puppetserver1001 conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-wdqs2005.codfw.wmnet [16:53:59] kamila@cumin1004 decommission (PID 3604340) is awaiting input [16:54:32] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply [16:54:37] !log kamila@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: deploy2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - kamila@cumin1004" [16:54:37] !log kamila@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [16:54:38] !log kamila@cumin1004 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts deploy2002.codfw.wmnet [16:54:43] !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply [16:54:55] 10ops-codfw, 06SRE, 06DC-Ops, 10procurement, and 2 others: decommission deploy2002.codfw.wmnet - https://phabricator.wikimedia.org/T426222#12485633 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by kamila@cumin1004 for hosts: `deploy2002.codfw.wmnet` - deploy2002.codfw.wmnet (**PASS**)... [16:55:33] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: Tab completion for cookbook names - https://phabricator.wikimedia.org/T367230#12485635 (10Lizimedia) to summarize our discussion from the most recent SRE Tooling and Automation Office Hours: Whenever a change to the cookbooks repo... [16:56:02] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1044-1057,1064].eqiad.wmnet [16:57:31] 10ops-codfw, 06SRE, 06DC-Ops, 10procurement, and 2 others: decommission deploy2002.codfw.wmnet - https://phabricator.wikimedia.org/T426222#12485661 (10Raine) a:05Raine→03None [16:59:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST pods) on k8s@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T1700) [17:02:44] o/ [17:04:04] (03CR) 10RLazarus: [C:03+2] mw-debug: Update to Envoy v1.40.1 for validation [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359439 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [17:04:19] 10SRE-swift-storage: IPv6 records inconsistent on the ms-be hosts - https://phabricator.wikimedia.org/T320947#12485715 (10Ladsgroup) >>! In T320947#10604227, @MatthewVernon wrote: > I expect it to be refreshed in Q1 or maybe Q2 (purchase date was 2020-08-11). They all have been decommissioned. Shall we close th... [17:04:31] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1044-1057,1064].eqiad.wmnet [17:04:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12485716 (10phaultfinder) [17:04:42] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1044-1057,1064].eqiad.wmnet [17:04:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST pods) on k8s@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [17:05:13] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1065-1079].eqiad.wmnet [17:06:04] (03Merged) 10jenkins-bot: mw-debug: Update to Envoy v1.40.1 for validation [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359439 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [17:06:04] FIRING: [4x] KubernetesCalicoDown: wikikube-worker1044.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:07:01] (03CR) 10Brennen Bearnes: [C:03+1] "I hadn't realized this didn't already restart. Thanks for fix." [puppet] - 10https://gerrit.wikimedia.org/r/1359770 (https://phabricator.wikimedia.org/T442177) (owner: 10Jelto) [17:08:30] !log rzl@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-debug: apply [17:09:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [17:10:54] !log rzl@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [17:11:28] !log rzl@deploy1003 helmfile [codfw] START helmfile.d/services/mw-debug: apply [17:13:43] !log rzl@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [17:13:48] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1065-1079].eqiad.wmnet [17:14:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [17:14:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [17:16:51] 10SRE-swift-storage: IPv6 records inconsistent on the ms-be hosts - https://phabricator.wikimedia.org/T320947#12485753 (10MatthewVernon) 05Open→03Resolved a:03MatthewVernon >>! In T320947#12485715, @Ladsgroup wrote: > They all have been decommissioned. Shall we close this ticket? Yes :) [17:19:44] RESOLVED: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [17:20:41] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [17:22:33] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1065-1079].eqiad.wmnet [17:22:43] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1065-1079].eqiad.wmnet [17:23:15] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1080-1081,1084-1087,1093-1095,1113-1118].eqiad.wmnet [17:23:19] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1073.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:24:49] PROBLEM - PyBal connections to etcd on lvs1019 is CRITICAL: CRITICAL: 62 connections established with conf1007.eqiad.wmnet:4001 (min=81) https://wikitech.wikimedia.org/wiki/PyBal [17:26:49] (03CR) 10RLazarus: [C:03+2] shellbox-syntaxhighlight: Update to Envoy 1.40.1 for validation [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359430 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [17:26:50] (03CR) 10RLazarus: [C:03+2] rest-gateway: Update to Envoy v1.40.1 in staging for validation [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359438 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [17:28:16] (03Merged) 10jenkins-bot: shellbox-syntaxhighlight: Update to Envoy 1.40.1 for validation [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359430 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [17:28:30] (03Merged) 10jenkins-bot: rest-gateway: Update to Envoy v1.40.1 in staging for validation [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359438 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [17:29:39] !log rzl@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply [17:29:49] RECOVERY - PyBal connections to etcd on lvs1019 is OK: OK: 81 connections established with conf1007.eqiad.wmnet:4001 (min=81) https://wikitech.wikimedia.org/wiki/PyBal [17:29:53] !log rzl@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [17:32:56] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1080-1081,1084-1087,1093-1095,1113-1118].eqiad.wmnet [17:33:49] !log rzl@deploy1003 helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply [17:34:11] !log rzl@deploy1003 helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [17:35:13] !log rzl@deploy1003 helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply [17:35:23] (03CR) 10Jelto: [V:03+1 C:03+2] phabricator: auto restart phd service [puppet] - 10https://gerrit.wikimedia.org/r/1359770 (https://phabricator.wikimedia.org/T442177) (owner: 10Jelto) [17:35:27] 10SRE-Access-Requests: Add FIDO-backed SSH keys for xcollazo - https://phabricator.wikimedia.org/T442741 (10xcollazo) 03NEW [17:35:36] !log rzl@deploy1003 helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [17:36:31] 10SRE-Access-Requests: Add FIDO-backed SSH keys for xcollazo - https://phabricator.wikimedia.org/T442741#12485837 (10xcollazo) At this time I will only be adding one key, and not removing my non-FIDO keys. Once I have a spare YubiKey I can remove the non-FIDO one. [17:38:23] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade rubocop ruby linter on operations/puppet - https://phabricator.wikimedia.org/T441847#12485842 (10hashar) [17:40:23] PROBLEM - VRRP status on cr1-eqiad is CRITICAL: VRRP CRITICAL - 8 misconfigured interfaces, 0 inconsistent interfaces https://wikitech.wikimedia.org/wiki/Network_monitoring%23VRRP_status [17:43:13] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1080-1081,1084-1087,1093-1095,1113-1118].eqiad.wmnet [17:43:19] FIRING: [7x] KubernetesCalicoDown: wikikube-worker1084.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:43:24] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1080-1081,1084-1087,1093-1095,1113-1118].eqiad.wmnet [17:43:27] (03PS1) 10RLazarus: rest-gateway: Update to Envoy v1.40.1 in staging, properly [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359923 (https://phabricator.wikimedia.org/T436338) [17:43:55] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1119-1133].eqiad.wmnet [17:44:17] (03PS1) 10Xcollazo: admin: Add one FIDO key for xcollazo [puppet] - 10https://gerrit.wikimedia.org/r/1359924 (https://phabricator.wikimedia.org/T442741) [17:44:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12485890 (10phaultfinder) [17:45:15] !log aqu@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply [17:45:45] !log aqu@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply [17:51:38] (03CR) 10Ssingh: [C:04-1] "Thanks for sharing that. So since a sandboxed agent was used and ~/.ssh was blocked, what is the workflow for copying the public key by th" [puppet] - 10https://gerrit.wikimedia.org/r/1359804 (https://phabricator.wikimedia.org/T442665) (owner: 10Fabian Kaelin) [17:52:57] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1119-1133].eqiad.wmnet [17:54:40] (03CR) 10Subramanya Sastry: [C:03+1] Turn off "report a visual" bug on wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358832 (https://phabricator.wikimedia.org/T430970) (owner: 10Arlolra) [17:54:56] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, October 28 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358832 (https://phabricator.wikimedia.org/T430970) (owner: 10Arlolra) [17:56:36] !log tchin@deploy1003 helmfile [codfw] START helmfile.d/services/eventstreams: apply [17:58:09] (03CR) 10JMeybohm: [C:03+1] rest-gateway: Update to Envoy v1.40.1 in staging, properly [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359923 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [17:58:50] (03CR) 10Jelto: [C:03+2] deployment_server::deploy_service::service_catalog: Run Wikifunctions checks [puppet] - 10https://gerrit.wikimedia.org/r/1359338 (owner: 10Jforrester) [18:00:05] dduvall and dancy: May I have your attention please! MediaWiki train - Utc-7 Version. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T1800) [18:00:09] 10SRE-Access-Requests: Request IRC #mediawiki_security access for Novem Linguae - https://phabricator.wikimedia.org/T442747 (10Novem_Linguae) 03NEW [18:00:57] o/ I'm lingering and making tea [18:01:04] FIRING: [14x] KubernetesCalicoDown: wikikube-worker1119.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:01:34] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1119-1133].eqiad.wmnet [18:01:44] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1119-1133].eqiad.wmnet [18:02:16] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1134-1148].eqiad.wmnet [18:04:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12486019 (10phaultfinder) [18:07:03] !log tchin@deploy1003 helmfile [codfw] DONE helmfile.d/services/eventstreams: apply [18:10:45] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1134-1148].eqiad.wmnet [18:10:59] !log tchin@deploy1003 helmfile [codfw] START helmfile.d/services/eventstreams: apply [18:12:55] (03CR) 10RLazarus: [C:03+2] rest-gateway: Update to Envoy v1.40.1 in staging, properly [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359923 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [18:13:03] (03PS1) 10Eric Gardner: ShareCard: Show page views as a monthly rate [extensions/WikimediaCustomizations] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359934 (https://phabricator.wikimedia.org/T439164) [18:13:15] (03PS2) 10Eric Gardner: ShareCard: Show page views as a monthly rate [extensions/WikimediaCustomizations] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359934 (https://phabricator.wikimedia.org/T439164) [18:14:09] (03PS1) 10TrainBranchBot: group1 to 1.48.0-wmf.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359936 (https://phabricator.wikimedia.org/T438223) [18:14:12] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by dduvall@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359936 (https://phabricator.wikimedia.org/T438223) (owner: 10TrainBranchBot) [18:14:20] (03PS1) 10Eric Gardner: ShareCard: Hide images on the bad image list [extensions/WikimediaCustomizations] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359937 (https://phabricator.wikimedia.org/T439161) [18:14:39] (03Merged) 10jenkins-bot: rest-gateway: Update to Envoy v1.40.1 in staging, properly [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359923 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [18:14:57] FIRING: ProbeDown: Service mw-web-next:4454 has failed probes (http_mw-web-next_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-web-next:4454 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:15:07] (03Merged) 10jenkins-bot: group1 to 1.48.0-wmf.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359936 (https://phabricator.wikimedia.org/T438223) (owner: 10TrainBranchBot) [18:17:01] (03CR) 10Jelto: "one comment in-line" [puppet] - 10https://gerrit.wikimedia.org/r/1359776 (https://phabricator.wikimedia.org/T442341) (owner: 10JavierMonton) [18:17:22] !log tchin@deploy1003 helmfile [codfw] DONE helmfile.d/services/eventstreams: apply [18:18:19] FIRING: [19x] KubernetesCalicoDown: wikikube-worker1119.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:18:30] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1134-1148].eqiad.wmnet [18:18:40] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1134-1148].eqiad.wmnet [18:19:12] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1149-1151,1153-1164].eqiad.wmnet [18:19:16] !log rzl@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [18:19:27] !log rzl@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [18:19:34] !log tchin@deploy1003 helmfile [codfw] START helmfile.d/services/eventstreams: apply [18:19:57] RESOLVED: ProbeDown: Service mw-web-next:4454 has failed probes (http_mw-web-next_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-web-next:4454 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:21:04] FIRING: [6x] KubernetesCalicoDown: wikikube-worker1134.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:24:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12486107 (10phaultfinder) [18:24:48] RESOLVED: PuppetFailure: Puppet has failed on build2004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [18:25:17] (03PS1) 10AKhatun: revision-feature-counts lac lambda: version update and reduce cpu [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359938 (https://phabricator.wikimedia.org/T441494) [18:25:36] !log tchin@deploy1003 helmfile [codfw] DONE helmfile.d/services/eventstreams: apply [18:25:48] !log tchin@deploy1003 helmfile [codfw] START helmfile.d/services/eventstreams: apply [18:25:51] !log dduvall@deploy1003 rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.1 refs T438223 [18:25:56] T438223: 1.48.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T438223 [18:26:26] !log tchin@deploy1003 helmfile [codfw] DONE helmfile.d/services/eventstreams: apply [18:26:35] dduvall: Smooth as butter! [18:27:12] dancy: you're making me nervous :D [18:28:03] nah, does look good though. phew! [18:28:05] haha. My enthusiasm is limited to the operation of the tooling. :-) [18:28:14] the hard part is behind us! nothing can stop us now! [18:28:27] haha [18:28:50] (03CR) 10Jelto: [C:04-1] "one comment in-line" [puppet] - 10https://gerrit.wikimedia.org/r/1359827 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [18:30:15] (03CR) 10Fabian Kaelin: "Good point, especially since presumably there will many such requests in the near future." [puppet] - 10https://gerrit.wikimedia.org/r/1359804 (https://phabricator.wikimedia.org/T442665) (owner: 10Fabian Kaelin) [18:30:44] FIRING: CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_cloudelastic_eqiad in eqiad (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=eqiad+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUnsta [18:30:57] !log rzl@apt1002:~$ sudo -i reprepro -C main includedeb bookworm-wikimedia /srv/wikimedia/pool/component/envoy-future/e/envoyproxy/envoyproxy_1.40.1-1_amd64.deb [18:31:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:31:34] !log rzl@apt1002:~$ sudo -i reprepro copy trixie-wikimedia bookworm-wikimedia envoyproxy [18:31:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:31:45] !log rzl@apt1002:~$ sudo -i reprepro copy bullseye-wikimedia bookworm-wikimedia envoyproxy # backwards, intended [18:31:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:32:09] (03CR) 10Jelto: deployment_server::deploy_service::service_catalog: Add eventgate-all (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359828 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [18:32:48] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1149-1151,1153-1164].eqiad.wmnet [18:35:22] (03PS1) 10RLazarus: envoy: Update to v1.40.1 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1359939 (https://phabricator.wikimedia.org/T436338) [18:35:45] RESOLVED: CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_cloudelastic_eqiad in eqiad (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=eqiad+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-cloudelastic - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUns [18:37:08] (03PS1) 10Krinkle: debug.json: Keep WikimediaDebug menu order consistent [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359940 (https://phabricator.wikimedia.org/T289745) [18:37:11] (03CR) 10RLazarus: [V:03+2] "Locally:" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1359939 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [18:39:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12486168 (10phaultfinder) [18:41:01] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1149-1151,1153-1164].eqiad.wmnet [18:41:04] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1158.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:41:11] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1149-1151,1153-1164].eqiad.wmnet [18:41:41] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1165,1240-1253].eqiad.wmnet [18:45:37] !log dancy@deploy1003 Installing scap version "4.307.0" for 155 host(s) [18:47:35] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12486194 (10cmooney) [18:48:42] (03PS1) 10Lizimedia: Add a basic completion script for cookbooks [software/spicerack] - 10https://gerrit.wikimedia.org/r/1359941 (https://phabricator.wikimedia.org/T367230) [18:50:43] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1165,1240-1253].eqiad.wmnet [18:50:53] (03CR) 10Ottomata: [C:03+1] revision-feature-counts lac lambda: version update and reduce cpu [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359938 (https://phabricator.wikimedia.org/T441494) (owner: 10AKhatun) [18:54:11] (03CR) 10CI reject: [V:04-1] Add a basic completion script for cookbooks [software/spicerack] - 10https://gerrit.wikimedia.org/r/1359941 (https://phabricator.wikimedia.org/T367230) (owner: 10Lizimedia) [18:57:19] (03CR) 10Ottomata: deployment_server::deploy_service::service_catalog: Add eventgate (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359827 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [18:57:55] !log Running `foreachwikiindblist group1.dblist cleanupWatchlistTemporaryAccountRows.php` for T439059 [18:58:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:58:01] T439059: Skip default watchlist additions during temp account creation - https://phabricator.wikimedia.org/T439059 [18:58:20] !log dancy@deploy1003 Installing scap version "4.307.0" for 2 host(s) [18:59:12] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1165,1240-1253].eqiad.wmnet [18:59:18] !log dancy@deploy1003 Installation of scap version "4.307.0" completed for 2 hosts [18:59:23] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1165,1240-1253].eqiad.wmnet [18:59:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12486235 (10phaultfinder) [18:59:59] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1254-1268].eqiad.wmnet [19:00:36] dancy: window clear? [19:01:01] ottomata: Lemme finish up a couple of things real quick [19:01:04] FIRING: [10x] KubernetesCalicoDown: wikikube-worker1240.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [19:01:24] ack [19:01:28] ottomatta: Done [19:06:00] (03PS2) 10Lizimedia: Add a basic completion script for cookbooks [software/spicerack] - 10https://gerrit.wikimedia.org/r/1359941 (https://phabricator.wikimedia.org/T367230) [19:09:06] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1254-1268].eqiad.wmnet [19:09:07] (03PS1) 10Tjones: Enable English Frames Second-Try Suggestions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359948 (https://phabricator.wikimedia.org/T434004) [19:10:36] (03CR) 10AKhatun: [C:03+2] revision-feature-counts lac lambda: version update and reduce cpu [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359938 (https://phabricator.wikimedia.org/T441494) (owner: 10AKhatun) [19:11:05] (03PS2) 10Tjones: Enable English Frames Second-Try Suggestions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359948 (https://phabricator.wikimedia.org/T434004) [19:12:13] (03Merged) 10jenkins-bot: revision-feature-counts lac lambda: version update and reduce cpu [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359938 (https://phabricator.wikimedia.org/T441494) (owner: 10AKhatun) [19:12:24] dancy: ty! [19:13:35] (03PS1) 10Ottomata: Prefetch the LAC revision_feature_counts artifact on group1 wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359949 [19:18:06] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1254-1268].eqiad.wmnet [19:18:17] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1254-1268].eqiad.wmnet [19:18:19] FIRING: [2x] KubernetesCalicoDown: wikikube-worker1256.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [19:18:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [19:18:48] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1269-1283].eqiad.wmnet [19:19:08] (03CR) 10Ahmon Dancy: deployment_server::deploy_service::service_catalog: Add eventgate (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359827 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [19:19:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12486277 (10phaultfinder) [19:21:52] !log [bking@deploy1003] `curl -XPOST -u $U -s https://opensearch-ttmserver.svc.eqiad.wmnet:30443/_cluster/reroute` Fix yellow status [19:21:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:22:48] !log akhatun@deploy1003 helmfile [staging] START helmfile.d/services/lac-lambdas: apply [19:23:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [19:23:52] !log akhatun@deploy1003 helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply [19:25:23] RECOVERY - VRRP status on cr1-eqiad is OK: VRRP OK - 0 misconfigured interfaces, 0 inconsistent interfaces https://wikitech.wikimedia.org/wiki/Network_monitoring%23VRRP_status [19:26:10] !log akhatun@deploy1003 helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply [19:26:26] !log akhatun@deploy1003 helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply [19:28:31] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1269-1283].eqiad.wmnet [19:28:47] (03CR) 10Dzahn: [C:03+1] phabricator: auto restart phd service [puppet] - 10https://gerrit.wikimedia.org/r/1359770 (https://phabricator.wikimedia.org/T442177) (owner: 10Jelto) [19:29:43] !log akhatun@deploy1003 helmfile [codfw] START helmfile.d/services/lac-lambdas: apply [19:30:12] !log akhatun@deploy1003 helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply [19:31:12] (03CR) 10Cathal Mooney: [C:03+2] cr1-eqiad: move row a sub-interfaces from ae1 to et-0/1/0 [homer/public] - 10https://gerrit.wikimedia.org/r/1359866 (https://phabricator.wikimedia.org/T441014) (owner: 10Cathal Mooney) [19:31:59] (03CR) 10AKhatun: [C:03+1] Prefetch the LAC revision_feature_counts artifact on group1 wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359949 (owner: 10Ottomata) [19:32:24] (03Merged) 10jenkins-bot: cr1-eqiad: move row a sub-interfaces from ae1 to et-0/1/0 [homer/public] - 10https://gerrit.wikimedia.org/r/1359866 (https://phabricator.wikimedia.org/T441014) (owner: 10Cathal Mooney) [19:32:50] (03PS1) 10Cathal Mooney: eqiad crs: move row b sub-interfaces from ae1 to et-0/1/0 [homer/public] - 10https://gerrit.wikimedia.org/r/1359956 (https://phabricator.wikimedia.org/T441014) [19:34:21] (03CR) 10Ottomata: deployment_server::deploy_service::service_catalog: Add eventgate (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359827 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [19:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:35:05] (03PS2) 10Cathal Mooney: eqiad crs: move row b sub-interfaces from ae1 to et-0/1/0 [homer/public] - 10https://gerrit.wikimedia.org/r/1359956 (https://phabricator.wikimedia.org/T441014) [19:36:48] (03CR) 10Cathal Mooney: [C:03+2] eqiad crs: move row b sub-interfaces from ae1 to et-0/1/0 [homer/public] - 10https://gerrit.wikimedia.org/r/1359956 (https://phabricator.wikimedia.org/T441014) (owner: 10Cathal Mooney) [19:37:11] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1269-1283].eqiad.wmnet [19:37:23] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1269-1283].eqiad.wmnet [19:37:52] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1284-1289,1291-1297,1299].eqiad.wmnet [19:37:55] (03Merged) 10jenkins-bot: eqiad crs: move row b sub-interfaces from ae1 to et-0/1/0 [homer/public] - 10https://gerrit.wikimedia.org/r/1359956 (https://phabricator.wikimedia.org/T441014) (owner: 10Cathal Mooney) [19:38:19] FIRING: [5x] KubernetesCalicoDown: wikikube-worker1276.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [19:39:21] (03CR) 10TrainBranchBot: [C:03+2] "Approved by akhatun@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359949 (owner: 10Ottomata) [19:39:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12486356 (10phaultfinder) [19:40:18] (03Merged) 10jenkins-bot: Prefetch the LAC revision_feature_counts artifact on group1 wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359949 (owner: 10Ottomata) [19:40:43] !log akhatun@deploy1003 Started scap sync-world: Backport for [[gerrit:1359949|Prefetch the LAC revision_feature_counts artifact on group1 wikipedias ]] [19:41:40] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12486364 (10cmooney) [19:45:05] !log akhatun@deploy1003 akhatun, otto: Backport for [[gerrit:1359949|Prefetch the LAC revision_feature_counts artifact on group1 wikipedias ]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:45:52] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1284-1289,1291-1297,1299].eqiad.wmnet [19:46:51] !log akhatun@deploy1003 akhatun, otto: Continuing with deployment [19:46:57] FIRING: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:49:00] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [19:51:57] RESOLVED: ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-ext:4447 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:52:13] !log pt1979@cumin1004 START - Cookbook sre.hosts.dhcp for host ml-serve1016.eqiad.wmnet (live debug) [19:52:31] !log akhatun@deploy1003 Finished scap sync-world: Backport for [[gerrit:1359949|Prefetch the LAC revision_feature_counts artifact on group1 wikipedias ]] (duration: 11m 48s) [19:52:37] FIRING: CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [19:52:39] FIRING: CoreBGPDown: Core BGP session down between cr2-codfw and cr2-eqiad (208.80.154.216) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr2-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [19:54:14] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row B interfaces on eqiad CRs - cmooney@cumin1004" [19:54:18] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row B interfaces on eqiad CRs - cmooney@cumin1004" [19:54:19] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [19:54:22] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1284-1289,1291-1297,1299].eqiad.wmnet [19:54:31] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1284-1289,1291-1297,1299].eqiad.wmnet [19:54:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12486443 (10phaultfinder) [19:55:01] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1301-1314].eqiad.wmnet [19:55:17] pt1979@cumin1004 dhcp (PID 3626703) is awaiting input [19:55:22] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [19:56:04] FIRING: [3x] KubernetesCalicoDown: wikikube-worker1287.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [19:56:54] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: eqiad row A/B switch refresh: move row A and B VC uplinks from core routers to spine switches - https://phabricator.wikimedia.org/T441014#12486454 (10cmooney) 05Open→03Resolved All done with this one. Unfortunately interrupt... [19:57:37] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [19:57:39] RESOLVED: CoreBGPDown: Core BGP session down between cr2-codfw and cr2-eqiad (208.80.154.216) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr2-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr2-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [19:59:12] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [20:00:05] RoanKattouw, urbanecm, TheresNoTime, and cjming: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for UTC late backport window . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T2000). [20:00:05] sbassett and arlolra: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:56] o/ [20:02:24] Hey arlorla - I see you have the other patch for the backport deploy now.  Mine is a private patch (not the linked gerrit change set).  If yours is quick, it might make more sense for you to go first. [20:02:46] mine is just a config change, should be quick, thanks [20:03:06] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row B interfaces on eqiad CRs - cmooney@cumin1004" [20:03:10] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update subint entries for row B interfaces on eqiad CRs - cmooney@cumin1004" [20:03:10] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [20:03:35] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1301-1314].eqiad.wmnet [20:03:48] (03PS5) 10Dzahn: etherpad: stop apache again when etherpad is running [puppet] - 10https://gerrit.wikimedia.org/r/1359846 (https://phabricator.wikimedia.org/T442232) (owner: 10Jelto) [20:04:06] (03CR) 10TrainBranchBot: [C:03+2] "Approved by arlolra@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358832 (https://phabricator.wikimedia.org/T430970) (owner: 10Arlolra) [20:05:48] (03Merged) 10jenkins-bot: Turn off "report a visual" bug on wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358832 (https://phabricator.wikimedia.org/T430970) (owner: 10Arlolra) [20:06:10] !log arlolra@deploy1003 Started scap sync-world: Backport for [[gerrit:1358832|Turn off "report a visual" bug on wikipedias (T430970)]] [20:06:17] T430970: Turn-off "Report a Visual Bug" functionality on all Wikipedias at the end of Q1 26-27 - https://phabricator.wikimedia.org/T430970 [20:09:36] 10ops-codfw, 06DC-Ops: Alert for device ps1-c4-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442765 (10phaultfinder) 03NEW [20:09:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12486583 (10phaultfinder) [20:09:54] (03PS1) 10Jforrester: [DNM] chromium-render: Drop --single-process, add tini [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359960 (https://phabricator.wikimedia.org/T441276) [20:10:17] !log cmooney@cumin1004 START - Cookbook sre.dns.netbox [20:10:34] !log arlolra@deploy1003 arlolra: Backport for [[gerrit:1358832|Turn off "report a visual" bug on wikipedias (T430970)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:10:38] (03PS1) 10Ahmon Dancy: scap: Wait for install-world in bootstrap-scap-target.sh [puppet] - 10https://gerrit.wikimedia.org/r/1359961 (https://phabricator.wikimedia.org/T382271) [20:11:48] (03CR) 10Dzahn: "currently apache is inactive but still enabled. merging this simplified version (PS5) for now to get it deactivated. open to any improveme" [puppet] - 10https://gerrit.wikimedia.org/r/1359846 (https://phabricator.wikimedia.org/T442232) (owner: 10Jelto) [20:11:57] !log arlolra@deploy1003 arlolra: Continuing with deployment [20:12:04] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1301-1314].eqiad.wmnet [20:12:14] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1301-1314].eqiad.wmnet [20:12:16] (03CR) 10Dzahn: [C:03+2] etherpad: stop apache again when etherpad is running [puppet] - 10https://gerrit.wikimedia.org/r/1359846 (https://phabricator.wikimedia.org/T442232) (owner: 10Jelto) [20:12:41] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1315-1328].eqiad.wmnet [20:13:19] FIRING: [5x] KubernetesCalicoDown: wikikube-worker1301.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [20:14:12] !log cmooney@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update last few IPs that got awawy - cmooney@cumin1004" [20:14:16] !log cmooney@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update last few IPs that got awawy - cmooney@cumin1004" [20:14:16] !log cmooney@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [20:15:47] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12486613 (10Dzahn) [x] verified key out-of-band [20:16:02] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12486616 (10Dzahn) [20:17:18] (03CR) 10Dzahn: [C:03+2] "before:" [puppet] - 10https://gerrit.wikimedia.org/r/1359846 (https://phabricator.wikimedia.org/T442232) (owner: 10Jelto) [20:17:36] !log arlolra@deploy1003 Finished scap sync-world: Backport for [[gerrit:1358832|Turn off "report a visual" bug on wikipedias (T430970)]] (duration: 11m 26s) [20:17:42] T430970: Turn-off "Report a Visual Bug" functionality on all Wikipedias at the end of Q1 26-27 - https://phabricator.wikimedia.org/T430970 [20:17:53] (03CR) 10Dzahn: [V:03+1 C:03+2] "follow-up https://gerrit.wikimedia.org/r/c/operations/puppet/+/1359846 fixed to properly stop httpd when etherpad is up" [puppet] - 10https://gerrit.wikimedia.org/r/1358284 (https://phabricator.wikimedia.org/T442232) (owner: 10Dzahn) [20:18:20] sbassett: all yours [20:18:42] FIRING: JobUnavailable: Reduced availability for job bacula in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:19:36] !log etherpad1004 - apache httpd disabled by puppet - etherpad running normally [20:19:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:20:33] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1315-1328].eqiad.wmnet [20:22:24] arlorla: thanks! [20:29:04] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1315-1328].eqiad.wmnet [20:29:14] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1315-1328].eqiad.wmnet [20:29:43] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1329-1342].eqiad.wmnet [20:30:23] (03PS1) 10Dzahn: admin: create user for Corbin Foster, analytics-privatedata [puppet] - 10https://gerrit.wikimedia.org/r/1359965 (https://phabricator.wikimedia.org/T442648) [20:31:04] FIRING: [6x] KubernetesCalicoDown: wikikube-worker1315.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [20:33:07] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12486678 (10Dzahn) Welcome @cfoster ! Confirming we have verified your key out of band. Almost all boxes are... [20:34:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12486679 (10phaultfinder) [20:40:05] 06SRE, 10SRE-Access-Requests, 06Data-Engineering: Requesting Kerberos identity for rsilvola - https://phabricator.wikimedia.org/T442695#12486705 (10Dzahn) If Kerberos is needed it's a request for a an upgrade from level 1 or 2 (which was not specified in the original task) to level 3. (this is T405517). th... [20:40:45] !log Deployed security mitigations for T435455, T439844, T442538 [20:40:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:42:32] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1329-1342].eqiad.wmnet [20:44:22] (03CR) 10Dzahn: "pending approval from mpopov for the additional group" [puppet] - 10https://gerrit.wikimedia.org/r/1359965 (https://phabricator.wikimedia.org/T442648) (owner: 10Dzahn) [20:45:54] FIRING: KubernetesAPILatency: High Kubernetes API latency (LIST pods) on k8s@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [20:47:03] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12486719 (10cfoster) @mpopov is out of office through this week - is there an alternate approver we could con... [20:50:19] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1329-1342].eqiad.wmnet [20:50:29] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1329-1342].eqiad.wmnet [20:50:54] RESOLVED: KubernetesAPILatency: High Kubernetes API latency (LIST pods) on k8s@eqiad - https://wikitech.wikimedia.org/wiki/Kubernetes - https://grafana.wikimedia.org/d/ddNd-sLnk/kubernetes-api-details?var-site=eqiad&var-cluster=k8s&var-latency_percentile=0.95&var-verb=LIST - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [20:50:57] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1343-1356].eqiad.wmnet [20:53:29] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12486743 (10Dzahn) Unfortunately only one is listed. Usually we recommend multiple ones to avoid this kind of... [20:54:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12486748 (10phaultfinder) [20:57:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:57:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [20:58:08] (03PS2) 10Jforrester: wikifunctions: Upgrade evaluators from 2026-10-14-185937 to 2026-10-28-122530 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359854 (https://phabricator.wikimedia.org/T433639) [20:59:22] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1343-1356].eqiad.wmnet [20:59:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12486770 (10phaultfinder) [21:00:05] Deploy window Wikifunctions Services UTC Late (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T2100) [21:00:46] !log jforrester@deploy1003 Started scap deploy-service wikifunctions: Weekly function-evaluator service deploy [21:00:46] !log jforrester@deploy1003 scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions [21:01:23] !log jforrester@deploy1003 scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions [21:01:32] (03CR) 10Dzahn: [C:03+2] scap: Wait for install-world in bootstrap-scap-target.sh [puppet] - 10https://gerrit.wikimedia.org/r/1359961 (https://phabricator.wikimedia.org/T382271) (owner: 10Ahmon Dancy) [21:01:52] !log jforrester@deploy1003 scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions [21:02:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:02:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [21:02:15] !log jforrester@deploy1003 Finished scap deploy-service wikifunctions: Weekly function-evaluator service deploy (duration: 01m 29s) [21:02:40] Yay! [21:02:48] FIRING: PuppetFailure: Puppet has failed on ml-staging2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [21:02:58] (03CR) 10Jforrester: [C:03+2] wikifunctions: Upgrade evaluators from 2026-10-14-185937 to 2026-10-28-122530 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359854 (https://phabricator.wikimedia.org/T433639) (owner: 10Jforrester) [21:03:40] dancy: Very smooth! Now for a non-test deploy. :-) [21:04:08] Excellent. Lemme know when you're done. I have a new release of scap to deploy. [21:04:15] Aha, cool. [21:04:25] (03CR) 10Dzahn: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1357359 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [21:04:26] (03Merged) 10jenkins-bot: wikifunctions: Upgrade evaluators from 2026-10-14-185937 to 2026-10-28-122530 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359854 (https://phabricator.wikimedia.org/T433639) (owner: 10Jforrester) [21:06:16] !log jforrester@deploy1003 Started scap deploy-service wikifunctions: Weekly f-e service deploy, this time with a bump commit [21:06:16] !log jforrester@deploy1003 scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions [21:06:41] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1343-1356].eqiad.wmnet [21:06:50] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1343-1356].eqiad.wmnet [21:07:17] !log jforrester@deploy1003 scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions [21:07:20] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1357-1370].eqiad.wmnet [21:07:42] !log jforrester@deploy1003 scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions [21:08:27] !log jforrester@deploy1003 Finished scap deploy-service wikifunctions: Weekly f-e service deploy, this time with a bump commit (duration: 02m 11s) [21:08:34] (03PS2) 10Jforrester: wikifunctions: Upgrade orchestrator from 2026-10-14-161253 to 2026-10-28-013937 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359855 (https://phabricator.wikimedia.org/T357858) [21:08:39] (03CR) 10Jforrester: [C:03+2] wikifunctions: Upgrade orchestrator from 2026-10-14-161253 to 2026-10-28-013937 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359855 (https://phabricator.wikimedia.org/T357858) (owner: 10Jforrester) [21:10:12] (03Merged) 10jenkins-bot: wikifunctions: Upgrade orchestrator from 2026-10-14-161253 to 2026-10-28-013937 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359855 (https://phabricator.wikimedia.org/T357858) (owner: 10Jforrester) [21:11:29] James_F: is this something we could likely get out during or just after the wikifunctions window: https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/1359970? [21:11:38] (03CR) 10Dzahn: [C:03+1] "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1357838 (https://phabricator.wikimedia.org/T441920) (owner: 10Hashar) [21:12:19] sbassett: Yes, I’m not touching the MW side at all, I think you might be able to deploy in parallel to me. [21:12:27] sbassett: Also I’ll be done in 2–3 mins. [21:12:30] Ok, I’ll try spiderpig [21:12:36] Thanks [21:12:53] !log jforrester@deploy1003 Started scap deploy-service wikifunctions: Weekly f-o service deploy [21:12:53] !log jforrester@deploy1003 scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions [21:13:19] !log jforrester@deploy1003 scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions [21:14:00] !log jforrester@deploy1003 scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions [21:14:40] !log jforrester@deploy1003 Finished scap deploy-service wikifunctions: Weekly f-o service deploy (duration: 01m 47s) [21:14:45] All done at my end. [21:14:47] (03PS1) 10AKhatun: revision-feature-counts lac-lambda: version bump [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359971 [21:14:57] dancy: Over to you, once sbassett finishes? [21:15:44] 06SRE, 10SRE-Access-Requests, 06Data-Engineering: Requesting Kerberos identity for rsilvola - https://phabricator.wikimedia.org/T442695#12486830 (10Rsilvola) I now need level 3 for data lake access. [21:16:59] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1357-1370].eqiad.wmnet [21:17:35] (03CR) 10AKhatun: [C:03+2] revision-feature-counts lac-lambda: version bump [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359971 (owner: 10AKhatun) [21:18:53] !log akhatun@deploy1003 helmfile [staging] START helmfile.d/services/lac-lambdas: apply [21:19:01] !log akhatun@deploy1003 helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply [21:19:04] (03Merged) 10jenkins-bot: revision-feature-counts lac-lambda: version bump [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359971 (owner: 10AKhatun) [21:20:37] !log akhatun@deploy1003 helmfile [staging] START helmfile.d/services/lac-lambdas: apply [21:20:41] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [21:20:53] !log akhatun@deploy1003 helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply [21:21:15] !log akhatun@deploy1003 helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply [21:21:35] !log akhatun@deploy1003 helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply [21:21:45] !log akhatun@deploy1003 helmfile [codfw] START helmfile.d/services/lac-lambdas: apply [21:22:05] !log akhatun@deploy1003 helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply [21:22:43] Works for me [21:24:52] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1357-1370].eqiad.wmnet [21:25:02] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1357-1370].eqiad.wmnet [21:25:31] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[1371-1384].eqiad.wmnet [21:26:06] !log pt1979@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1016.eqiad.wmnet (live debug) [21:26:12] (03PS2) 10SBassett: Bump symfony/mailer and symfony/mime to 7.4.22 [vendor] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359972 (https://phabricator.wikimedia.org/T442768) [21:27:23] dancy: is the scap version with https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1357 already installed on all targets so nothing still reads git_server from DEPLOY_HEAD? and should deploy2003 get the same cleanup? [21:29:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12486927 (10phaultfinder) [21:30:56] mutante: Yes on the scap version. regarding deploy2003: It was my understanding that there's a timer that syncs /srv/deployment to secondary deployment servers periodically. Lemme find evidence of that. [21:31:28] that's true, it shoyld sync [21:31:36] ok good [21:33:51] !log deploy1003 - sudo find . -path '*/.git/DEPLOY_HEAD' -exec sed -i -E '/^git_(server|scheme|deploy_dir):/d' {} + (backup in /root/DEPLOY_HEAD-backup.tgz) - T197470 [21:33:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:33:57] T197470: find a way to systematically update the deployment server name across all repos - https://phabricator.wikimedia.org/T197470 [21:34:02] root@deploy1003:~# sudo find . -path '*/.git/DEPLOY_HEAD' -exec grep -HE '^git_(server|scheme|deploy_dir):' {} + | wc -l [21:34:05] 0 [21:34:20] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[1371-1384].eqiad.wmnet [21:34:21] dancy: the command is slightly different but does the same thing. because something with backslashes and copy/pasting [21:34:29] dancy: done and shows 0 now [21:34:45] Thanks!! bugs_from_2018-- [21:35:01] indeed!:) and I made that ticket. thank you for that fix :) [21:35:31] came up once per new deployment server only [21:36:01] Nod. Raine experienced it recently. [21:36:27] (03PS2) 10SBassett: Bump symfony/mailer and symfony/mime to 7.4.22 [vendor] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359974 (https://phabricator.wikimedia.org/T442768) [21:37:49] yea, it was always like "how the hell is tin.eqiad.wmnet still in there, how does it even know that machine, it's gone so long" [21:37:57] but we kept syncing these files :) [21:38:25] Zombie hosts [21:38:29] haha, yea [21:39:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12486952 (10phaultfinder) [21:40:17] (03CR) 10Dzahn: [V:03+1 C:03+1] "sorry, I was confused, previously, but not anymore:) this is good to go! https://puppet-compiler.wmflabs.org/output/1357838/8265/contint10" [puppet] - 10https://gerrit.wikimedia.org/r/1357838 (https://phabricator.wikimedia.org/T441920) (owner: 10Hashar) [21:40:29] (03CR) 10Dzahn: [V:03+1 C:03+2] ci: remove zuul-legacy.wikimedia.org, never used [puppet] - 10https://gerrit.wikimedia.org/r/1357838 (https://phabricator.wikimedia.org/T441920) (owner: 10Hashar) [21:40:44] (03CR) 10Dzahn: [V:03+1 C:03+2] "> I did this change as the first of my series (https://gerrit.wikimedia.org/r/q/bug:T441920 ) to simplify the Puppet manifest and because " [puppet] - 10https://gerrit.wikimedia.org/r/1357838 (https://phabricator.wikimedia.org/T441920) (owner: 10Hashar) [21:42:03] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker[1371-1384].eqiad.wmnet [21:42:13] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker[1371-1384].eqiad.wmnet [21:42:17] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker-exp1001.eqiad.wmnet [21:42:48] RESOLVED: PuppetFailure: Puppet has failed on ml-staging2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [21:42:50] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker-exp1001.eqiad.wmnet [21:44:48] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12486967 (10phaultfinder) [21:45:05] (03PS4) 10SBassett: Bump symfony/mailer and symfony/mime to 7.4.22 [vendor] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359972 (https://phabricator.wikimedia.org/T442768) [21:46:11] PROBLEM - MariaDB Replica Lag: s4 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 601.41 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:46:17] (03CR) 10Dzahn: [C:03+2] Gemfile: pin openssl below 4 instead of net-ssh below 8 [puppet] - 10https://gerrit.wikimedia.org/r/1357590 (owner: 10Dzahn) [21:46:58] !log jasmine@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker-exp1001.eqiad.wmnet [21:47:01] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker-exp1001.eqiad.wmnet [21:47:01] !log jasmine@cumin2003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-worker-eqiad [21:50:43] (03CR) 10Reedy: "Probably should revert the file additions" [vendor] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359972 (https://phabricator.wikimedia.org/T442768) (owner: 10SBassett) [21:51:06] (03CR) 10Dzahn: [V:03+1 C:03+2] "confirmed this was a noop!" [puppet] - 10https://gerrit.wikimedia.org/r/1357838 (https://phabricator.wikimedia.org/T441920) (owner: 10Hashar) [21:51:35] !log ammarpad@deploy1003 mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikimania2007wiki --logwiki=metawiki MogosCZ Mogoskii # T442713 [21:51:40] T442713: Unblock stuck global rename of Mogoskii - https://phabricator.wikimedia.org/T442713 [21:54:37] (03PS1) 10Arlolra: Revert "Turn off "report a visual" bug on wikipedias" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359978 (https://phabricator.wikimedia.org/T430970) [21:55:45] (03CR) 10Subramanya Sastry: [C:03+1] Revert "Turn off "report a visual" bug on wikipedias" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359978 (https://phabricator.wikimedia.org/T430970) (owner: 10Arlolra) [21:56:26] hnowlan: bblack: Can I deploy that revert above from a config change I made an hour ago? [21:57:26] It's creating a lot of js errors [21:59:34] (03PS3) 10Lizimedia: Add a basic completion script for cookbooks [software/spicerack] - 10https://gerrit.wikimedia.org/r/1359941 (https://phabricator.wikimedia.org/T367230) [22:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261028T2200) [22:00:29] Hey all, I'm going to deploy a few patches for the Readers window momentarily if Scap is open [22:00:59] Me too EricGardner [22:01:05] I didnt see you in the doc so I assume it was just me [22:01:07] I have one config change and two backports [22:01:13] thought I added them [22:02:18] oh whoops. Looks like I added the heading but not the link. I have one config change here - it's for a KR so needs to go out today. I'm okay to go after you as long as it does go out by the end of the hour. [22:02:22] (03CR) 10CI reject: [V:04-1] Add a basic completion script for cookbooks [software/spicerack] - 10https://gerrit.wikimedia.org/r/1359941 (https://phabricator.wikimedia.org/T367230) (owner: 10Lizimedia) [22:02:51] (03CR) 10SBassett: "Composer seemed to want to add those for whatever reason. But I can get rid of them for this patch." [vendor] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359972 (https://phabricator.wikimedia.org/T442768) (owner: 10SBassett) [22:03:30] @egyou want to go first then? I can come back in 30m [22:03:30] Jdlrobson: if you are just doing a config deploy you can go first [22:03:33] ok [22:03:49] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359380 (https://phabricator.wikimedia.org/T441705) (owner: 10Jdlrobson) [22:03:55] sorry for the doc confusion [22:03:59] no prob [22:04:20] (03PS5) 10SBassett: Bump symfony/mailer and symfony/mime to 7.4.22 [vendor] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359972 (https://phabricator.wikimedia.org/T442768) [22:04:22] (03CR) 10Lizimedia: [V:03+2] Add a basic completion script for cookbooks [software/spicerack] - 10https://gerrit.wikimedia.org/r/1359941 (https://phabricator.wikimedia.org/T367230) (owner: 10Lizimedia) [22:04:37] Jdlrobson: are you going to deploy my revert? [22:04:38] (03Merged) 10jenkins-bot: Enable Vector ReadingList CTA on group 1 wikipedias+mw+wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359380 (https://phabricator.wikimedia.org/T441705) (owner: 10Jdlrobson) [22:04:56] arlolra: i can do it after Eric's if we have time [22:05:02] (since it requires a scap) [22:05:07] !log jdlrobson@deploy1003 Started scap sync-world: Backport for [[gerrit:1359380|Enable Vector ReadingList CTA on group 1 wikipedias+mw+wikitech (T441705)]] [22:05:13] T441705: Desktop CTA: Deploy the new config variable to launch the desktop CTA - https://phabricator.wikimedia.org/T441705 [22:06:08] (03CR) 10SBassett: "Removed in latest PS. Though the symfony/yaml updates are retained (that weren't in wmf.1 or master)." [vendor] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359972 (https://phabricator.wikimedia.org/T442768) (owner: 10SBassett) [22:07:50] (03CR) 10Dzahn: [C:03+2] codesearch: show a preview banner on the production frontend [puppet] - 10https://gerrit.wikimedia.org/r/1357359 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [22:09:24] !log jdlrobson@deploy1003 jdlrobson: Backport for [[gerrit:1359380|Enable Vector ReadingList CTA on group 1 wikipedias+mw+wikitech (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:11:38] !log jdlrobson@deploy1003 jdlrobson: Continuing with deployment [22:12:39] All yours @EricGardner when the sync is done [22:12:53] Ok, starting momentarily [22:14:53] EricGardner: can you let me know when you’re done? I’d like to get a mediawiki/vendor change out today still... [22:15:38] sbassett sure thing. I am deploying one config change and then I have a set of two regular backports. One of those touches i18n files so be warned [22:17:02] EricGardner: if you’re using spiderpig, do you mind just tacking on 1359972 and 1359974? [22:17:20] !log jdlrobson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1359380|Enable Vector ReadingList CTA on group 1 wikipedias+mw+wikitech (T441705)]] (duration: 12m 13s) [22:17:27] T441705: Desktop CTA: Deploy the new config variable to launch the desktop CTA - https://phabricator.wikimedia.org/T441705 [22:18:01] sbassett sure, will you be testing those changes once they are up? [22:18:38] I will do my config change first, then the other backports including those two [22:18:56] (03CR) 10TrainBranchBot: [C:03+2] "Approved by egardner@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357787 (owner: 10DErenrich) [22:19:43] (03Merged) 10jenkins-bot: Disable Discord QuickSurvey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357787 (owner: 10DErenrich) [22:20:08] !log egardner@deploy1003 Started scap sync-world: Backport for [[gerrit:1357787|Disable Discord QuickSurvey]] [22:20:27] EricGardner: they are minor version bumps that address a published security issue. i would just monitor logstash error rates and if anything went awry, we could stop the deploy. but if you just want to get yours out, that’s understandable. [22:21:03] sbassett I don't mind bringing yours along for the ride if you can monitor the relevant logs [22:21:58] yep, i plan to, thanks [22:24:27] !log egardner@deploy1003 derenrich, egardner: Backport for [[gerrit:1357787|Disable Discord QuickSurvey]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:25:10] !log egardner@deploy1003 derenrich, egardner: Continuing with deployment [22:25:13] (03PS2) 10Jforrester: chromium-render: Drop --single-process, add tini, up to 2026-10-28-215517-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359960 (https://phabricator.wikimedia.org/T441276) [22:30:50] !log egardner@deploy1003 Finished scap sync-world: Backport for [[gerrit:1357787|Disable Discord QuickSurvey]] (duration: 10m 41s) [22:31:40] (03CR) 10TrainBranchBot: [C:03+2] "Approved by egardner@deploy1003 using scap backport" [extensions/WikimediaCustomizations] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359934 (https://phabricator.wikimedia.org/T439164) (owner: 10Eric Gardner) [22:31:40] (03CR) 10TrainBranchBot: [C:03+2] "Approved by egardner@deploy1003 using scap backport" [extensions/WikimediaCustomizations] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359937 (https://phabricator.wikimedia.org/T439161) (owner: 10Eric Gardner) [22:31:41] (03CR) 10TrainBranchBot: [C:03+2] "Approved by egardner@deploy1003 using scap backport" [vendor] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359972 (https://phabricator.wikimedia.org/T442768) (owner: 10SBassett) [22:31:42] (03CR) 10TrainBranchBot: [C:03+2] "Approved by egardner@deploy1003 using scap backport" [vendor] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359974 (https://phabricator.wikimedia.org/T442768) (owner: 10SBassett) [22:32:58] (03Merged) 10jenkins-bot: ShareCard: Show page views as a monthly rate [extensions/WikimediaCustomizations] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359934 (https://phabricator.wikimedia.org/T439164) (owner: 10Eric Gardner) [22:34:17] (03Merged) 10jenkins-bot: ShareCard: Hide images on the bad image list [extensions/WikimediaCustomizations] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359937 (https://phabricator.wikimedia.org/T439161) (owner: 10Eric Gardner) [22:34:20] (03Merged) 10jenkins-bot: Bump symfony/mailer and symfony/mime to 7.4.22 [vendor] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359972 (https://phabricator.wikimedia.org/T442768) (owner: 10SBassett) [22:34:25] (03Merged) 10jenkins-bot: Bump symfony/mailer and symfony/mime to 7.4.22 [vendor] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1359974 (https://phabricator.wikimedia.org/T442768) (owner: 10SBassett) [22:34:59] !log egardner@deploy1003 Started scap sync-world: Backport for [[gerrit:1359934|ShareCard: Show page views as a monthly rate (T439164)]] (1.48.0-wmf.1), [[gerrit:1359937|ShareCard: Hide images on the bad image list (T439161)]] (1.48.0-wmf.1), Bump symfony/mailer and symfony/mime to 7.4.22 (T442768) ([[gerrit:1359972|1.47.0-wmf.24]], [[gerrit:1359974|1.48.0-wmf.1]]) [22:35:07] T439164: ShareCard: Design updates - https://phabricator.wikimedia.org/T439164 [22:35:08] T439161: [Epic] ShareCard: article sharing on mobile web - https://phabricator.wikimedia.org/T439161 [22:45:11] RECOVERY - MariaDB Replica Lag: s4 on db2239 is OK: OK slave_sql_lag Replication lag: 18.53 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:51:42] EricGardner: are you done? [22:51:53] No, still waiting [22:52:04] k k [22:52:04] It's been 20 min and we're not at test servers yet [22:54:04] yeah, just the reality of some sync-worlds these days [22:54:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12487217 (10phaultfinder) [22:55:01] (03PS1) 10Dzahn: gitlab_runner: add both docker registries to allowed_services [puppet] - 10https://gerrit.wikimedia.org/r/1359987 (https://phabricator.wikimedia.org/T439046) [22:56:37] (03PS2) 10Dzahn: gitlab_runner: add both docker registries to allowed_services [puppet] - 10https://gerrit.wikimedia.org/r/1359987 (https://phabricator.wikimedia.org/T439046) [22:56:48] !log egardner@deploy1003 sbassett, egardner: Backport for [[gerrit:1359934|ShareCard: Show page views as a monthly rate (T439164)]] (1.48.0-wmf.1), [[gerrit:1359937|ShareCard: Hide images on the bad image list (T439161)]] (1.48.0-wmf.1), Bump symfony/mailer and symfony/mime to 7.4.22 (T442768) ([[gerrit:1359972|1.47.0-wmf.24]], [[gerrit:1359974|1.48.0-wmf.1]]) synced to the testservers (see https://wikitech.wikimedia.org/ [22:56:48] wiki/Mwdebug). Changes can now be verified there. [22:56:57] T439164: ShareCard: Design updates - https://phabricator.wikimedia.org/T439164 [22:56:57] T439161: [Epic] ShareCard: article sharing on mobile web - https://phabricator.wikimedia.org/T439161 [22:58:22] Ok, test servers look good on my end. sbassett I will proceed with deploy unless you see any issues [23:00:01] !log egardner@deploy1003 sbassett, egardner: Continuing with deployment [23:00:18] sounds good [23:00:39] Jdlrobson after these changes are done you can take over, that's it for me today [23:01:05] sbassett: I'll do the new scap release deployment tomorrow. [23:01:11] Have a good day everyone! [23:04:25] dancy: cool, thanks [23:10:19] (03PS1) 10Lizimedia: Run the cookbook command to generate a static bash completion file [puppet] - 10https://gerrit.wikimedia.org/r/1359989 (https://phabricator.wikimedia.org/T367230) [23:10:55] EricGardner: I talked to Arlo and we'll going to do the change tomorrow now. So we're done when you are [23:11:22] Sounds good. My deploy should be wrapped up soon [23:11:25] (03CR) 10CI reject: [V:04-1] Run the cookbook command to generate a static bash completion file [puppet] - 10https://gerrit.wikimedia.org/r/1359989 (https://phabricator.wikimedia.org/T367230) (owner: 10Lizimedia) [23:11:57] !log egardner@deploy1003 Finished scap sync-world: Backport for [[gerrit:1359934|ShareCard: Show page views as a monthly rate (T439164)]] (1.48.0-wmf.1), [[gerrit:1359937|ShareCard: Hide images on the bad image list (T439161)]] (1.48.0-wmf.1), Bump symfony/mailer and symfony/mime to 7.4.22 (T442768) ([[gerrit:1359972|1.47.0-wmf.24]], [[gerrit:1359974|1.48.0-wmf.1]]) (duration: 36m 58s) [23:12:04] T439164: ShareCard: Design updates - https://phabricator.wikimedia.org/T439164 [23:12:05] T439161: [Epic] ShareCard: article sharing on mobile web - https://phabricator.wikimedia.org/T439161 [23:12:55] ok, sounds like we are done for the day [23:13:05] (03PS2) 10Lizimedia: Run the cookbook command to generate a static bash completion file [puppet] - 10https://gerrit.wikimedia.org/r/1359989 (https://phabricator.wikimedia.org/T367230) [23:15:01] (03CR) 10Arlolra: [C:04-1] "Let's try to backport a fix tomorrow instead of reverting" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359978 (https://phabricator.wikimedia.org/T430970) (owner: 10Arlolra) [23:16:15] PROBLEM - Swift https backend on ms-fe2015 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Swift [23:17:05] RECOVERY - Swift https backend on ms-fe2015 is OK: HTTP OK: HTTP/1.1 200 OK - 572 bytes in 0.192 second response time https://wikitech.wikimedia.org/wiki/Swift [23:21:11] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 610.37 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [23:29:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12487304 (10phaultfinder) [23:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:44:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12487391 (10phaultfinder)