[00:00:03] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox: sync [00:00:07] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [00:07:11] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:18:42] FIRING: JobUnavailable: Reduced availability for job bacula in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:19:57] FIRING: ProbeDown: Service mw-api-int:4446 has failed probes (http_mw-api-int_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-int:4446 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:24:57] RESOLVED: ProbeDown: Service mw-api-int:4446 has failed probes (http_mw-api-int_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-api-int:4446 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:30:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:31:04] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [00:35:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:35:37] (03PS1) 10Ladsgroup: ParserOutputAccess: Don't load dirty entry if not needed [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359999 (https://phabricator.wikimedia.org/T442669) [00:35:49] (03PS1) 10Ladsgroup: ParserOutputAccess: Don't load dirty entry if not needed [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360000 (https://phabricator.wikimedia.org/T442669) [00:39:30] (03CR) 10CI reject: [V:04-1] ParserOutputAccess: Don't load dirty entry if not needed [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359999 (https://phabricator.wikimedia.org/T442669) (owner: 10Ladsgroup) [00:39:57] FIRING: ProbeDown: Service mw-web:4450 has failed probes (http_mw-web_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-web:4450 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:40:06] (03CR) 10CI reject: [V:04-1] ParserOutputAccess: Don't load dirty entry if not needed [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360000 (https://phabricator.wikimedia.org/T442669) (owner: 10Ladsgroup) [00:40:24] jouncebot: nowandnext [00:40:24] No deployments scheduled for the next 1 hour(s) and 19 minute(s) [00:40:25] In 1 hour(s) and 19 minute(s): Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T0200) [00:40:31] booo to ci reject [00:41:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:42:30] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1360001 [00:42:30] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1360001 (owner: 10TrainBranchBot) [00:44:57] RESOLVED: ProbeDown: Service mw-web:4450 has failed probes (http_mw-web_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#mw-web:4450 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:46:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:47:17] (03CR) 10Ladsgroup: "recheck" [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359999 (https://phabricator.wikimedia.org/T442669) (owner: 10Ladsgroup) [00:47:25] (03CR) 10Ladsgroup: "recheck" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360000 (https://phabricator.wikimedia.org/T442669) (owner: 10Ladsgroup) [00:49:11] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1360001 (owner: 10TrainBranchBot) [00:51:45] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:52:00] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:54:53] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12487565 (10phaultfinder) [00:57:14] FIRING: [3x] ProbeDown: Service people1005:30443 has failed probes (http_static_codereview_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [00:58:13] (03PS1) 10RLazarus: rest-gateway: Update to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360002 (https://phabricator.wikimedia.org/T436338) [00:58:15] (03PS1) 10RLazarus: mw-videoscaler: Update to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360003 (https://phabricator.wikimedia.org/T436338) [00:58:18] (03PS1) 10RLazarus: mw-*: Update canary deployments to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360004 (https://phabricator.wikimedia.org/T436338) [00:58:18] (03PS1) 10RLazarus: mw-*: Update all deployments to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360005 [01:00:24] (03PS1) 10RLazarus: kubernetes: Update default envoy version to 1.40.1 [puppet] - 10https://gerrit.wikimedia.org/r/1360006 (https://phabricator.wikimedia.org/T436338) [01:02:14] FIRING: [7x] ProbeDown: Service people1005:30443 has failed probes (http_15_wikipedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:04:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12487581 (10phaultfinder) [01:07:14] FIRING: [8x] ProbeDown: Service people1005:30443 has failed probes (http_15_wikipedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:12:14] FIRING: [8x] ProbeDown: Service people1005:30443 has failed probes (http_15_wikipedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:12:23] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1360010 [01:12:23] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1360010 (owner: 10TrainBranchBot) [01:13:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:14:29] (03Abandoned) 10Arlolra: Revert "Turn off "report a visual" bug on wikipedias" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359978 (https://phabricator.wikimedia.org/T430970) (owner: 10Arlolra) [01:17:14] FIRING: [7x] ProbeDown: Service people1005:30443 has failed probes (http_15_wikipedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:18:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:18:33] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1360010 (owner: 10TrainBranchBot) [01:20:41] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [01:22:14] FIRING: [6x] ProbeDown: Service people1005:30443 has failed probes (http_annual_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:26:40] FIRING: KubernetesRsyslogDown: rsyslog on dse-k8s-worker1001:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=dse-k8s-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [01:27:14] RESOLVED: [6x] ProbeDown: Service people1005:30443 has failed probes (http_annual_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:36:40] RESOLVED: KubernetesRsyslogDown: rsyslog on dse-k8s-worker1001:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=dse-k8s-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [01:45:40] FIRING: KubernetesRsyslogDown: rsyslog on dse-k8s-worker1001:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=dse-k8s-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [01:50:40] RESOLVED: KubernetesRsyslogDown: rsyslog on dse-k8s-worker1001:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=dse-k8s-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T0200) [02:13:42] FIRING: [3x] JobUnavailable: Reduced availability for job bacula in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:18:42] FIRING: [3x] JobUnavailable: Reduced availability for job bacula in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:19:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12487675 (10phaultfinder) [02:34:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12487678 (10phaultfinder) [02:43:40] FIRING: KubernetesRsyslogDown: rsyslog on dse-k8s-worker1001:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=dse-k8s-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [02:54:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:55:00] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12487684 (10phaultfinder) [02:58:40] RESOLVED: KubernetesRsyslogDown: rsyslog on dse-k8s-worker1001:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=dse-k8s-worker1001 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [02:59:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [03:02:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [03:02:31] FIRING: ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:07:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [03:07:31] RESOLVED: ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [03:16:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [03:21:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [03:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:39:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12487726 (10phaultfinder) [03:57:43] PROBLEM - Host db1262 #page is DOWN: PING CRITICAL - Packet loss = 100% [03:58:46] !incodents [03:58:54] !incidents [03:58:55] 8708 (UNACKED) Host db1262 (paged) [03:58:55] 8707 (RESOLVED) [2x] ProbeDown sre () [03:58:55] 8706 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [03:58:55] 8705 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [03:58:56] 8704 (RESOLVED) Host db1158 (paged) [03:58:56] 8703 (RESOLVED) Host db1250 (paged) [03:58:56] 8702 (RESOLVED) Host db1237 (paged) [03:58:57] 8701 (RESOLVED) Host db1255 (paged) [03:58:57] 8700 (RESOLVED) Host db1257 (paged) [03:58:58] 8699 (RESOLVED) Host db1273 (paged) [03:58:58] 8698 (RESOLVED) Host es1049 (paged) [03:58:59] 8697 (RESOLVED) Host db1260 (paged) [03:58:59] 8696 (RESOLVED) Host db1157 (paged) [03:59:00] 8695 (RESOLVED) Host db1235 (paged) [03:59:00] 8694 (RESOLVED) Host db1267 (paged) [03:59:01] 8693 (RESOLVED) PHPFPMTooBusy sre (mw-api-ext main codfw) [03:59:01] 8692 (RESOLVED) Host pc1021 (paged) [03:59:02] 8691 (RESOLVED) Host es1042 (paged) [03:59:02] 8689 (RESOLVED) Host db1272 (paged) [03:59:13] good job sirenbot [04:01:01] RECOVERY - Host db1262 #page is UP: PING OK - Packet loss = 0%, RTA = 0.45 ms [04:02:01] PROBLEM - MariaDB Replica SQL: x4 #page on db1262 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [04:02:03] PROBLEM - MariaDB Event Scheduler x4 on db1262 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [04:02:13] PROBLEM - MariaDB Replica IO: x4 #page on db1262 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [04:02:57] PROBLEM - mysqld processes on db1262 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [04:03:23] PROBLEM - MariaDB Events x4 on db1262 is CRITICAL: CRITICAL - Failed to query events: ERROR 2002 (HY000): Cant connect to local server through socket /run/mysqld/mysqld.sock (2) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [04:03:35] PROBLEM - MariaDB read only x4 on db1262 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [04:04:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12487749 (10phaultfinder) [04:07:24] !list [04:07:42] !incidents [04:07:43] 8709 (UNACKED) db1262 (paged)/MariaDB Replica SQL: x4 (paged) [04:07:43] 8710 (UNACKED) db1262 (paged)/MariaDB Replica IO: x4 (paged) [04:07:43] 8708 (RESOLVED) Host db1262 (paged) [04:07:43] 8707 (RESOLVED) [2x] ProbeDown sre () [04:07:44] 8706 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [04:07:44] 8705 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [04:07:44] 8704 (RESOLVED) Host db1158 (paged) [04:07:45] 8703 (RESOLVED) Host db1250 (paged) [04:07:45] 8702 (RESOLVED) Host db1237 (paged) [04:07:46] 8701 (RESOLVED) Host db1255 (paged) [04:07:46] 8700 (RESOLVED) Host db1257 (paged) [04:07:47] 8699 (RESOLVED) Host db1273 (paged) [04:07:47] 8698 (RESOLVED) Host es1049 (paged) [04:07:48] 8697 (RESOLVED) Host db1260 (paged) [04:07:48] 8696 (RESOLVED) Host db1157 (paged) [04:07:49] 8695 (RESOLVED) Host db1235 (paged) [04:07:49] 8694 (RESOLVED) Host db1267 (paged) [04:07:50] 8693 (RESOLVED) PHPFPMTooBusy sre (mw-api-ext main codfw) [04:07:50] 8692 (RESOLVED) Host pc1021 (paged) [04:07:51] 8691 (RESOLVED) Host es1042 (paged) [04:07:51] 8689 (RESOLVED) Host db1272 (paged) [04:09:02] hey slyngs I got paged on the host down with 2 mins on the clock [04:09:17] shall we depool db1262 ? [04:09:23] Let's [04:09:48] I think it pages yesterday as well [04:09:51] PROBLEM - MariaDB Replica Lag: x4 #page on db1262 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [04:10:24] !ack 8709 [04:10:25] 8709 (ACKED) db1262 (paged)/MariaDB Replica SQL: x4 (paged) [04:10:32] !ack 8719 [04:10:32] Attempt to ack incident 8719 failed. [04:10:38] !ack 8710 [04:10:41] 8710 (ACKED) db1262 (paged)/MariaDB Replica IO: x4 (paged) [04:10:45] !incidents [04:10:45] 8709 (ACKED) db1262 (paged)/MariaDB Replica SQL: x4 (paged) [04:10:45] 8710 (ACKED) db1262 (paged)/MariaDB Replica IO: x4 (paged) [04:10:45] 8711 (UNACKED) db1262 (paged)/MariaDB Replica Lag: x4 (paged) [04:10:45] 8708 (RESOLVED) Host db1262 (paged) [04:10:46] 8707 (RESOLVED) [2x] ProbeDown sre () [04:10:46] 8706 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [04:10:46] 8705 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [04:10:47] 8704 (RESOLVED) Host db1158 (paged) [04:10:47] 8703 (RESOLVED) Host db1250 (paged) [04:10:48] 8702 (RESOLVED) Host db1237 (paged) [04:10:48] 8701 (RESOLVED) Host db1255 (paged) [04:10:49] 8700 (RESOLVED) Host db1257 (paged) [04:10:49] 8699 (RESOLVED) Host db1273 (paged) [04:10:50] 8698 (RESOLVED) Host es1049 (paged) [04:10:50] 8697 (RESOLVED) Host db1260 (paged) [04:10:51] 8696 (RESOLVED) Host db1157 (paged) [04:10:51] 8695 (RESOLVED) Host db1235 (paged) [04:10:52] 8694 (RESOLVED) Host db1267 (paged) [04:10:52] 8693 (RESOLVED) PHPFPMTooBusy sre (mw-api-ext main codfw) [04:10:53] 8692 (RESOLVED) Host pc1021 (paged) [04:10:53] 8691 (RESOLVED) Host es1042 (paged) [04:11:17] !log herron@cumin1004 START - Cookbook sre.mysql.depool depool db1262: depooling replica after host flap [04:11:49] hello ? [04:11:52] hello! [04:11:58] reading up [04:12:00] deja vu [04:12:34] Failed to connect to the database on attempt 1: (2003, "Can't connect to MySQL server on 'db1262.eqiad.wmnet' ([Errno 111] Connection refused)") [04:12:40] ==> CAUTION: we failed to check the connections 3 times, proceed anyway? [04:12:53] I haven't ran into that before, should I go ahead? [04:13:43] So if the host went down, Mariadb isn't startet automatically, so it makes sense [04:13:47] of it is a replica it should not cause a problem [04:14:04] ok, go-ing [04:14:08] !log herron@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1262: depooling replica after host flap [04:15:32] We should get the dbas a gift basket or something [04:15:43] hahah [04:15:59] Why are you on-call again? [04:16:14] I swapped with filippo [04:16:33] for today [04:16:54] Ah okay... Not that I mind :-) [04:17:18] herron: cookbook went well? [04:17:38] effie: yeah, passed [04:18:02] !incidents [04:18:03] 8709 (ACKED) db1262 (paged)/MariaDB Replica SQL: x4 (paged) [04:18:03] 8710 (ACKED) db1262 (paged)/MariaDB Replica IO: x4 (paged) [04:18:03] 8711 (UNACKED) db1262 (paged)/MariaDB Replica Lag: x4 (paged) [04:18:03] 8708 (RESOLVED) Host db1262 (paged) [04:18:04] 8707 (RESOLVED) [2x] ProbeDown sre () [04:18:04] 8706 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [04:18:04] 8705 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [04:18:05] 8704 (RESOLVED) Host db1158 (paged) [04:18:05] 8703 (RESOLVED) Host db1250 (paged) [04:18:06] 8702 (RESOLVED) Host db1237 (paged) [04:18:06] 8701 (RESOLVED) Host db1255 (paged) [04:18:07] 8700 (RESOLVED) Host db1257 (paged) [04:18:07] 8699 (RESOLVED) Host db1273 (paged) [04:18:08] 8698 (RESOLVED) Host es1049 (paged) [04:18:08] 8697 (RESOLVED) Host db1260 (paged) [04:18:09] 8696 (RESOLVED) Host db1157 (paged) [04:18:09] 8695 (RESOLVED) Host db1235 (paged) [04:18:10] 8694 (RESOLVED) Host db1267 (paged) [04:18:10] 8693 (RESOLVED) PHPFPMTooBusy sre (mw-api-ext main codfw) [04:18:11] 8692 (RESOLVED) Host pc1021 (paged) [04:18:11] 8691 (RESOLVED) Host es1042 (paged) [04:18:15] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [04:18:20] !ack 8711 [04:18:45] runbook has one more step -- please open a task to give dba a heads up https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=DBA [04:19:01] we can do this sure sure [04:19:11] cheers keith, thanks [04:19:42] I'll start typing the task [04:19:56] alrighty, lets not meet like this again for a while! [04:20:13] * effie stops writing a task [04:20:26] haahhaa yes indeed [04:20:48] I am still getting calls wtf [04:20:55] thanks effie slyngs, going to go offline now ✌️ [04:21:04] thank you! [04:21:30] !incidents [04:21:31] 8709 (ACKED) db1262 (paged)/MariaDB Replica SQL: x4 (paged) [04:21:31] 8710 (ACKED) db1262 (paged)/MariaDB Replica IO: x4 (paged) [04:21:31] 8711 (UNACKED) db1262 (paged)/MariaDB Replica Lag: x4 (paged) [04:21:31] 8708 (RESOLVED) Host db1262 (paged) [04:21:31] 8707 (RESOLVED) [2x] ProbeDown sre () [04:21:32] 8706 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [04:21:32] 8705 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [04:21:33] 8704 (RESOLVED) Host db1158 (paged) [04:21:33] 8703 (RESOLVED) Host db1250 (paged) [04:21:34] 8702 (RESOLVED) Host db1237 (paged) [04:21:34] 8701 (RESOLVED) Host db1255 (paged) [04:21:35] 8700 (RESOLVED) Host db1257 (paged) [04:21:35] 8699 (RESOLVED) Host db1273 (paged) [04:21:36] 8698 (RESOLVED) Host es1049 (paged) [04:21:36] 8697 (RESOLVED) Host db1260 (paged) [04:21:37] 8696 (RESOLVED) Host db1157 (paged) [04:21:37] 8695 (RESOLVED) Host db1235 (paged) [04:21:38] 8694 (RESOLVED) Host db1267 (paged) [04:21:38] 8693 (RESOLVED) PHPFPMTooBusy sre (mw-api-ext main codfw) [04:22:03] !ack 8711 [04:22:14] Ah, I did a list and then acked, apparently listing the incidents crashed sirenbot [04:23:17] ok I acked it on whatsapp [04:23:54] slyngs: we are ok then? [04:24:05] Yes, task is created [04:24:42] We'll have to fix Siren bot at some point, so it doesn't list so much [04:24:53] sure [04:25:38] great! thank you! [04:26:11] Talk to later, I'll try to get back to sleep :-) [04:31:05] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [04:34:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12487770 (10phaultfinder) [04:59:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12487785 (10phaultfinder) [05:04:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12487786 (10phaultfinder) [05:15:10] (03PS1) 10KartikMistry: Update recommendation-api to 2026-10-29-040822-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360229 (https://phabricator.wikimedia.org/T442253) [05:17:12] (03CR) 10KartikMistry: [C:03+2] Update recommendation-api to 2026-10-29-040822-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360229 (https://phabricator.wikimedia.org/T442253) (owner: 10KartikMistry) [05:18:42] (03Merged) 10jenkins-bot: Update recommendation-api to 2026-10-29-040822-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360229 (https://phabricator.wikimedia.org/T442253) (owner: 10KartikMistry) [05:20:36] !log kartik@deploy1003 helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [05:20:41] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [05:23:49] !log kartik@deploy1003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [05:27:20] !log kartik@deploy1003 helmfile [ml-serve-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' . [05:27:37] FIRING: CoreRouterInterfaceDown: Core router interface down - cr1-eqiad:et-1/1/2 (Transport: cr1-codfw:et-1/0/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [05:27:39] FIRING: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [05:30:22] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [05:36:20] (03PS1) 10Kevin Bazira: ml-services: fix timeouts on long TTS sections [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360232 (https://phabricator.wikimedia.org/T441983) [05:39:13] (03PS1) 10KartikMistry: Update MinT to 2026-10-29-053229-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360234 (https://phabricator.wikimedia.org/T442253) [05:41:16] (03CR) 10Kevin Bazira: [C:03+2] ml-services: fix timeouts on long TTS sections [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360232 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [05:42:38] (03CR) 10KartikMistry: [C:03+2] Update MinT to 2026-10-29-053229-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360234 (https://phabricator.wikimedia.org/T442253) (owner: 10KartikMistry) [05:43:18] (03Merged) 10jenkins-bot: ml-services: fix timeouts on long TTS sections [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360232 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [05:44:51] (03Merged) 10jenkins-bot: Update MinT to 2026-10-29-053229-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360234 (https://phabricator.wikimedia.org/T442253) (owner: 10KartikMistry) [05:45:25] !log kartik@deploy1003 helmfile [staging] START helmfile.d/services/machinetranslation: apply [05:46:21] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 233.48 ms [05:48:20] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [05:48:25] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [05:48:38] !log kartik@deploy1003 helmfile [staging] DONE helmfile.d/services/machinetranslation: apply [05:49:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12487817 (10phaultfinder) [05:49:55] !log kartik@deploy1003 helmfile [codfw] START helmfile.d/services/machinetranslation: apply [05:54:47] !log kartik@deploy1003 helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply [05:56:07] (03CR) 10Giuseppe Lavagetto: [C:03+1] rest-gateway: Update to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360002 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [05:58:18] (03CR) 10Giuseppe Lavagetto: [C:03+1] mw-videoscaler: Update to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360003 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [05:58:51] FIRING: Transit, peering or transport OUT traffic above 90% capacity - cr3-eqsin:xe-0/1/3 (Peering: Equinix (Wikimedia-SG1-IX-00 Singapore, ... [05:58:51] MAC filter)) #page - https://w.wiki/Gbyf - https://grafana.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?var-site=eqsin+prometheus%2Fops&var-device=cr3-eqsin:9804&var-interface=xe-0%2F1%2F3 - https://alerts.wikimedia.org/?q=alertname%3DTransitPeeringTransportOutSaturation [05:59:13] (03CR) 10Giuseppe Lavagetto: [C:03+1] mw-*: Update canary deployments to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360004 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [06:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T0600) [06:00:05] marostegui, cezmunsta, and federico3: gettimeofday() says it's time for Primary database switchover. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T0600) [06:00:57] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [06:01:09] (03CR) 10Giuseppe Lavagetto: [C:03+1] "Not strictly related to your change, but wouldn't it make sense to have the mesh image version be part of global-values.yaml instead, and " [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360005 (owner: 10RLazarus) [06:02:39] !ack 8712 [06:02:39] 8712 (ACKED) TransitPeeringTransportOutSaturation network sre (cr3-eqsin:9804 Peering: Equinix (Wikimedia-SG1-IX-00 Singapore, MAC filter) xe-0/1/3 gnmi eqsin) [06:03:05] !log kartik@deploy1003 helmfile [eqiad] START helmfile.d/services/machinetranslation: apply [06:04:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12487820 (10phaultfinder) [06:10:22] !log kartik@deploy1003 helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply [06:15:46] (03PS1) 10Marostegui: db1262: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1360237 (https://phabricator.wikimedia.org/T442800) [06:15:56] FIRING: [2x] TransportLinksInUsageNoRedundancy: eqsin inbound transports link usage is at redundancy capacity - https://wikitech.wikimedia.org/wiki/Network_monitoring#TransportLinksInUsageNoRedundancy - https://alerts.wikimedia.org/?q=alertname%3DTransportLinksInUsageNoRedundancy [06:17:30] (03CR) 10Marostegui: [C:03+2] db1262: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1360237 (https://phabricator.wikimedia.org/T442800) (owner: 10Marostegui) [06:18:42] FIRING: JobUnavailable: Reduced availability for job bacula in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:30:56] FIRING: [2x] TransportLinksInUsageNoRedundancy: eqsin inbound transports link usage is at redundancy capacity - https://wikitech.wikimedia.org/wiki/Network_monitoring#TransportLinksInUsageNoRedundancy - https://alerts.wikimedia.org/?q=alertname%3DTransportLinksInUsageNoRedundancy [06:33:51] RESOLVED: Transit, peering or transport OUT traffic above 90% capacity - cr3-eqsin:xe-0/1/3 (Peering: Equinix (Wikimedia-SG1-IX-00 Singapore, ... [06:33:51] MAC filter)) #page - https://w.wiki/Gbyf - https://grafana.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?var-site=eqsin+prometheus%2Fops&var-device=cr3-eqsin:9804&var-interface=xe-0%2F1%2F3 - https://alerts.wikimedia.org/?q=alertname%3DTransitPeeringTransportOutSaturation [06:35:48] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section x1 [06:36:17] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section x1 [06:36:23] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d8-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [06:37:57] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section x3 [06:38:14] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section x3 [06:40:24] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section x4 [06:40:56] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section x4 [06:40:56] RESOLVED: [2x] TransportLinksInUsageNoRedundancy: eqsin inbound transports link usage is at redundancy capacity - https://wikitech.wikimedia.org/wiki/Network_monitoring#TransportLinksInUsageNoRedundancy - https://alerts.wikimedia.org/?q=alertname%3DTransportLinksInUsageNoRedundancy [06:43:28] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section es6 [06:43:59] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section es6 [06:44:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12487939 (10phaultfinder) [06:46:00] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section es7 [06:46:43] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section es7 [06:47:17] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s6 [06:47:36] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s6 [06:49:21] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s5 [06:49:49] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s5 [06:50:17] (03PS1) 10Ryan Kemper: airflow-experiment-platform: upgrade to Airflow 3.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360245 (https://phabricator.wikimedia.org/T442431) [06:50:30] PROBLEM - OSPF status on cr4-ulsfo is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:50:30] PROBLEM - OSPF status on cr2-drmrs is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:50:36] PROBLEM - OSPF status on cr2-codfw is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:50:42] PROBLEM - OSPF status on cr3-eqsin is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:50:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [06:50:47] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s2 [06:51:00] PROBLEM - OSPF status on cr2-esams is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:51:09] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s2 [06:51:30] RECOVERY - OSPF status on cr4-ulsfo is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:51:30] RECOVERY - OSPF status on cr2-drmrs is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:51:36] RECOVERY - OSPF status on cr2-codfw is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:51:42] RECOVERY - OSPF status on cr3-eqsin is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:51:46] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s3 [06:52:00] RECOVERY - OSPF status on cr2-esams is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [06:52:02] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s3 [06:52:10] FIRING: [4x] BFDdown: BFD session down between cr2-codfw and 208.80.153.214 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [06:52:24] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s7 [06:52:39] FIRING: [7x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [06:52:41] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s7 [06:53:07] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s8 [06:53:21] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s8 [06:53:41] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s4 [06:53:58] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s4 [06:55:37] !log marostegui@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section s1 [06:55:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [06:56:14] !log marostegui@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section s1 [06:56:21] !log DB circular replication has been disabled everywhere T436501 [06:56:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:56:27] T436501: Post DC switchover: Disable eqiad -> codfw replication - https://phabricator.wikimedia.org/T436501 [06:57:10] RESOLVED: [5x] BFDdown: BFD session down between cr2-codfw and 208.80.153.214 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [06:57:39] FIRING: [7x] CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [07:00:04] Amir1, urbanecm, and awight: #bothumor I � Unicode. All rise for UTC morning backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T0700). [07:00:04] No Gerrit patches in the queue for this window AFAICS. [07:04:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12488034 (10phaultfinder) [07:09:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12488042 (10phaultfinder) [07:13:15] o/ [07:13:27] well, there are no patches to deploy :) [07:16:47] (03CR) 10Elukey: [C:03+2] registryctl: accept relative values for --older-than [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1359879 (https://phabricator.wikimedia.org/T437297) (owner: 10Ahmon Dancy) [07:18:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [07:19:43] (03Merged) 10jenkins-bot: registryctl: accept relative values for --older-than [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1359879 (https://phabricator.wikimedia.org/T437297) (owner: 10Ahmon Dancy) [07:23:43] lovely day :) [07:23:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [07:33:58] (03CR) 10JMeybohm: [C:03+1] envoy: Update to v1.40.1 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1359939 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [07:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:36:29] (03CR) 10JMeybohm: [C:03+1] kubernetes: Update default envoy version to 1.40.1 [puppet] - 10https://gerrit.wikimedia.org/r/1360006 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [07:37:33] (03PS1) 10Kevin Bazira: ml-services: revert TTS timeout increases [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360331 (https://phabricator.wikimedia.org/T441983) [07:39:20] (03CR) 10Jelto: [C:03+2] deployment_server::deploy_service::service_catalog: Add eventgate-all (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359828 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [07:39:33] (03CR) 10Jelto: [C:03+2] deployment_server::deploy_service::service_catalog: Add eventgate [puppet] - 10https://gerrit.wikimedia.org/r/1359827 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [07:40:11] (03CR) 10Kevin Bazira: [C:03+2] ml-services: revert TTS timeout increases [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360331 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [07:41:58] (03Merged) 10jenkins-bot: ml-services: revert TTS timeout increases [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360331 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [07:43:35] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [07:43:39] !log kevinbazira@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' . [07:58:42] RESOLVED: JobUnavailable: Reduced availability for job bacula in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:58:58] (03CR) 10Jelto: [C:03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1359761 (https://phabricator.wikimedia.org/T392465) (owner: 10Hashar) [07:59:47] (03CR) 10Jelto: [C:03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1359762 (https://phabricator.wikimedia.org/T392465) (owner: 10Hashar) [08:00:05] hashar and jelto: Deploy window Switch Gerrit from Java 17 to Java 21 (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T0800) [08:00:26] 👋 I'm around [08:00:48] (03PS1) 10Bartosz Wójtowicz: ml: add vLLM 0.32 image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360334 (https://phabricator.wikimedia.org/T442138) [08:00:51] jelto: o/ :) [08:01:14] I have split the operations in small changes, the first one is to have Java 21 packages installed on all hosts [08:01:35] it does not change anything else, the gerrit systemd unit points to the versioned path to java [08:01:51] oh also: Guten Morgen! :) [08:02:21] Bonjour :) great than I'll first merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1359761 and run puppet on all gerrit hosts? [08:02:31] correct [08:03:09] not sure whether there is a cumin way to do it :b [08:03:40] (03CR) 10Jelto: [C:03+2] gerrit: add Java 21 on hosts [puppet] - 10https://gerrit.wikimedia.org/r/1359761 (https://phabricator.wikimedia.org/T392465) (owner: 10Hashar) [08:04:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12488079 (10phaultfinder) [08:08:17] (give me a sec, it needs two puppet runs for every host because of some dependency error in openjdk-21-dbg) [08:10:36] what's the error? [08:12:35] E: Unable to locate package openjdk-21-dbg, but this resolved after the second run. I guess there is some dependency missing in the puppet code to first setup the new component and then install the package [08:13:22] possibly yes [08:15:15] java21 is installed on all three gerrit hosts and puppet is happy now [08:15:40] yeah, looking at the puppet logs on gerrit1003, the refresh of the apt source comes after it attempts to install the -dbg package [08:16:19] hashar: the next patch is https://gerrit.wikimedia.org/r/c/operations/puppet/+/1359762/1 ? [08:16:28] the joices of Puppet [08:16:57] jelto: yeah, let me double check the java path exists :) [08:17:13] * jelto waits [08:17:29] jelto: it is there :) [08:17:44] I'll proceed with merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1359762 then [08:18:14] that will change the unit ExecStart with the new paths, and then I guess we gotta restart Gerrit on the spare/replica ( gerrit2002 / gerrit2003 ) [08:18:39] (03CR) 10Jelto: [C:03+2] gerrit: switch replica and spare to Java 21 [puppet] - 10https://gerrit.wikimedia.org/r/1359762 (https://phabricator.wikimedia.org/T392465) (owner: 10Hashar) [08:21:19] (03CR) 10Slyngshede: [C:03+1] "I'll be working on updating the switch-over documentation and will have this removed from future swith-overs." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359940 (https://phabricator.wikimedia.org/T289745) (owner: 10Krinkle) [08:23:35] hashar: puppet run on gerrit replicas/spare and the gerrit.config changed :) [08:24:40] ah [08:24:57] (03PS1) 10Muehlenhoff: De-confuse the Jenkins repository config [puppet] - 10https://gerrit.wikimedia.org/r/1360338 (https://phabricator.wikimedia.org/T442030) [08:25:22] oh I see the gerrit.config is for javaHome, and I am pretty sure we do not use it [08:25:32] it is set for the upstream provided startup script which we do not use [08:25:58] so I guess we can restart gerrit on gerrit2002/gerrit2003 and verify [08:26:19] yeah the gerrit.service unit also changed, sorry for the confusion. Are you restarting or should I ? [08:28:16] I'll do it [08:28:57] Oct 29 08:28:35 gerrit2003 java[1007723]: INFO: Using MemorySegmentIndexInput with Java 21; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false [08:28:57] fun [08:29:17] it is an info, I did not caught that one locally. I am keeping note of it for later [08:30:08] !log imported Jenkins 2.580.2 to thirdparty/jenkins for trixie-wikimedia T442030 [08:30:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:30:35] !log Restarted Gerrit on gerrit2002/gerrit2003 (spare/replica) to switch to Java 21 | T392465 [08:30:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:30:40] T392465: Switch Gerrit from Java 17 to Java 21 - https://phabricator.wikimedia.org/T392465 [08:30:53] jelto: at least gitiles works ( https://gerrit-replica.wikimedia.org/r/plugins/gitiles/ ) [08:30:57] and the daemons are running [08:31:05] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [08:32:52] yes looks good, at least metrics and gitiles. So should we proceed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1359763 ? [08:34:12] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reimage for host lvs6003.drmrs.wmnet with OS trixie [08:34:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [08:35:09] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs6003.drmrs.wmnet (push config only) [08:35:15] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6003.drmrs.wmnet (push config only) [08:37:57] jelto: yeah we can do the primary :) [08:38:25] !log jelto@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on gerrit1003.wikimedia.org with reason: Java upgrade [08:38:46] (03CR) 10Jelto: [C:03+2] gerrit: switch primary to Java 21 [puppet] - 10https://gerrit.wikimedia.org/r/1359763 (https://phabricator.wikimedia.org/T392465) (owner: 10Hashar) [08:38:58] !log jelto@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on 14 hosts with reason: Java upgrade [08:39:40] ok merging and running puppet at the moment [08:39:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [08:41:54] hashar: gerrit1003 is ready for a restart [08:42:30] doing it :) [08:42:37] !log restarting Gerrit [08:42:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:42:48] oh it went fast this time [08:43:47] yeah gerrit is up for me [08:44:56] that is great [08:45:06] as for the issue we might encounter, well probably none [08:45:16] but we might get some performance gains / better memory usage here and there [08:45:21] honestly I don't know [08:45:29] I'll monitor it for the rest of the day [08:45:50] the good news is it opens the path to upgrade Gerrit :] [08:46:02] thank you jelto ! [08:46:30] Thank you too. I'll monitor the metrics a bit more, there are still some 5xx requests [08:47:38] (03PS1) 10Bartosz Wójtowicz: ml-services: update revscoring image tag to 2026-10-29-082429-publish [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360346 [08:47:43] 10ops-codfw, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: wikikube-worker2280 unreachable - https://phabricator.wikimedia.org/T423395#12488177 (10MLechvien-WMF) @Jhancock.wm do you have updates on this? [08:48:56] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs6003.drmrs.wmnet (remove config only) [08:49:03] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6003.drmrs.wmnet (remove config only) [08:50:31] jelto: the 503 might be issued by Apache displaying the "Gerrit is under maintenance" while Gerrit is not reachable [08:51:56] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on lvs6003.drmrs.wmnet with reason: host reimage [08:53:33] I have marked the task resolved ( T392465 ) [08:53:34] T392465: Switch Gerrit from Java 17 to Java 21 - https://phabricator.wikimedia.org/T392465 [08:54:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12488197 (10phaultfinder) [08:56:08] (03CR) 10Hashar: [C:03+1] De-confuse the Jenkins repository config (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1360338 (https://phabricator.wikimedia.org/T442030) (owner: 10Muehlenhoff) [08:56:58] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs6003.drmrs.wmnet with reason: host reimage [08:58:19] hashar: yes all 5xx recovered :) [08:59:54] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12488233 (10phaultfinder) [09:02:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [09:04:06] (03CR) 10Kevin Bazira: [C:03+1] ml-services: update revscoring image tag to 2026-10-29-082429-publish [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360346 (owner: 10Bartosz Wójtowicz) [09:05:02] jelto: I guess we could invent a new 5xx error: "under maintenance" [09:05:29] which I guess is covered by 503 already [09:07:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [09:10:26] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 66%, RTA = 4566.54 ms [09:11:19] (03CR) 10Bartosz Wójtowicz: [C:03+2] ml-services: update revscoring image tag to 2026-10-29-082429-publish [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360346 (owner: 10Bartosz Wójtowicz) [09:13:28] (03Merged) 10jenkins-bot: ml-services: update revscoring image tag to 2026-10-29-082429-publish [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360346 (owner: 10Bartosz Wójtowicz) [09:15:09] (03CR) 10Muehlenhoff: [C:03+2] De-confuse the Jenkins repository config [puppet] - 10https://gerrit.wikimedia.org/r/1360338 (https://phabricator.wikimedia.org/T442030) (owner: 10Muehlenhoff) [09:15:28] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 250.35 ms [09:16:26] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs6003.drmrs.wmnet with OS trixie [09:17:38] !log installing ca-certificates updates on Bookworm [09:17:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:20:07] PROBLEM - Host db1228 #page is DOWN: PING CRITICAL - Packet loss = 100% [09:20:42] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [09:22:00] !incidents [09:22:00] 8709 (ACKED) db1262 (paged)/MariaDB Replica SQL: x4 (paged) [09:22:00] 8710 (ACKED) db1262 (paged)/MariaDB Replica IO: x4 (paged) [09:22:00] 8711 (ACKED) db1262 (paged)/MariaDB Replica Lag: x4 (paged) [09:22:01] 8713 (UNACKED) Host db1228 (paged) [09:22:01] 8712 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr3-eqsin:9804 Peering: Equinix (Wikimedia-SG1-IX-00 Singapore, MAC filter) xe-0/1/3 gnmi eqsin) [09:22:01] 8708 (RESOLVED) Host db1262 (paged) [09:22:01] 8707 (RESOLVED) [2x] ProbeDown sre () [09:22:02] 8706 (RESOLVED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [09:22:02] 8705 (RESOLVED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [09:22:03] 8704 (RESOLVED) Host db1158 (paged) [09:22:03] 8703 (RESOLVED) Host db1250 (paged) [09:22:04] 8702 (RESOLVED) Host db1237 (paged) [09:22:04] 8701 (RESOLVED) Host db1255 (paged) [09:22:05] 8700 (RESOLVED) Host db1257 (paged) [09:22:05] 8699 (RESOLVED) Host db1273 (paged) [09:22:06] 8698 (RESOLVED) Host es1049 (paged) [09:22:06] 8697 (RESOLVED) Host db1260 (paged) [09:22:07] 8696 (RESOLVED) Host db1157 (paged) [09:22:07] 8695 (RESOLVED) Host db1235 (paged) [09:22:08] 8694 (RESOLVED) Host db1267 (paged) [09:22:08] 8693 (RESOLVED) PHPFPMTooBusy sre (mw-api-ext main codfw) [09:22:13] (03PS1) 10Vgutierrez: hieradata: Depool lvs6002 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1360349 (https://phabricator.wikimedia.org/T401832) [09:22:30] !ack db1228 ( [09:22:33] !ack db1228 [09:22:41] sirenbot flooded off it seems :/ [09:22:41] right it borks [09:22:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [09:22:49] you need the incident number not the hostname [09:22:54] and yes.. sirenbot is out :( [09:23:09] vgutierrez: yes sorry, my phone was ringing again [09:23:46] !ack 8713 [09:23:46] 8713 (ACKED) Host db1228 (paged) [09:23:50] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360349 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [09:23:50] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [09:24:58] (03PS1) 10Marostegui: db1228: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1360351 [09:25:11] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [09:27:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [09:27:56] !log jiji@cumin2003 START - Cookbook sre.mysql.depool depool db1228: Depooling replica - T442815 [09:28:01] T442815: db1228 crashed - https://phabricator.wikimedia.org/T442815 [09:28:04] (03PS2) 10Slyngshede: db1228: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1360351 (https://phabricator.wikimedia.org/T442815) (owner: 10Marostegui) [09:28:09] !log jiji@cumin2003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1228: Depooling replica - T442815 [09:28:47] (03PS1) 10Blake: bgp: increase trigger time for NodeBGPSessionStatusNotEstabilshed [alerts] - 10https://gerrit.wikimedia.org/r/1360350 [09:28:59] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . [09:29:12] (03CR) 10CI reject: [V:04-1] db1228: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1360351 (https://phabricator.wikimedia.org/T442815) (owner: 10Marostegui) [09:29:14] (03CR) 10Effie Mouzeli: [C:03+1] db1228: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1360351 (https://phabricator.wikimedia.org/T442815) (owner: 10Marostegui) [09:29:47] (03PS3) 10Slyngshede: db1228: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1360351 (https://phabricator.wikimedia.org/T442815) (owner: 10Marostegui) [09:30:22] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [09:31:12] (03CR) 10Slyngshede: [C:03+2] db1228: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1360351 (https://phabricator.wikimedia.org/T442815) (owner: 10Marostegui) [09:31:50] 10ops-eqiad, 06DBA, 06DC-Ops: db1228 crashed - https://phabricator.wikimedia.org/T442815#12488342 (10Marostegui) @VRiley-WMF @wiki_willy I've added this crash to Dell's thread. This host has a long history: T435891 T436743 T430934. Nothing on logs as usual. [09:32:35] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [09:32:58] !log dpogorzelski@cumin1004 conftool action : set/pooled=true; selector: dnsdisc=inference,name=eqiad [09:33:06] !log dpogorzelski@cumin1004 conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-ml-serve,name=eqiad [09:33:16] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [09:34:25] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . [09:36:08] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [09:36:29] (03CR) 10Daniel Kertesz: [C:03+1] hieradata: Depool lvs6002 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1360349 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [09:36:41] (03CR) 10Vgutierrez: [C:03+2] hieradata: Depool lvs6002 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1360349 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [09:37:24] !log installing Bind security updates [09:37:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:38:24] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [09:39:04] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 50%, RTA = 3455.22 ms [09:40:42] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs6002.*} and A:liberica [09:41:00] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs6002.*} and A:liberica [09:41:33] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . [09:42:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [09:44:06] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 227.74 ms [09:45:36] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [09:46:52] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reimage for host lvs6002.drmrs.wmnet with OS trixie [09:47:23] (03PS1) 10JavierMonton: flink: script for puppet deployments [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360359 (https://phabricator.wikimedia.org/T442341) [09:47:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [09:47:49] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs6002.drmrs.wmnet (push config only) [09:47:54] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6002.drmrs.wmnet (push config only) [09:48:22] (03CR) 10JavierMonton: spiderpig: Add pageview-trending-relative (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359776 (https://phabricator.wikimedia.org/T442341) (owner: 10JavierMonton) [09:48:44] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [09:49:24] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . [09:49:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12488401 (10phaultfinder) [09:50:36] (03PS2) 10JavierMonton: spiderpig: Add pageview-trending-relative [puppet] - 10https://gerrit.wikimedia.org/r/1359776 (https://phabricator.wikimedia.org/T442341) [09:52:10] (03PS1) 10Hashar: Merge branch 'upstream/stable-3.11' into wmf/stable-3.11 [software/gerrit] (wmf/stable-3.11) - 10https://gerrit.wikimedia.org/r/1360360 (https://phabricator.wikimedia.org/T379714) [09:52:15] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [09:53:55] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [09:59:29] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T1000) [10:01:14] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs6002.drmrs.wmnet (remove config only) [10:01:21] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6002.drmrs.wmnet (remove config only) [10:03:22] (03CR) 10Klausman: ml: add vLLM 0.32 image (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360334 (https://phabricator.wikimedia.org/T442138) (owner: 10Bartosz Wójtowicz) [10:04:14] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on lvs6002.drmrs.wmnet with reason: host reimage [10:04:27] (03PS1) 10Vgutierrez: purged: Disable kafka events channel [puppet] - 10https://gerrit.wikimedia.org/r/1360365 (https://phabricator.wikimedia.org/T442513) [10:04:39] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [10:05:29] PROBLEM - Host mr1-eqsin.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [10:05:40] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [10:06:36] (03PS2) 10Vgutierrez: purged: Disable kafka events channel [puppet] - 10https://gerrit.wikimedia.org/r/1360365 (https://phabricator.wikimedia.org/T442513) [10:06:47] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360365 (https://phabricator.wikimedia.org/T442513) (owner: 10Vgutierrez) [10:07:15] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs6002.drmrs.wmnet with reason: host reimage [10:07:32] (03PS1) 10Elukey: Release version 0.0.23 [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1360366 [10:08:33] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . [10:10:32] RECOVERY - Host mr1-eqsin.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 273.32 ms [10:10:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:10:56] (03CR) 10Daniel Kertesz: [C:03+1] purged: Disable kafka events channel [puppet] - 10https://gerrit.wikimedia.org/r/1360365 (https://phabricator.wikimedia.org/T442513) (owner: 10Vgutierrez) [10:11:37] !log bwojtowicz@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [10:11:46] (03CR) 10Elukey: [C:03+2] Release version 0.0.23 [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1360366 (owner: 10Elukey) [10:12:39] !log bwojtowicz@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [10:13:57] !log bwojtowicz@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . [10:14:37] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1206 - https://phabricator.wikimedia.org/T442039#12488474 (10MoritzMuehlenhoff) 05Resolved→03Open The server is down since 1d 18h 58m 12s, did it break again after the firmware update? [10:14:50] (03Merged) 10jenkins-bot: Release version 0.0.23 [docker-images/docker-report] - 10https://gerrit.wikimedia.org/r/1360366 (owner: 10Elukey) [10:15:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:16:31] (03CR) 10Elukey: Add a systemd timer to periodically drop the old MediaWiki images (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359287 (https://phabricator.wikimedia.org/T437297) (owner: 10Elukey) [10:17:04] (03PS2) 10Elukey: Add a systemd timer to periodically drop the old MediaWiki images [puppet] - 10https://gerrit.wikimedia.org/r/1359287 (https://phabricator.wikimedia.org/T437297) [10:18:52] (03PS1) 10STran: Enable SuggestedInvestigations case generation on ptwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360367 (https://phabricator.wikimedia.org/T442582) [10:19:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12488498 (10phaultfinder) [10:20:20] (03PS7) 10Brouberol: admin/data: add spiderpig to the airflow users groups [puppet] - 10https://gerrit.wikimedia.org/r/1358221 (https://phabricator.wikimedia.org/T412941) [10:23:16] (03CR) 10Blake: [C:03+2] mediawiki-common: enable k8s native sidecars. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356812 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:24:22] !log installing Unbound security updates [10:24:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:25:15] (03Merged) 10jenkins-bot: mediawiki-common: enable k8s native sidecars. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356812 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:25:50] (03CR) 10Elukey: Add a systemd timer to periodically drop the old MediaWiki images (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359287 (https://phabricator.wikimedia.org/T437297) (owner: 10Elukey) [10:26:04] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs6002.drmrs.wmnet with OS trixie [10:26:59] !log blake@deploy1003 Started scap sync-world: Enable non-batch k8s native sidecars (T417800) [10:27:05] T417800: Implement proper sidecar container support in mediawiki pods - https://phabricator.wikimedia.org/T417800 [10:28:59] 10ops-eqiad, 06DBA, 06DC-Ops: db1228 crashed - https://phabricator.wikimedia.org/T442815#12488526 (10CWilliams-WMF) Not sure if relevant, but: ` [R/W] [R/W] [Key = iDRAC.Embedded.1#CPU1Temp] CP... [10:30:43] (03CR) 10Bartosz Wójtowicz: ml: add vLLM 0.32 image (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360334 (https://phabricator.wikimedia.org/T442138) (owner: 10Bartosz Wójtowicz) [10:30:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:34:20] !log blake@deploy1003 Rolling back deployment [10:34:30] !log installing Apache httpd security updates [10:34:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:34:52] 10SRE-swift-storage, 10MediaWiki-File-management, 06MediaWiki-Media-Platform-Team: Unknown error while undeleting a file on Wikimedia Commons - https://phabricator.wikimedia.org/T437114#12488575 (10JTweed-WMF) p:05Medium→03Low [10:35:21] 10SRE-swift-storage, 10MediaWiki-File-management, 06MediaWiki-Media-Platform-Team: Unknown error while undeleting a file on Wikimedia Commons - https://phabricator.wikimedia.org/T437114#12488579 (10JTweed-WMF) p:05Low→03Lowest [10:35:56] (03PS1) 10Blake: mw-cron: fix duplicate features stanza [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360369 (https://phabricator.wikimedia.org/T417800) [10:36:23] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d8-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:36:30] (03CR) 10Blake: [C:03+2] mw-cron: fix duplicate features stanza [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360369 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:36:55] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host apt-staging2001.codfw.wmnet [10:37:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 22.39% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:37:17] (03CR) 10Mszwarc: [C:04-1] "Not ready to be merged yet; `ptwiki` needs to have SI tables created." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360367 (https://phabricator.wikimedia.org/T442582) (owner: 10STran) [10:37:49] (03Merged) 10jenkins-bot: mw-cron: fix duplicate features stanza [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360369 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake) [10:38:16] (03PS1) 10Vgutierrez: Revert "hieradata: Depool lvs6002 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360371 [10:38:58] (03CR) 10Ladsgroup: "recheck" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360000 (https://phabricator.wikimedia.org/T442669) (owner: 10Ladsgroup) [10:39:01] (03PS2) 10Vgutierrez: Revert "hieradata: Depool lvs6002 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360371 (https://phabricator.wikimedia.org/T401832) [10:39:08] (03CR) 10Ladsgroup: "recheck" [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359999 (https://phabricator.wikimedia.org/T442669) (owner: 10Ladsgroup) [10:39:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12488593 (10phaultfinder) [10:39:43] (03CR) 10Daniel Kertesz: [C:03+1] Revert "hieradata: Depool lvs6002 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360371 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [10:40:23] !log blake@deploy1003 Failed scap sync-world: Enable non-batch k8s native sidecars (T417800) (duration: 14m 32s) [10:40:30] T417800: Implement proper sidecar container support in mediawiki pods - https://phabricator.wikimedia.org/T417800 [10:40:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:40:44] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt-staging2001.codfw.wmnet [10:41:15] !log blake@deploy1003 Started scap sync-world: Enable non-batch k8s native sidecars (T417800) [10:42:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 21.18% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:42:28] PROBLEM - people.wikimedia.org requires authentication on people2004 is CRITICAL: connect to address 10.192.32.28 and port 443: Connection refused https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [10:42:50] PROBLEM - people.wikimedia.org tls expiry on people2004 is CRITICAL: connect to address 10.192.32.28 and port 443: Connection refused https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [10:43:16] PROBLEM - people.wikimedia.org tls expiry on people1005 is CRITICAL: connect to address 10.64.32.95 and port 443: Connection refused https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [10:43:20] PROBLEM - people.wikimedia.org requires authentication on people1005 is CRITICAL: connect to address 10.64.32.95 and port 443: Connection refused https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [10:44:06] FIRING: [2x] ProbeDown: Service people1005:443 has failed probes (http_people_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:45:29] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360367 (https://phabricator.wikimedia.org/T442582) (owner: 10STran) [10:45:36] (03CR) 10STran: "👍 Yep, wanted something to put on the deployment calendar. Will create tables as part of deployment." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360367 (https://phabricator.wikimedia.org/T442582) (owner: 10STran) [10:46:21] (03CR) 10Vgutierrez: [C:03+2] Revert "hieradata: Depool lvs6002 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360371 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [10:46:37] (03PS1) 10Ozge: knative-serving: Allow port 8112 in the activator ingress policy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360372 (https://phabricator.wikimedia.org/T442150) [10:47:50] RECOVERY - people.wikimedia.org tls expiry on people2004 is OK: OK - Certificate peopleweb.discovery.wmnet will expire on Fri 13 Nov 2026 07:01:00 AM GMT +0000. https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [10:48:28] RECOVERY - people.wikimedia.org requires authentication on people2004 is OK: HTTP OK: Status line output matched HTTP/1.1 302 - 626 bytes in 0.186 second response time https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [10:48:49] !log blake@deploy1003 Finished scap sync-world: Enable non-batch k8s native sidecars (T417800) (duration: 07m 55s) [10:48:56] T417800: Implement proper sidecar container support in mediawiki pods - https://phabricator.wikimedia.org/T417800 [10:49:07] FIRING: [2x] ProbeDown: Service people1005:443 has failed probes (http_people_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:49:09] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs6002.*} and A:liberica [10:49:16] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs6002.*} and A:liberica [10:50:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 23.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:51:25] (03PS1) 10Vgutierrez: hieradata: Depool lvs6001 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1360373 (https://phabricator.wikimedia.org/T401832) [10:52:39] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360373 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [10:54:55] (03PS1) 10Ladsgroup: Upgrading symfony/mailer and symfony/mime (v7.4.19 => v7.4.22) [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360375 (https://phabricator.wikimedia.org/T442768) [10:55:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 23.45% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:55:31] (03PS2) 10Ladsgroup: ParserOutputAccess: Don't load dirty entry if not needed [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360000 (https://phabricator.wikimedia.org/T442669) [10:55:46] (03CR) 10Daniel Kertesz: [C:03+1] hieradata: Depool lvs6001 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1360373 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [10:56:10] (03CR) 10Fabfur: [C:03+1] hieradata: Depool lvs6001 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1360373 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [10:56:15] (03CR) 10Vgutierrez: [C:03+2] hieradata: Depool lvs6001 before reimaging [puppet] - 10https://gerrit.wikimedia.org/r/1360373 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [10:57:39] FIRING: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [10:58:16] RECOVERY - people.wikimedia.org tls expiry on people1005 is OK: OK - Certificate peopleweb.discovery.wmnet will expire on Fri 13 Nov 2026 06:53:00 AM GMT +0000. https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [10:58:20] RECOVERY - people.wikimedia.org requires authentication on people1005 is OK: HTTP OK: Status line output matched HTTP/1.1 302 - 626 bytes in 0.059 second response time https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [10:58:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:59:06] RESOLVED: [2x] ProbeDown: Service people1005:443 has failed probes (http_people_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [10:59:15] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs6001.*} and A:liberica [10:59:33] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs6001.*} and A:liberica [11:02:21] (03PS3) 10Ladsgroup: ParserOutputAccess: Don't load dirty entry if not needed [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360000 (https://phabricator.wikimedia.org/T442669) [11:03:14] (03PS4) 10Ladsgroup: ParserOutputAccess: Don't load dirty entry if not needed [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360000 (https://phabricator.wikimedia.org/T442669) [11:06:11] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reimage for host lvs6001.drmrs.wmnet with OS trixie [11:07:03] (03CR) 10Mszwarc: [C:03+1] "Then, I'll remove the -1 vote, so that nothing prevents merging when the tables are there." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360367 (https://phabricator.wikimedia.org/T442582) (owner: 10STran) [11:07:08] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs6001.drmrs.wmnet (push config only) [11:07:13] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6001.drmrs.wmnet (push config only) [11:07:54] (03CR) 10DCausse: [C:03+1] "lgtm," [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359948 (https://phabricator.wikimedia.org/T434004) (owner: 10Tjones) [11:08:45] (03CR) 10JMeybohm: [C:03+1] job: enable switch for k8s native sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357484 (https://phabricator.wikimedia.org/T386694) (owner: 10Blake) [11:09:56] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12488743 (10phaultfinder) [11:11:12] jouncebot: nowandnext [11:11:12] No deployments scheduled for the next 0 hour(s) and 48 minute(s) [11:11:12] In 0 hour(s) and 48 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T1200) [11:12:36] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360375 (https://phabricator.wikimedia.org/T442768) (owner: 10Ladsgroup) [11:12:37] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360000 (https://phabricator.wikimedia.org/T442669) (owner: 10Ladsgroup) [11:13:35] (03CR) 10Btullis: [C:03+1] "Looks good to me." [puppet] - 10https://gerrit.wikimedia.org/r/1358221 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [11:13:50] (03PS1) 10Gkyziridis: ml-services: place edit-check on 64G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360376 (https://phabricator.wikimedia.org/T441680) [11:14:36] (03PS3) 10Blake: job: enable switch for k8s native sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357484 (https://phabricator.wikimedia.org/T386694) [11:16:00] (03PS4) 10Blake: job: enable switch for k8s native sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357484 (https://phabricator.wikimedia.org/T386694) [11:16:47] (03Merged) 10jenkins-bot: Upgrading symfony/mailer and symfony/mime (v7.4.19 => v7.4.22) [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360375 (https://phabricator.wikimedia.org/T442768) (owner: 10Ladsgroup) [11:16:57] (03Merged) 10jenkins-bot: ParserOutputAccess: Don't load dirty entry if not needed [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360000 (https://phabricator.wikimedia.org/T442669) (owner: 10Ladsgroup) [11:18:28] (03CR) 10Blake: [C:03+2] job: enable switch for k8s native sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357484 (https://phabricator.wikimedia.org/T386694) (owner: 10Blake) [11:18:31] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1360375|Upgrading symfony/mailer and symfony/mime (v7.4.19 => v7.4.22) (T442768)]] (1.48.0-wmf.1), [[gerrit:1360000|ParserOutputAccess: Don't load dirty entry if not needed (T442669)]] (1.48.0-wmf.1) [11:18:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [11:20:14] !log upload purged 0.26 to trixie-wikimedia (apt.wm.o ) - T442736 [11:20:17] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs6001.drmrs.wmnet (remove config only) [11:20:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:20:19] T442736: Stop using the deprecated kafka Events channel on purged - https://phabricator.wikimedia.org/T442736 [11:20:24] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs6001.drmrs.wmnet (remove config only) [11:20:42] (03Merged) 10jenkins-bot: job: enable switch for k8s native sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357484 (https://phabricator.wikimedia.org/T386694) (owner: 10Blake) [11:22:41] !log cwilliams@cumin1004 START - Cookbook sre.backup.offsite [11:22:41] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.backup.offsite (exit_code=0) [11:23:11] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on lvs6001.drmrs.wmnet with reason: host reimage [11:23:15] !log ladsgroup@deploy1003 ladsgroup: Backport for [[gerrit:1360375|Upgrading symfony/mailer and symfony/mime (v7.4.19 => v7.4.22) (T442768)]] (1.48.0-wmf.1), [[gerrit:1360000|ParserOutputAccess: Don't load dirty entry if not needed (T442669)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:23:38] (03CR) 10Joal: [C:03+1] deployment_server: Absent the datasets-config kubeconfigs [puppet] - 10https://gerrit.wikimedia.org/r/1359809 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [11:23:52] (03CR) 10Joal: [C:03+1] deployment_server: Remove the datasets-config kubeconfigs [puppet] - 10https://gerrit.wikimedia.org/r/1359810 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [11:23:59] (03Abandoned) 10A-pizzata: EventStreamConfig: Add time to persist config for page_change [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359824 (https://phabricator.wikimedia.org/T440968) (owner: 10A-pizzata) [11:26:40] (03CR) 10Vgutierrez: [C:03+2] purged: Disable kafka events channel [puppet] - 10https://gerrit.wikimedia.org/r/1360365 (https://phabricator.wikimedia.org/T442513) (owner: 10Vgutierrez) [11:27:19] !log ladsgroup@deploy1003 ladsgroup: Continuing with deployment [11:28:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [11:29:01] (03CR) 10Btullis: [C:03+2] deployment_server: Absent the datasets-config kubeconfigs [puppet] - 10https://gerrit.wikimedia.org/r/1359809 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [11:29:05] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs6001.drmrs.wmnet with reason: host reimage [11:33:06] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1360375|Upgrading symfony/mailer and symfony/mime (v7.4.19 => v7.4.22) (T442768)]] (1.48.0-wmf.1), [[gerrit:1360000|ParserOutputAccess: Don't load dirty entry if not needed (T442669)]] (1.48.0-wmf.1) (duration: 14m 35s) [11:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:35:31] (03PS1) 10Ladsgroup: Upgrading symfony/mailer and symfony/mime (v7.4.17 => v7.4.22) [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1360384 (https://phabricator.wikimedia.org/T442768) (owner: 10Ladsgroup-claude) [11:36:03] (03PS2) 10Ladsgroup: ParserOutputAccess: Don't load dirty entry if not needed [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359999 (https://phabricator.wikimedia.org/T442669) [11:36:13] (03CR) 10Klausman: [C:03+1] ml: add vLLM 0.32 image (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360334 (https://phabricator.wikimedia.org/T442138) (owner: 10Bartosz Wójtowicz) [11:36:18] (03CR) 10Ladsgroup: [C:03+2] Upgrading symfony/mailer and symfony/mime (v7.4.17 => v7.4.22) [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1360384 (https://phabricator.wikimedia.org/T442768) (owner: 10Ladsgroup-claude) [11:36:26] (03CR) 10Ladsgroup: [C:03+2] ParserOutputAccess: Don't load dirty entry if not needed [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359999 (https://phabricator.wikimedia.org/T442669) (owner: 10Ladsgroup) [11:40:18] (03CR) 10CI reject: [V:04-1] ParserOutputAccess: Don't load dirty entry if not needed [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359999 (https://phabricator.wikimedia.org/T442669) (owner: 10Ladsgroup) [11:40:42] (03CR) 10CI reject: [V:04-1] Upgrading symfony/mailer and symfony/mime (v7.4.17 => v7.4.22) [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1360384 (https://phabricator.wikimedia.org/T442768) (owner: 10Ladsgroup-claude) [11:41:14] (03CR) 10Kevin Bazira: [C:03+1] knative-serving: Allow port 8112 in the activator ingress policy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360372 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [11:42:50] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12488873 (10cmooney) Just to document the results here this has had a pronounced affect on the profiling of traffic on our internet ed... [11:44:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [11:46:34] 10ops-eqiad, 06SRE, 06DC-Ops, 07Essential-Work: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12488878 (10elukey) From the UEFI shell: * `drivers` ` V VERSION E G G #D #C DRIVER NAME IMAGE NAME == ======== = = = == == ======================... [11:46:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [11:47:44] (03PS2) 10Ladsgroup: Upgrading symfony/mailer and symfony/mime (v7.4.17 => v7.4.22) [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1360384 (https://phabricator.wikimedia.org/T442768) (owner: 10Ladsgroup-claude) [11:47:45] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs6001.drmrs.wmnet with OS trixie [11:48:35] (03CR) 10Kevin Bazira: [C:03+1] ml-services: place edit-check on 64G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360376 (https://phabricator.wikimedia.org/T441680) (owner: 10Gkyziridis) [11:49:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [11:50:47] (03CR) 10Gkyziridis: [C:03+2] ml-services: place edit-check on 64G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360376 (https://phabricator.wikimedia.org/T441680) (owner: 10Gkyziridis) [11:51:32] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1228 crashed - https://phabricator.wikimedia.org/T442815#12488896 (10Jclark-ctr) This server is out of warranty {F111233831} Support Services and Warranty: ProSupport Ended on July 28, 2026 [11:52:19] (03Merged) 10jenkins-bot: ml-services: place edit-check on 64G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360376 (https://phabricator.wikimedia.org/T441680) (owner: 10Gkyziridis) [11:52:43] (03PS1) 10Vgutierrez: Revert "hieradata: Depool lvs6001 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360387 [11:52:59] (03PS2) 10Vgutierrez: Revert "hieradata: Depool lvs6001 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360387 (https://phabricator.wikimedia.org/T401832) [11:53:14] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360387 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [11:53:30] (03CR) 10Btullis: [C:03+2] deployment_server: Remove the datasets-config kubeconfigs [puppet] - 10https://gerrit.wikimedia.org/r/1359810 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [11:53:37] (03PS2) 10Btullis: deployment_server: Remove the datasets-config kubeconfigs [puppet] - 10https://gerrit.wikimedia.org/r/1359810 (https://phabricator.wikimedia.org/T442701) [11:53:39] !log gkyziridis@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . [11:53:39] (03CR) 10Ladsgroup: [C:03+2] Upgrading symfony/mailer and symfony/mime (v7.4.17 => v7.4.22) [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1360384 (https://phabricator.wikimedia.org/T442768) (owner: 10Ladsgroup-claude) [11:53:47] (03CR) 10Ladsgroup: [C:03+2] "..." [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359999 (https://phabricator.wikimedia.org/T442669) (owner: 10Ladsgroup) [11:54:02] (03CR) 10CI reject: [V:04-1] ParserOutputAccess: Don't load dirty entry if not needed [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359999 (https://phabricator.wikimedia.org/T442669) (owner: 10Ladsgroup) [11:56:02] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1228 crashed - https://phabricator.wikimedia.org/T442815#12488909 (10Marostegui) Yeah, that's one of the ones that we worked on with several crashes and Dell simply didn't help and then the warranty expired and then they quickly said the warranty expired. It's very di... [11:56:37] (03PS3) 10Ladsgroup: ParserOutputAccess: Don't load dirty entry if not needed [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359999 (https://phabricator.wikimedia.org/T442669) [11:56:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [11:56:44] (03CR) 10Ladsgroup: "again" [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359999 (https://phabricator.wikimedia.org/T442669) (owner: 10Ladsgroup) [11:56:59] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'edit-check' for release 'main' . [11:57:08] !log gkyziridis@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' . [11:59:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1360384 (https://phabricator.wikimedia.org/T442768) (owner: 10Ladsgroup-claude) [11:59:18] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy1003 using scap backport" [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359999 (https://phabricator.wikimedia.org/T442669) (owner: 10Ladsgroup) [11:59:36] RECOVERY - Host db1228 #page is UP: PING OK - Packet loss = 0%, RTA = 0.45 ms [11:59:37] PROBLEM - MariaDB read only s4 on db1228 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [11:59:46] PROBLEM - MariaDB Replica IO: s4 #page on db1228 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [11:59:49] (03Merged) 10jenkins-bot: Upgrading symfony/mailer and symfony/mime (v7.4.17 => v7.4.22) [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1360384 (https://phabricator.wikimedia.org/T442768) (owner: 10Ladsgroup-claude) [11:59:57] (03CR) 10Daniel Kertesz: [C:03+1] Revert "hieradata: Depool lvs6001 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360387 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [11:59:57] PROBLEM - MariaDB Replica Lag: s4 #page on db1228 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [12:00:01] PROBLEM - mysqld processes on db1228 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [12:00:04] (03CR) 10Vgutierrez: [C:03+2] Revert "hieradata: Depool lvs6001 before reimaging" [puppet] - 10https://gerrit.wikimedia.org/r/1360387 (https://phabricator.wikimedia.org/T401832) (owner: 10Vgutierrez) [12:00:05] Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T1200) [12:00:06] PROBLEM - MariaDB Replica SQL: s4 #page on db1228 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [12:00:06] PROBLEM - MariaDB Event Scheduler s4 on db1228 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [12:00:07] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:00:10] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:00:13] PROBLEM - MariaDB Events s4 on db1228 is CRITICAL: CRITICAL - Failed to query events: ERROR 2002 (HY000): Cant connect to local server through socket /run/mysqld/mysqld.sock (2) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [12:00:13] !log root@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:00:17] !log root@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:00:44] great way to start the shift [12:00:47] !ack [12:00:48] 8714 (ACKED) db1228 (paged)/MariaDB Replica IO: s4 (paged) [12:00:48] 8715 (ACKED) db1228 (paged)/MariaDB Replica Lag: s4 (paged) [12:00:48] 8716 (ACKED) db1228 (paged)/MariaDB Replica SQL: s4 (paged) [12:00:58] I think it's expired downtime? [12:01:02] volans: it is ok, check corto [12:01:11] haven't had yet the time :D ack [12:01:42] it just mentions db1262, not db1228 [12:01:57] oh sorry that is my bad [12:01:59] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357767 (https://phabricator.wikimedia.org/T442057) (owner: 10Codename Noreste) [12:02:06] volans: let me sort it [12:02:52] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs6001.*} and A:liberica [12:02:59] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs6001.*} and A:liberica [12:03:01] I'm here [12:03:03] !incidents [12:03:04] 8709 (ACKED) db1262 (paged)/MariaDB Replica SQL: x4 (paged) [12:03:04] 8710 (ACKED) db1262 (paged)/MariaDB Replica IO: x4 (paged) [12:03:04] 8711 (ACKED) db1262 (paged)/MariaDB Replica Lag: x4 (paged) [12:03:04] 8714 (ACKED) db1228 (paged)/MariaDB Replica IO: s4 (paged) [12:03:05] 8715 (ACKED) db1228 (paged)/MariaDB Replica Lag: s4 (paged) [12:03:05] 8716 (ACKED) db1228 (paged)/MariaDB Replica SQL: s4 (paged) [12:03:05] 8713 (RESOLVED) Host db1228 (paged) [12:03:05] 8712 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr3-eqsin:9804 Peering: Equinix (Wikimedia-SG1-IX-00 Singapore, MAC filter) xe-0/1/3 gnmi eqsin) [12:03:06] 8708 (RESOLVED) Host db1262 (paged) [12:03:38] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reimage for host lvs3010.esams.wmnet with OS trixie [12:04:34] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs3010.esams.wmnet (push config only) [12:04:40] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs3010.esams.wmnet (push config only) [12:04:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12488947 (10phaultfinder) [12:05:14] Sorry, yeah db1228 is pining for the fjords [12:09:33] (03Merged) 10jenkins-bot: ParserOutputAccess: Don't load dirty entry if not needed [core] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1359999 (https://phabricator.wikimedia.org/T442669) (owner: 10Ladsgroup) [12:09:33] (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360397 [12:09:59] !log ladsgroup@deploy1003 Started scap sync-world: Backport for [[gerrit:1360384|Upgrading symfony/mailer and symfony/mime (v7.4.17 => v7.4.22)]] (1.47.0-wmf.24), [[gerrit:1359999|ParserOutputAccess: Don't load dirty entry if not needed (T442669)]] (1.47.0-wmf.24) [12:11:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [12:13:01] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12489222 (10cmooney) Actually @ayounsi alerted me to a better stat to share. We actually had a saturation incident earlier today on o... [12:13:32] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1228 crashed - https://phabricator.wikimedia.org/T442815#12489223 (10Jclark-ctr) a:03Jclark-ctr There is a newer verion of bios i am upgrading to right now [12:13:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [12:14:25] !log ladsgroup@deploy1003 ladsgroup, ladsgroup-claude: Backport for [[gerrit:1360384|Upgrading symfony/mailer and symfony/mime (v7.4.17 => v7.4.22)]] (1.47.0-wmf.24), [[gerrit:1359999|ParserOutputAccess: Don't load dirty entry if not needed (T442669)]] (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:16:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [12:17:08] !log ladsgroup@deploy1003 ladsgroup, ladsgroup-claude: Continuing with deployment [12:18:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [12:21:51] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12489247 (10BBlack) Super-cool outcome, nice work all! [12:22:59] !log ladsgroup@deploy1003 Finished scap sync-world: Backport for [[gerrit:1360384|Upgrading symfony/mailer and symfony/mime (v7.4.17 => v7.4.22)]] (1.47.0-wmf.24), [[gerrit:1359999|ParserOutputAccess: Don't load dirty entry if not needed (T442669)]] (1.47.0-wmf.24) (duration: 13m 00s) [12:24:18] (03PS1) 10Gkyziridis: ml-services: Move revertrisk-wikidata on 64G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360398 (https://phabricator.wikimedia.org/T441680) [12:24:47] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp[6012,6016].drmrs.wmnet} and A:cp [12:26:46] (03CR) 10Dbrant: [C:03+2] push-notifications: update to 2026-10-28-153049 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359905 (owner: 10Dbrant) [12:27:16] (03CR) 10JMeybohm: bgp: increase trigger time for NodeBGPSessionStatusNotEstabilshed (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1360350 (owner: 10Blake) [12:27:57] 06SRE, 10LDAP-Access-Requests: Grant Access to Logstash for SomeRandomDeveloper - https://phabricator.wikimedia.org/T442855 (10SomeRandomDeveloper) 03NEW [12:28:15] (03Merged) 10jenkins-bot: push-notifications: update to 2026-10-28-153049 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359905 (owner: 10Dbrant) [12:29:09] (03CR) 10Reedy: InitialiseSettings: Remove private $wgOATHAuth2FAForAllWarnings (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355738 (owner: 10Reedy) [12:29:23] (03PS3) 10Reedy: InitialiseSettings: Remove private $wgOATHAuth2FAForAllWarnings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355738 [12:29:29] !log dbrant@deploy1003 helmfile [staging] START helmfile.d/services/push-notifications: apply [12:29:51] !log dbrant@deploy1003 helmfile [staging] DONE helmfile.d/services/push-notifications: apply [12:29:53] !log vgutierrez@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs3010.esams.wmnet with OS trixie [12:30:03] (03PS4) 10Reedy: InitialiseSettings: Remove $wgOATHAuth2FAForAllWarnings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355738 [12:30:11] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reimage for host lvs3010.esams.wmnet with OS trixie [12:30:20] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs3010.esams.wmnet (push config only) [12:30:25] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs3010.esams.wmnet (push config only) [12:30:40] (03CR) 10Reedy: InitialiseSettings: Remove $wgOATHAuth2FAForAllWarnings (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355738 (owner: 10Reedy) [12:30:49] !log dbrant@deploy1003 helmfile [eqiad] START helmfile.d/services/push-notifications: apply [12:31:05] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [12:31:10] (03PS2) 10Gkyziridis: ml-services: Move revertrisk-wikidata on 64G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360398 (https://phabricator.wikimedia.org/T442857) [12:31:12] (03CR) 10Btullis: [C:03+2] deployment_server: Remove the datasets-config kubeconfigs [puppet] - 10https://gerrit.wikimedia.org/r/1359810 (https://phabricator.wikimedia.org/T442701) (owner: 10Btullis) [12:31:21] !log dbrant@deploy1003 helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply [12:31:32] !log dbrant@deploy1003 helmfile [codfw] START helmfile.d/services/push-notifications: apply [12:31:58] (03CR) 10JMeybohm: "The global change is in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1360006" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360005 (owner: 10RLazarus) [12:32:04] !log dbrant@deploy1003 helmfile [codfw] DONE helmfile.d/services/push-notifications: apply [12:32:12] (03CR) 10JMeybohm: sre.k8s: add upgrade-kubelets-inplace cookbook (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1346036 (https://phabricator.wikimedia.org/T437480) (owner: 10Jelto) [12:34:13] (03CR) 10JMeybohm: [C:03+1] gateway-api-crds: add chart with the Gateway API CRDs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357212 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [12:34:30] jouncebot: nowandnext [12:34:30] For the next 0 hour(s) and 25 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T1200) [12:34:30] In 0 hour(s) and 25 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T1300) [12:34:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12489331 (10phaultfinder) [12:34:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [12:35:10] (03CR) 10Reedy: [C:03+2] InitialiseSettings: Remove $wgOATHAuth2FAForAllWarnings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355738 (owner: 10Reedy) [12:35:37] !log vgutierrez@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp6012.drmrs.wmnet [12:35:57] (03Merged) 10jenkins-bot: InitialiseSettings: Remove $wgOATHAuth2FAForAllWarnings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355738 (owner: 10Reedy) [12:37:36] 10ops-esams, 06DC-Ops, 06Traffic: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859 (10Vgutierrez) 03NEW [12:38:28] !log reedy@deploy1003 Started scap sync-world: Backport for [[gerrit:1355738|InitialiseSettings: Remove $wgOATHAuth2FAForAllWarnings]] [12:39:26] 06SRE, 06Infrastructure-Foundations: Upgrade rpki hosts to Trixie - https://phabricator.wikimedia.org/T438122#12489366 (10MoritzMuehlenhoff) a:03MoritzMuehlenhoff [12:41:16] !log vgutierrez@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs3010.esams.wmnet with OS trixie [12:41:19] !log root@cumin2003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw [12:41:23] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet [12:41:24] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet [12:41:48] (03CR) 10Blake: bgp: increase trigger time for NodeBGPSessionStatusNotEstabilshed (031 comment) [alerts] - 10https://gerrit.wikimedia.org/r/1360350 (owner: 10Blake) [12:42:50] !log reedy@deploy1003 reedy: Backport for [[gerrit:1355738|InitialiseSettings: Remove $wgOATHAuth2FAForAllWarnings]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:45:36] (03PS1) 10Vgutierrez: varnish: Check ?download before stripping query params [puppet] - 10https://gerrit.wikimedia.org/r/1360402 (https://phabricator.wikimedia.org/T442313) [12:46:11] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1228 crashed - https://phabricator.wikimedia.org/T442815#12489383 (10Jclark-ctr) Updated Bios Server is back up now @Marostegui 1245 they have not really responded in over a month or assistated at all that i have seen [12:46:30] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet [12:46:32] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet [12:46:38] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet [12:46:39] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet [12:47:21] (03CR) 10Jelto: [C:03+1] "looks reasonable" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360359 (https://phabricator.wikimedia.org/T442341) (owner: 10JavierMonton) [12:48:06] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360403 [12:48:09] !log reedy@deploy1003 reedy: Continuing with deployment [12:48:59] (03CR) 10Jelto: [C:03+1] "lgtm, I can merge this after I05ffe21556b515471a32a2e1f3dddc6e7a2cf056 is merged" [puppet] - 10https://gerrit.wikimedia.org/r/1359776 (https://phabricator.wikimedia.org/T442341) (owner: 10JavierMonton) [12:50:27] (03PS1) 10GergesShamon: [arwiktionary] Add "وق" as a project namespace alias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360404 (https://phabricator.wikimedia.org/T442847) [12:50:39] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12489400 (10Fabfur) Very nice to hear this, good job! [12:51:11] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360404 (https://phabricator.wikimedia.org/T442847) (owner: 10GergesShamon) [12:51:44] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet [12:51:46] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet [12:51:46] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw [12:51:50] !log installing NSS security updates [12:51:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:52:02] !log root@cumin2003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad [12:52:06] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet [12:52:07] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet [12:53:54] !log reedy@deploy1003 Finished scap sync-world: Backport for [[gerrit:1355738|InitialiseSettings: Remove $wgOATHAuth2FAForAllWarnings]] (duration: 15m 26s) [12:54:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [12:57:15] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet [12:57:18] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet [12:57:23] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet [12:57:25] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet [13:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: Time to do the UTC afternoon backport window deploy. Don't look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T1300). [13:00:05] Thiemo_WMDE, Tran, codenamenoreste, and _Gerges: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:09] <_Gerges> here [13:00:11] o/ [13:00:17] here [13:00:29] (03CR) 10Kevin Bazira: [C:03+1] ml-services: Move revertrisk-wikidata on 64G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360398 (https://phabricator.wikimedia.org/T442857) (owner: 10Gkyziridis) [13:00:48] I’m in a meeting, hope someone else can deploy [13:01:12] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1005.eqiad.wmnet [13:02:21] !log root@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet [13:02:24] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet [13:02:24] !log root@cumin2003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad [13:03:05] (03PS1) 10Sergio Gimeno: GrowthExperiments: enable SpecialHome on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360406 (https://phabricator.wikimedia.org/T442733) [13:03:52] <_Gerges> @urbanecm and [13:03:56] <_Gerges> @TheresNoTime [13:04:11] <_Gerges> any one can deploy [13:04:27] (03CR) 10Sergio Gimeno: Home Experience: Feature flag to enable Home (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1352515 (https://phabricator.wikimedia.org/T439738) (owner: 10EMcFarland) [13:04:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12489435 (10phaultfinder) [13:05:01] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1005.eqiad.wmnet [13:05:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [13:06:21] I can deploy in just a moment [13:07:11] I can deploy if we can wait for me to create some tables? My config patch is dependent on them. [13:07:19] Otherwise please skip mine until I can do so and I can deploy my own patch. [13:07:23] (03CR) 10Gkyziridis: [C:03+2] ml-services: Move revertrisk-wikidata on 64G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360398 (https://phabricator.wikimedia.org/T442857) (owner: 10Gkyziridis) [13:07:34] I'll start with Thiemo_WMDE's [13:08:19] (03CR) 10Michael Große: [C:03+1] GrowthExperiments: enable SpecialHome on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360406 (https://phabricator.wikimedia.org/T442733) (owner: 10Sergio Gimeno) [13:08:46] (03Merged) 10jenkins-bot: ml-services: Move revertrisk-wikidata on 64G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360398 (https://phabricator.wikimedia.org/T442857) (owner: 10Gkyziridis) [13:08:59] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Test the Docker Registry's GC with Releng/MediaWiki images - https://phabricator.wikimedia.org/T437297#12489452 (10elukey) Next steps: * Review and merge https://gerrit.wikimedia.org/r/c/operations... [13:09:25] Thiemo_WMDE: "Change '1355485' has dependency '1351987' targeting the master branch of MediaWiki code project 'mediawiki/extensions/Wikibase', but the dependency is not present in live train branch: wmf/1.47.0-wmf.24" [13:09:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:09:54] I am here for patch 1357767 [13:10:01] Hi TheresNoTime, would ya let me know when window is closed? I have some unrelated deployments to do (and I want to try spiderpig + k8s service for the first time!) [13:10:12] ottomata: ack! [13:10:14] ty! [13:10:21] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host urldownloader2005.wikimedia.org [13:11:18] codenamenoreste: are you ready? I'm going to move to yours [13:11:34] Uh huh, yes [13:11:49] (03CR) 10TrainBranchBot: [C:03+2] "Approved by samtar@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357767 (https://phabricator.wikimedia.org/T442057) (owner: 10Codename Noreste) [13:12:42] (03Merged) 10jenkins-bot: Require 5 edits for autoconfirmed users on English Wikiversity [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357767 (https://phabricator.wikimedia.org/T442057) (owner: 10Codename Noreste) [13:13:06] !log samtar@deploy1003 Started scap sync-world: Backport for [[gerrit:1357767|Require 5 edits for autoconfirmed users on English Wikiversity (T442057)]] [13:13:12] T442057: Require 5 edits for autoconfirmed users on English Wikiversity - https://phabricator.wikimedia.org/T442057 [13:14:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:14:52] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2005.wikimedia.org [13:15:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [13:15:58] !log jmm@cumin2003 START - Cookbook sre.hosts.reimage for host rpki2003.codfw.wmnet with OS trixie [13:16:10] 06SRE, 06Infrastructure-Foundations: Upgrade rpki hosts to Trixie - https://phabricator.wikimedia.org/T438122#12489498 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host rpki2003.codfw.wmnet with OS trixie [13:16:26] !log vgutierrez@cumin1004 cookbooks.sre.cdn.roll-reboot finished rebooting cp6016.drmrs.wmnet [13:16:26] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp[6012,6016].drmrs.wmnet} and A:cp [13:16:57] !log jmm@cumin2003 START - Cookbook sre.hosts.dhcp for host rpki2003.codfw.wmnet (push config only) [13:17:02] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki2003.codfw.wmnet (push config only) [13:17:44] !log samtar@deploy1003 samtar, codenamenoreste: Backport for [[gerrit:1357767|Require 5 edits for autoconfirmed users on English Wikiversity (T442057)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:18:28] codenamenoreste: ready for testing ^ [13:19:08] I don’t have my laptop right now [13:19:30] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host urldownloader2006.wikimedia.org [13:19:56] !log samtar@deploy1003 samtar, codenamenoreste: Continuing with deployment [13:20:22] !log gkyziridis@deploy1003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [13:20:32] !log gkyziridis@deploy1003 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . [13:20:42] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [13:21:42] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host aux-k8s-etcd1004.eqiad.wmnet [13:21:55] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1228 crashed - https://phabricator.wikimedia.org/T442815#12489532 (10CWilliams-WMF) Thanks for the update @Jclark-ctr [13:22:46] TheresNoTime: "the dependency is not present in live train branch: wmf/1.47.0-wmf.24". Sorry, I'm not sure I understand. We are at 1.48.0-wmf.1 as far as I can see. [13:23:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [13:23:57] Thiemo_WMDE: ah, might just need to pull out that 'depends-on' then? [13:24:00] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2006.wikimedia.org [13:24:13] just now seeing https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1355485/comments/82fd6c1c_e34b8e9a [13:24:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:24:52] The depends-on is merged. [13:25:26] I'm still awaiting… [13:25:44] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aux-k8s-etcd1004.eqiad.wmnet [13:25:44] I’m in my local college right now so I can bring out my laptop soon [13:25:50] !log samtar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1357767|Require 5 edits for autoconfirmed users on English Wikiversity (T442057)]] (duration: 12m 43s) [13:25:55] T442057: Require 5 edits for autoconfirmed users on English Wikiversity - https://phabricator.wikimedia.org/T442057 [13:26:31] codenamenoreste: deployed [13:27:18] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1228 crashed - https://phabricator.wikimedia.org/T442815#12489538 (10CWilliams-WMF) I have started replication to let the instance catch-up. [13:27:24] I don't understand where the message "the dependency is not present in live train branch" comes from, what it means and what I'm meant to do about it. group1 is at 1.48.0-wmf.1 according to https://versions.toolforge.org. wikidatawiki is in group1. The config change only affects wikidatawiki. [13:27:54] Thiemo_WMDE: group2 is still at 1.47.0-wmf.24, so afaict spiderpig will refuse to merge it, hence the error "Master dependencies must be cherry-picked to all live train branches.". I'll remove the 'depends-on' line and try again [13:28:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [13:28:42] I didn't expect the train to be broken for so long. We can also wait. The patch is not urgent. [13:28:42] FIRING: JobUnavailable: Reduced availability for job routinator in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:29:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12489546 (10phaultfinder) [13:30:10] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for kevinbazira - https://phabricator.wikimedia.org/T442865 (10kevinbazira) 03NEW [13:30:22] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [13:30:25] !log jmm@cumin2003 START - Cookbook sre.hosts.dhcp for host rpki2003.codfw.wmnet (remove config only) [13:30:29] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki2003.codfw.wmnet (remove config only) [13:30:33] Thiemo_WMDE: if its not urgent, I'd personally feel a bit comfier saying wait until next week? [13:30:57] I move it. [13:31:23] ty! [13:31:27] _Gerges: ready? [13:31:33] <_Gerges> Yes [13:32:09] (03CR) 10TrainBranchBot: [C:03+2] "Approved by samtar@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360404 (https://phabricator.wikimedia.org/T442847) (owner: 10GergesShamon) [13:32:54] !log jmm@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on rpki2003.codfw.wmnet with reason: host reimage [13:33:02] (03Merged) 10jenkins-bot: [arwiktionary] Add "وق" as a project namespace alias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360404 (https://phabricator.wikimedia.org/T442847) (owner: 10GergesShamon) [13:33:06] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 29 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355485 (https://phabricator.wikimedia.org/T437054) (owner: 10Thiemo Kreuz (WMDE)) [13:33:27] !log samtar@deploy1003 Started scap sync-world: Backport for [[gerrit:1360404|[arwiktionary] Add "وق" as a project namespace alias (T442847)]] [13:33:32] T442847: Add "وق" as a project namespace alias on Arabic Wiktionary - https://phabricator.wikimedia.org/T442847 [13:34:16] (03CR) 10Brouberol: [C:03+2] admin/data: add spiderpig to the airflow users groups [puppet] - 10https://gerrit.wikimedia.org/r/1358221 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [13:34:53] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12489601 (10Aklapper) Hi and welcome @cfoster, could you please also connect your [SUL staff account](https:/... [13:35:01] (03CR) 10Dpogorzelski: [C:03+2] gateway-api-crds: add chart with the Gateway API CRDs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1357212 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [13:37:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:37:47] !log samtar@deploy1003 gergesshamon, samtar: Backport for [[gerrit:1360404|[arwiktionary] Add "وق" as a project namespace alias (T442847)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:38:23] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ldap-rw1001.wikimedia.org [13:38:26] _Gerges: ready for testing ^ [13:38:50] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki2003.codfw.wmnet with reason: host reimage [13:38:51] <_Gerges> I Tested By curl api. Everything looks good and is ready for deployment. [13:39:04] (03PS1) 10Kamila Součková: Revert^2 "mw-parsoid: temporarily override image to PHP 8.5 in eqiad" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360411 [13:39:07] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-rw1001.wikimedia.org [13:39:09] !log samtar@deploy1003 gergesshamon, samtar: Continuing with deployment [13:42:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:44:33] TheresNoTime: Can you give me a quick ping when deployment is done, so I can move the primary deployment server to codfw? [13:44:42] slyngs: ack [13:45:06] !log samtar@deploy1003 Finished scap sync-world: Backport for [[gerrit:1360404|[arwiktionary] Add "وق" as a project namespace alias (T442847)]] (duration: 11m 40s) [13:45:13] T442847: Add "وق" as a project namespace alias on Arabic Wiktionary - https://phabricator.wikimedia.org/T442847 [13:45:14] Thanks, don't rush, I'll need to step for ~20 minutes anyway [13:45:17] (I know ottomata also wanted to be pinged after, just fyi slyngs) [13:45:26] _Gerges: deployed [13:45:39] Tran: are you ready to self-deploy yours? [13:45:50] Yep [13:46:03] Tran: go ahead, please ping me and ottomata after you're done :) [13:46:11] I'm doing some table creates, a config deploy, and a private deploy and will ping as necessary after [13:46:18] thanks! :) [13:46:45] FIRING: WidespreadPuppetFailure: Puppet has failed in eqiad - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [13:48:05] (03CR) 10STran: "`" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360367 (https://phabricator.wikimedia.org/T442582) (owner: 10STran) [13:48:15] tables created, deploying config [13:48:36] (03CR) 10TrainBranchBot: [C:03+2] "Approved by stran@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360367 (https://phabricator.wikimedia.org/T442582) (owner: 10STran) [13:48:45] (03PS2) 10Kamila Součková: Revert^2 "mw-parsoid: temporarily override image to PHP 8.5 in eqiad" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360411 [13:49:07] 10SRE-Access-Requests: Add FIDO-backed SSH keys for dreamyjazz - https://phabricator.wikimedia.org/T442869 (10Dreamy_Jazz) 03NEW [13:49:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12489663 (10phaultfinder) [13:49:57] (03PS1) 10Kevin Bazira: admin: Add FIDO SSH key for kevinbazira [puppet] - 10https://gerrit.wikimedia.org/r/1360416 (https://phabricator.wikimedia.org/T442865) [13:50:11] 10SRE-Access-Requests: Add FIDO-backed SSH keys for dreamyjazz - https://phabricator.wikimedia.org/T442869#12489671 (10Dreamy_Jazz) [13:50:16] (03Merged) 10jenkins-bot: Enable SuggestedInvestigations case generation on ptwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360367 (https://phabricator.wikimedia.org/T442582) (owner: 10STran) [13:50:41] !log stran@deploy1003 Started scap sync-world: Backport for [[gerrit:1360367|Enable SuggestedInvestigations case generation on ptwiki (T442582)]] [13:50:47] T442582: Enable Suggested Investigations on Portuguese Wikipedia (ptwiki) - https://phabricator.wikimedia.org/T442582 [13:51:38] (03PS3) 10Hashar: Merge branch 'upstream/stable-3.11' into wmf/stable-3.11 [software/gerrit] (wmf/stable-3.11) - 10https://gerrit.wikimedia.org/r/1360360 (https://phabricator.wikimedia.org/T379714) [13:51:38] (03CR) 10Hashar: "https://gerrit.wikimedia.org/r/c/operations/software/gerrit/+/1360417" [software/gerrit] (wmf/stable-3.11) - 10https://gerrit.wikimedia.org/r/1360360 (https://phabricator.wikimedia.org/T379714) (owner: 10Hashar) [13:52:12] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ldap-rw2001.wikimedia.org [13:52:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [13:52:27] (03CR) 10Kamila Součková: [C:03+2] Revert^2 "mw-parsoid: temporarily override image to PHP 8.5 in eqiad" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360411 (owner: 10Kamila Součková) [13:52:52] PROBLEM - Host sretest2001 is DOWN: PING CRITICAL - Packet loss = 100% [13:53:04] !log kamila@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [13:53:08] PROBLEM - Host sretest2004 is DOWN: PING CRITICAL - Packet loss = 100% [13:53:08] PROBLEM - Host sretest2003 is DOWN: PING CRITICAL - Packet loss = 100% [13:53:20] RECOVERY - Host sretest2001 is UP: PING OK - Packet loss = 0%, RTA = 32.95 ms [13:53:38] RECOVERY - Host sretest2004 is UP: PING OK - Packet loss = 0%, RTA = 33.17 ms [13:53:40] RECOVERY - Host sretest2003 is UP: PING OK - Packet loss = 0%, RTA = 32.97 ms [13:54:17] (03Merged) 10jenkins-bot: Revert^2 "mw-parsoid: temporarily override image to PHP 8.5 in eqiad" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360411 (owner: 10Kamila Součková) [13:54:48] jouncebot: nowandnext [13:54:48] For the next 0 hour(s) and 5 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T1300) [13:54:48] In 0 hour(s) and 35 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T1430) [13:55:14] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki2003.codfw.wmnet with OS trixie [13:55:15] !log stran@deploy1003 stran: Backport for [[gerrit:1360367|Enable SuggestedInvestigations case generation on ptwiki (T442582)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:55:20] !log kamila@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [13:55:26] 06SRE, 06Infrastructure-Foundations: Upgrade rpki hosts to Trixie - https://phabricator.wikimedia.org/T438122#12489700 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host rpki2003.codfw.wmnet with OS trixie completed: - rpki2003 (**PASS**) - Downtimed on Icinga/Alert... [13:55:49] !log kamila@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [13:55:51] looks like the backport window is still happening :) [13:56:00] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-rw2001.wikimedia.org [13:56:13] yes sorry I'm mid-deploy [13:56:14] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host urldownloader1005.wikimedia.org [13:56:32] testing now [13:57:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [13:57:36] looks okay for now but as it's not user-facing, I'll keep on eye on logs/db for a bit afterwards. Continuing. [13:57:41] !log stran@deploy1003 stran: Continuing with deployment [13:57:51] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host apt1002.wikimedia.org [13:58:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:59:10] !log Upgrading and restarting Jenkins on releases1003 | https://releases-jenkins.wikimedia.org/ [13:59:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:00:45] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1005.wikimedia.org [14:01:15] (03CR) 10Muehlenhoff: [C:03+1] "Looks good and verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1360416 (https://phabricator.wikimedia.org/T442865) (owner: 10Kevin Bazira) [14:02:00] (03CR) 10Muehlenhoff: [C:03+2] admin: Add FIDO SSH key for kevinbazira [puppet] - 10https://gerrit.wikimedia.org/r/1360416 (https://phabricator.wikimedia.org/T442865) (owner: 10Kevin Bazira) [14:02:13] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host urldownloader1006.wikimedia.org [14:03:05] !log kamila@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [14:03:35] !log stran@deploy1003 Finished scap sync-world: Backport for [[gerrit:1360367|Enable SuggestedInvestigations case generation on ptwiki (T442582)]] (duration: 12m 53s) [14:03:40] T442582: Enable Suggested Investigations on Portuguese Wikipedia (ptwiki) - https://phabricator.wikimedia.org/T442582 [14:03:45] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1002.wikimedia.org [14:03:47] starting the private deploy component of this [14:04:30] !log installing libpng1.6 security updates [14:04:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:05:02] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1228 crashed - https://phabricator.wikimedia.org/T442815#12489745 (10Jclark-ctr) 05Open→03Resolved [14:05:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [14:06:33] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader1006.wikimedia.org [14:06:53] (03CR) 10Bartosz Wójtowicz: [C:03+1] admin_ng: install the Gateway API CRDs on ml-staging-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358200 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [14:07:38] (03CR) 10Dpogorzelski: [C:03+2] admin_ng: install the Gateway API CRDs on ml-staging-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358200 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [14:08:16] (03CR) 10Vgutierrez: "`0 tests failed, 0 tests skipped, 45 tests passed` varnishtests for text are happy" [puppet] - 10https://gerrit.wikimedia.org/r/1360402 (https://phabricator.wikimedia.org/T442313) (owner: 10Vgutierrez) [14:08:42] RESOLVED: JobUnavailable: Reduced availability for job routinator in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:10:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [14:11:17] (03CR) 10Daniel Kertesz: [C:03+1] varnish: Check ?download before stripping query params [puppet] - 10https://gerrit.wikimedia.org/r/1360402 (https://phabricator.wikimedia.org/T442313) (owner: 10Vgutierrez) [14:11:52] (03PS1) 10Vgutierrez: hieradata: Let lvs3008 handle upload while lvs3010 is down [puppet] - 10https://gerrit.wikimedia.org/r/1360420 (https://phabricator.wikimedia.org/T442859) [14:11:54] jouncebot: nowandnext [14:11:54] No deployments scheduled for the next 0 hour(s) and 18 minute(s) [14:11:54] In 0 hour(s) and 18 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T1430) [14:12:12] (03CR) 10JavierMonton: [C:03+2] flink: script for puppet deployments [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360359 (https://phabricator.wikimedia.org/T442341) (owner: 10JavierMonton) [14:12:24] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360420 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [14:12:32] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] admin_ng: install the Gateway API CRDs on ml-staging-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358200 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [14:12:34] (03CR) 10Clément Goubert: [C:03+2] php/8.3: update to 8.3.36-1 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1359909 (owner: 10Clément Goubert) [14:13:27] !log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [14:14:08] 06SRE, 06Infrastructure-Foundations: Upgrade rpki hosts to Trixie - https://phabricator.wikimedia.org/T438122#12489789 (10MoritzMuehlenhoff) [14:14:55] (03CR) 10Fabfur: [C:03+1] varnish: Check ?download before stripping query params [puppet] - 10https://gerrit.wikimedia.org/r/1360402 (https://phabricator.wikimedia.org/T442313) (owner: 10Vgutierrez) [14:15:36] !log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [14:16:33] (03PS2) 10Vgutierrez: hieradata: Let lvs3008 handle upload while lvs3010 is down [puppet] - 10https://gerrit.wikimedia.org/r/1360420 (https://phabricator.wikimedia.org/T442859) [14:16:58] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360420 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [14:17:10] (03Merged) 10jenkins-bot: flink: script for puppet deployments [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360359 (https://phabricator.wikimedia.org/T442341) (owner: 10JavierMonton) [14:18:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [14:19:56] (03PS1) 10Bartosz Wójtowicz: kserve: allow deploy users to delete Knative revisions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360421 (https://phabricator.wikimedia.org/T442872) [14:20:46] (03PS3) 10Vgutierrez: hieradata: Let lvs3008 handle upload while lvs3010 is down [puppet] - 10https://gerrit.wikimedia.org/r/1360420 (https://phabricator.wikimedia.org/T442859) [14:21:09] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360420 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [14:21:11] (03CR) 10Jforrester: "CI image upgrade at I1faa5ad396e6a5923ad85677a56ca073641f33fa / I8a10e6a60420e2ceaa8b597b7d1513e640270a4c." [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1359909 (owner: 10Clément Goubert) [14:21:33] (03PS1) 10Gkyziridis: ml-services: Deploy revise-tone-task-generator on 64G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360424 (https://phabricator.wikimedia.org/T442877) [14:22:22] FIRING: GnmiInterfaceCountersDrop: ... [14:22:23] lsw1-f2-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=lsw1-f2-eqiad:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:22:29] (03PS1) 10Tryvix1509: Fix Ib5079d9f9057b5a6bc66ddde6ba41606d3694775 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360425 (https://phabricator.wikimedia.org/T440216) [14:23:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [14:23:43] (03PS1) 10Muehlenhoff: Bump image versions for latest Java security releases [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360426 (https://phabricator.wikimedia.org/T442744) [14:24:28] Hello, can I request backport now (it is a fix for merged patch) [14:24:33] (03PS1) 10Dpogorzelski: liftwing-studio: call eqiad inference directly [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360427 [14:24:35] https://gerrit.wikimedia.org/r/1360425 [14:24:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12489916 (10phaultfinder) [14:26:37] Tran: are you done with your deployment? [14:26:43] literally just finished, about to ping everyone [14:27:02] I was about to do a full-build deploy to update to the new php8.3-fpm images [14:27:23] RESOLVED: [2x] GnmiInterfaceCountersDrop: lsw1-f2-eqiad is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [14:27:37] TheresNoTime ottomata pinging as requested, sorry for going so over [14:28:05] ah there are more folks in queue + Hide_on_rosie then [14:28:13] I also gotta upgrade the CI Jenkins (if all goes fine it comes back in a minute or so) [14:28:19] (03CR) 10Vgutierrez: [C:03+2] varnish: Check ?download before stripping query params [puppet] - 10https://gerrit.wikimedia.org/r/1360402 (https://phabricator.wikimedia.org/T442313) (owner: 10Vgutierrez) [14:28:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [14:28:29] and slyngs is in the queue too :D [14:28:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [14:28:48] !log Deployed patch for T441775 [14:28:49] I can move mine to tomorrow morning when no one needs the deployment server [14:28:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:28:59] (03CR) 10Bartosz Wójtowicz: [C:03+1] istio/ml-serve: add an overlay enabling the Inference Extension [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358201 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [14:29:40] claime: There shouldn't be any issue with pushing the deployment server switch over until tomorrow? [14:29:51] (03CR) 10Lizimedia: "recheck" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1359941 (https://phabricator.wikimedia.org/T367230) (owner: 10Lizimedia) [14:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T1430) [14:30:15] slyngs: none [14:30:54] Excellent. TheresNoTime I'll move my stuff to tomorrow :-) [14:31:06] ack, all yours ottomata, I think? [14:31:10] 10SRE-swift-storage, 10Ceph, 06Data-Persistence, 06Infrastructure-Foundations: Raise S3 quota for the Docker Registry account - https://phabricator.wikimedia.org/T442862#12489960 (10MatthewVernon) After T438339 there is 5T quota for docker-registry (documented [[ https://wikitech.wikimedia.org/wiki/Ceph/Ce... [14:31:16] (/me needs to head to a meeting now so good luck all!) [14:31:53] (03CR) 10Klausman: [C:03+1] liftwing-studio: call eqiad inference directly [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360427 (owner: 10Dpogorzelski) [14:32:22] well if the backport window is done, I guess I can roll the CI Jenkins upgrade now :) [14:32:36] (03CR) 10Dpogorzelski: [C:03+2] liftwing-studio: call eqiad inference directly [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360427 (owner: 10Dpogorzelski) [14:33:14] !log Upgrading CI Jenkins [14:33:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:33:18] ottomata's change is a service deployment (hopefully by Spiderpig) [14:33:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [14:33:47] Jenkins upgraded [14:34:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [14:34:27] (03PS1) 10Muehlenhoff: Add FIDO SSH keys for dreamyjazz [puppet] - 10https://gerrit.wikimedia.org/r/1360429 (https://phabricator.wikimedia.org/T442869) [14:34:34] (03Merged) 10jenkins-bot: liftwing-studio: call eqiad inference directly [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360427 (owner: 10Dpogorzelski) [14:35:20] (03CR) 10Tryvix1509: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359140 (https://phabricator.wikimedia.org/T442485) (owner: 10Nvdtn19) [14:36:01] (03CR) 10Dreamy Jazz: [C:03+1] "Keys match what I have" [puppet] - 10https://gerrit.wikimedia.org/r/1360429 (https://phabricator.wikimedia.org/T442869) (owner: 10Muehlenhoff) [14:36:23] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d8-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:36:44] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO SSH keys for dreamyjazz [puppet] - 10https://gerrit.wikimedia.org/r/1360429 (https://phabricator.wikimedia.org/T442869) (owner: 10Muehlenhoff) [14:37:17] (03CR) 10Daniel Kertesz: [C:03+1] hieradata: Let lvs3008 handle upload while lvs3010 is down [puppet] - 10https://gerrit.wikimedia.org/r/1360420 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [14:38:48] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync [14:39:15] !log dpogorzelski@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync [14:39:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [14:39:36] (03PS1) 10Klausman: ml-services/admin_ng: Remove article-description service and NS [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359839 (https://phabricator.wikimedia.org/T438033) [14:39:38] I'm going to deploy ottomata's change [14:39:51] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [14:39:51] (03PS1) 10Klausman: deployment-server: remove article-description config [puppet] - 10https://gerrit.wikimedia.org/r/1359837 (https://phabricator.wikimedia.org/T438033) [14:40:01] (03CR) 10Bartosz Wójtowicz: [C:03+1] "LGTM! The only thing is that we will need to restart the controller to pick up the CRDs once this and https://gerrit.wikimedia.org/r/c/ope" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358202 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [14:40:13] (03CR) 10Phuedx: [C:03+2] [eventgate] Bump to v1.35.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359381 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [14:40:58] (03CR) 10Dpogorzelski: [C:03+2] istio/ml-serve: add an overlay enabling the Inference Extension [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358201 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [14:41:50] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device ssw1-a8-codfw [14:41:50] (03Merged) 10jenkins-bot: [eventgate] Bump to v1.35.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359381 (https://phabricator.wikimedia.org/T442177) (owner: 10Ottomata) [14:42:02] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a8-codfw [14:42:04] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device ssw1-a1-codfw [14:42:16] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-a1-codfw [14:42:19] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b7-codfw [14:42:30] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b7-codfw [14:42:33] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b4-codfw [14:42:34] (03Merged) 10jenkins-bot: istio/ml-serve: add an overlay enabling the Inference Extension [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358201 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [14:42:37] I'll backport Hide_on_rosie 's patch and then do my rebuild because that's going to take a while [14:42:44] Okay [14:42:44] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b4-codfw [14:42:46] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-b3-codfw [14:42:58] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-codfw [14:43:00] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a7-codfw [14:43:07] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cgoubert@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360425 (https://phabricator.wikimedia.org/T440216) (owner: 10Tryvix1509) [14:43:09] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a7-codfw [14:43:11] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a8-codfw [14:43:23] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a8-codfw [14:43:25] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device lsw1-a3-codfw [14:43:35] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Degraded RAID on an-presto1006 - https://phabricator.wikimedia.org/T441831#12490064 (10BTullis) @Jclark-ctr - I have a question for you. Now that one drive has been removed, I was unable to make a RAID 10 logical volume with the... [14:43:37] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-a3-codfw [14:43:39] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device ssw1-e1-eqiad [14:43:46] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-e1-eqiad [14:43:50] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device cr2-eqsin [14:43:56] (03Merged) 10jenkins-bot: Fix Ib5079d9f9057b5a6bc66ddde6ba41606d3694775 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360425 (https://phabricator.wikimedia.org/T440216) (owner: 10Tryvix1509) [14:43:56] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Degraded RAID on an-presto1006 - https://phabricator.wikimedia.org/T441831#12490067 (10BTullis) a:05BTullis→03Jclark-ctr [14:44:23] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-eqsin [14:44:24] !log cgoubert@deploy1003 Started scap sync-world: Backport for [[gerrit:1360425|Fix Ib5079d9f9057b5a6bc66ddde6ba41606d3694775 (T440216)]] [14:44:32] T440216: Enable extended confirmed user group on Lithuanian Wikipedia - https://phabricator.wikimedia.org/T440216 [14:45:26] (03PS1) 10Arlolra: Continue to serve config for cached js [extensions/ParserMigration] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360430 (https://phabricator.wikimedia.org/T430970) [14:45:58] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 29 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [extensions/ParserMigration] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360430 (https://phabricator.wikimedia.org/T430970) (owner: 10Arlolra) [14:46:03] !log phuedx@deploy1003 Started scap deploy-service eventgate-all: Updating all instances of EventGate to v1.35.0 [14:46:03] !log phuedx@deploy1003 scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main [14:46:23] RESOLVED: CertAlmostExpired: gNMI TLS certificate for lsw1-d8-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [14:46:27] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host bast5005.wikimedia.org [14:47:04] (03CR) 10Dpogorzelski: [C:03+2] kserve-llmisvc-resources: install the Inference Extension CRDs on ml-staging-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358202 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [14:47:30] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet [14:50:40] (03PS1) 10Cathal Mooney: sre.network.tls: update calls to Python datetime.not_valid_after: [cookbooks] - 10https://gerrit.wikimedia.org/r/1360432 (https://phabricator.wikimedia.org/T442883) [14:52:41] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast5005.wikimedia.org [14:53:16] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet [14:53:57] !log phuedx@deploy1003 scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main [14:54:08] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host build2004.codfw.wmnet [14:54:17] (03CR) 10Bking: [C:03+1] Bump image versions for latest Java security releases [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360426 (https://phabricator.wikimedia.org/T442744) (owner: 10Muehlenhoff) [14:54:25] !log phuedx@deploy1003 scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main [14:54:51] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [14:54:58] !log phuedx@deploy1003 scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main [14:56:17] (03PS4) 10Lizimedia: Add a basic completion script for cookbooks [software/spicerack] - 10https://gerrit.wikimedia.org/r/1359941 (https://phabricator.wikimedia.org/T367230) [14:56:50] (03Merged) 10jenkins-bot: kserve-llmisvc-resources: install the Inference Extension CRDs on ml-staging-codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358202 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [14:57:33] (03CR) 10Atsuko: [C:03+1] "`-many +0` diffs are the best!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359839 (https://phabricator.wikimedia.org/T438033) (owner: 10Klausman) [14:57:39] FIRING: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [14:57:50] (03CR) 10Atsuko: [C:03+1] deployment-server: remove article-description config [puppet] - 10https://gerrit.wikimedia.org/r/1359837 (https://phabricator.wikimedia.org/T438033) (owner: 10Klausman) [14:58:20] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host build2004.codfw.wmnet [14:58:28] (03CR) 10Ahmon Dancy: [C:03+1] Add a systemd timer to periodically drop the old MediaWiki images (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1359287 (https://phabricator.wikimedia.org/T437297) (owner: 10Elukey) [14:58:41] (03CR) 10JavierMonton: "The other is merged, so whenever you want, thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/1359776 (https://phabricator.wikimedia.org/T442341) (owner: 10JavierMonton) [14:59:30] !log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. [15:00:53] !log phuedx@deploy1003 scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main [15:01:44] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Bump image versions for latest Java security releases [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360426 (https://phabricator.wikimedia.org/T442744) (owner: 10Muehlenhoff) [15:01:45] !log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. [15:02:43] !log cgoubert@deploy1003 cgoubert, tryvix1509: Backport for [[gerrit:1360425|Fix Ib5079d9f9057b5a6bc66ddde6ba41606d3694775 (T440216)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:02:48] T440216: Enable extended confirmed user group on Lithuanian Wikipedia - https://phabricator.wikimedia.org/T440216 [15:02:49] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device cr2-eqiad [15:02:49] !log cmooney@cumin1004 END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device cr2-eqiad [15:03:19] Hide_on_rosie: you can test your change with X-Wikimedia-Debug now if you want [15:03:40] or i can just go ahead and continue with deploying [15:03:50] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Degraded RAID on an-presto1006 - https://phabricator.wikimedia.org/T441831#12490244 (10Jclark-ctr) Global Hot Spare might be best idea @BTullis [15:04:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [15:05:10] !log phuedx@deploy1003 Finished scap deploy-service eventgate-all: Updating all instances of EventGate to v1.35.0 (duration: 19m 07s) [15:05:20] (03CR) 10Vgutierrez: [C:03+2] hieradata: Let lvs3008 handle upload while lvs3010 is down [puppet] - 10https://gerrit.wikimedia.org/r/1360420 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [15:05:24] claime: just go ahead. [15:05:27] ack [15:05:38] !log cgoubert@deploy1003 cgoubert, tryvix1509: Continuing with deployment [15:08:24] (03CR) 10Klausman: [V:03+2 C:03+2] ml-services/admin_ng: Remove article-description service and NS [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359839 (https://phabricator.wikimedia.org/T438033) (owner: 10Klausman) [15:08:25] (03PS2) 10Cathal Mooney: sre.network.tls: update calls to Python datetime.not_valid_after: [cookbooks] - 10https://gerrit.wikimedia.org/r/1360432 (https://phabricator.wikimedia.org/T442883) [15:08:30] (03CR) 10Klausman: [V:03+2 C:03+2] deployment-server: remove article-description config [puppet] - 10https://gerrit.wikimedia.org/r/1359837 (https://phabricator.wikimedia.org/T438033) (owner: 10Klausman) [15:08:36] !log cmooney@cumin1004 START - Cookbook sre.network.tls for network device cr2-eqiad [15:08:36] !log cmooney@cumin1004 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device cr2-eqiad [15:09:26] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3008.esams.wmnet} and A:liberica [15:09:44] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3008.esams.wmnet} and A:liberica [15:11:31] (03PS1) 10Neriah: ukwiki: limit "user" namespace to only registered users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360437 (https://phabricator.wikimedia.org/T438016) [15:12:33] !log brouberol@deploy1003 Started scap deploy-service airflow: Test [15:12:33] !log brouberol@deploy1003 scap deploy-service airflow: [PRODUCTION/CANARY] dse-k8s-eqiad Deploying airflow-test-k8s [15:13:31] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 29 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360437 (https://phabricator.wikimedia.org/T438016) (owner: 10Neriah) [15:13:39] claime: Please ping me when you're done with your deployment. [15:14:08] dancy: i need to fully rebuild the images afterwards, what do you need and can we combine? [15:14:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12490349 (10phaultfinder) [15:14:38] claime: I need to deploy a new release of scap. [15:14:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12490348 (10phaultfinder) [15:14:46] (03PS1) 10Brouberol: Revert "admin/data: add spiderpig to the airflow users groups" [puppet] - 10https://gerrit.wikimedia.org/r/1360440 [15:15:01] dancy: ok, want to do that before i start a full rebuild i guess in the interest of time? [15:15:20] claime: Yes, that would be preferable. [15:15:36] 06SRE, 10LDAP-Access-Requests: Grant Access to Logstash for SomeRandomDeveloper - https://phabricator.wikimedia.org/T442855#12490373 (10Urbanecm) Full support. @SomeRandomDeveloper is a very useful member of the volunteer community, and I do think logstash access would be very useful for them. [15:15:52] dancy: no problem. 40% left in the deployment, i'll ping you, then you ping me when you're done [15:15:57] (03PS7) 10Clare Ming: Add Test Kitchen service to SpiderPig [puppet] - 10https://gerrit.wikimedia.org/r/1357383 (https://phabricator.wikimedia.org/T442889) [15:15:58] claime: Perfect [15:16:08] (03PS1) 10Vgutierrez: lvs: Bring high-traffic1 services to lvs3009 [puppet] - 10https://gerrit.wikimedia.org/r/1360441 (https://phabricator.wikimedia.org/T442859) [15:16:12] (03CR) 10CI reject: [V:04-1] Revert "admin/data: add spiderpig to the airflow users groups" [puppet] - 10https://gerrit.wikimedia.org/r/1360440 (owner: 10Brouberol) [15:16:19] (03Merged) 10jenkins-bot: ml-services/admin_ng: Remove article-description service and NS [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359839 (https://phabricator.wikimedia.org/T438033) (owner: 10Klausman) [15:16:23] (03PS1) 10Muehlenhoff: Fix headers for Java JDK variants [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360442 [15:16:32] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360441 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [15:16:47] (03PS2) 10Brouberol: Revert "admin/data: add spiderpig to the airflow users groups" [puppet] - 10https://gerrit.wikimedia.org/r/1360440 [15:18:11] !log cgoubert@deploy1003 Finished scap sync-world: Backport for [[gerrit:1360425|Fix Ib5079d9f9057b5a6bc66ddde6ba41606d3694775 (T440216)]] (duration: 33m 47s) [15:18:13] dancy: done [15:18:14] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for dreamyjazz - https://phabricator.wikimedia.org/T442869#12490414 (10Dreamy_Jazz) Both FIDO based SSH keys work as expected. Please remove my old SSH key which is the one as follows: ` ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGe1B8y/aFj... [15:18:17] T440216: Enable extended confirmed user group on Lithuanian Wikipedia - https://phabricator.wikimedia.org/T440216 [15:19:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [15:19:53] (03CR) 10Clare Ming: Add Test Kitchen service to SpiderPig (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1357383 (https://phabricator.wikimedia.org/T442889) (owner: 10Clare Ming) [15:20:03] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Fix headers for Java JDK variants [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360442 (owner: 10Muehlenhoff) [15:20:27] !log dancy@deploy1003 Installing scap version "4.309.0" for 155 host(s) [15:20:31] (03CR) 10Daniel Kertesz: [C:03+1] lvs: Bring high-traffic1 services to lvs3009 [puppet] - 10https://gerrit.wikimedia.org/r/1360441 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [15:21:53] hmm 30 minutes for a backport feels long, I think that's build time because of having two versions of php [15:22:02] cc dancy ^ [15:22:21] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [15:22:47] Actually it's not the backport before that was 12 minutes [15:22:48] (03CR) 10Vgutierrez: [C:03+2] lvs: Bring high-traffic1 services to lvs3009 [puppet] - 10https://gerrit.wikimedia.org/r/1360441 (https://phabricator.wikimedia.org/T442859) (owner: 10Vgutierrez) [15:22:49] hmm [15:23:15] claime: Looks like it was a full image build (judging from your scap-image-build-and-push-log) [15:23:40] dancy: ah, backport pulls in new base images by default? [15:23:46] (03PS1) 10Muehlenhoff: Remove legacy SSH key for dreamyjazz [puppet] - 10https://gerrit.wikimedia.org/r/1360444 (https://phabricator.wikimedia.org/T442869) [15:23:52] I forgot [15:23:57] Yes. [15:24:12] ok so i don't need to re-rebuild, great news :D [15:24:27] !log dancy@deploy1003 Installation of scap version "4.309.0" completed for 155 hosts [15:24:49] (03PS1) 10Ayounsi: Add _get_bgp_config [software/homer/deploy] - 10https://gerrit.wikimedia.org/r/1360446 (https://phabricator.wikimedia.org/T439408) [15:25:00] (03PS1) 10Majavah: dynamicproxy: Add status option to temporarily lock a proxy [puppet] - 10https://gerrit.wikimedia.org/r/1360447 (https://phabricator.wikimedia.org/T442870) [15:25:03] (03PS1) 10Majavah: openstack: wmcs-webproxy: Support setting the proxy status field [puppet] - 10https://gerrit.wikimedia.org/r/1360448 (https://phabricator.wikimedia.org/T442870) [15:25:45] claime: Back to you [15:25:54] !log vgutierrez@cumin1004 START - Cookbook sre.loadbalancer.admin config_reloading P{lvs3009.esams.wmnet} and A:liberica [15:26:02] dancy: thanks, i actually don't need it given the backport did the full rebuild <3 [15:26:12] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs3009.esams.wmnet} and A:liberica [15:26:14] (03CR) 10CI reject: [V:04-1] dynamicproxy: Add status option to temporarily lock a proxy [puppet] - 10https://gerrit.wikimedia.org/r/1360447 (https://phabricator.wikimedia.org/T442870) (owner: 10Majavah) [15:26:34] (03CR) 10CI reject: [V:04-1] openstack: wmcs-webproxy: Support setting the proxy status field [puppet] - 10https://gerrit.wikimedia.org/r/1360448 (https://phabricator.wikimedia.org/T442870) (owner: 10Majavah) [15:26:43] (03CR) 10CI reject: [V:04-1] Add _get_bgp_config [software/homer/deploy] - 10https://gerrit.wikimedia.org/r/1360446 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [15:26:58] (03CR) 10Btullis: [C:03+1] Revert "admin/data: add spiderpig to the airflow users groups" [puppet] - 10https://gerrit.wikimedia.org/r/1360440 (owner: 10Brouberol) [15:27:04] (03CR) 10Brouberol: [C:03+2] Revert "admin/data: add spiderpig to the airflow users groups" [puppet] - 10https://gerrit.wikimedia.org/r/1360440 (owner: 10Brouberol) [15:27:30] (03PS2) 10Majavah: dynamicproxy: Add status option to temporarily lock a proxy [puppet] - 10https://gerrit.wikimedia.org/r/1360447 (https://phabricator.wikimedia.org/T442870) [15:27:30] (03PS2) 10Majavah: openstack: wmcs-webproxy: Support setting the proxy status field [puppet] - 10https://gerrit.wikimedia.org/r/1360448 (https://phabricator.wikimedia.org/T442870) [15:29:25] (03PS1) 10Ayounsi: Config Transit BGP from netbox-bgp [homer/public] - 10https://gerrit.wikimedia.org/r/1360452 (https://phabricator.wikimedia.org/T439408) [15:30:33] (03PS2) 10Ayounsi: Add _get_bgp_config [software/homer/deploy] - 10https://gerrit.wikimedia.org/r/1360446 (https://phabricator.wikimedia.org/T439408) [15:30:39] (03CR) 10CI reject: [V:04-1] Config Transit BGP from netbox-bgp [homer/public] - 10https://gerrit.wikimedia.org/r/1360452 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [15:30:49] (03CR) 10Clare Ming: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1357383 (https://phabricator.wikimedia.org/T442889) (owner: 10Clare Ming) [15:32:34] (03CR) 10Ayounsi: "recheck" [homer/public] - 10https://gerrit.wikimedia.org/r/1360452 (https://phabricator.wikimedia.org/T439408) (owner: 10Ayounsi) [15:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:34:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [15:35:12] (03PS3) 10Majavah: openstack: wmcs-webproxy: Support setting the proxy status field [puppet] - 10https://gerrit.wikimedia.org/r/1360448 (https://phabricator.wikimedia.org/T442870) [15:35:12] (03PS1) 10Majavah: openstack: wmcs-webproxy: Display proxy status in list [puppet] - 10https://gerrit.wikimedia.org/r/1360454 (https://phabricator.wikimedia.org/T442870) [15:37:00] 10ops-esams, 06SRE, 06DC-Ops, 06Traffic: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859#12490513 (10ssingh) p:05Triage→03High [15:37:21] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [15:37:32] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-09 - 2026-10-30): Degraded RAID on an-presto1006 - https://phabricator.wikimedia.org/T441831#12490514 (10Jclark-ctr) a:05Jclark-ctr→03BTullis [15:38:30] (03CR) 10Dreamy Jazz: [C:03+1] Remove legacy SSH key for dreamyjazz [puppet] - 10https://gerrit.wikimedia.org/r/1360444 (https://phabricator.wikimedia.org/T442869) (owner: 10Muehlenhoff) [15:40:49] (03PS1) 10Ebernhardson: Reapply "cirrus: Send more_like traffic to eqiad" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360455 [15:41:16] CirrusSearch is dropping a bunch of traffic, north of 200 req/s. I'd like to deploy ^^ which will route a portion of the traffic back to the eqiad cluster [15:41:30] (03CR) 10DCausse: [C:03+1] Reapply "cirrus: Send more_like traffic to eqiad" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360455 (owner: 10Ebernhardson) [15:41:50] 10SRE-swift-storage, 10Ceph, 06Data-Persistence, 06Infrastructure-Foundations: Raise S3 quota for the Docker Registry account - https://phabricator.wikimedia.org/T442862#12490570 (10elukey) 05Open→03Resolved a:03elukey @MatthewVernon totally missed/forgot the other task, sorry for the noise! 5TB... [15:42:51] FIRING: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [15:43:17] (03CR) 10Ayounsi: [C:03+1] "lgtm! Worth testing it with test-cookbook if not done already." [cookbooks] - 10https://gerrit.wikimedia.org/r/1360432 (https://phabricator.wikimedia.org/T442883) (owner: 10Cathal Mooney) [15:44:41] (03CR) 10Bking: [C:03+1] Reapply "cirrus: Send more_like traffic to eqiad" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360455 (owner: 10Ebernhardson) [15:44:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [15:45:24] (03PS1) 10Klausman: helm/rest-gw: Remove articledesc rewrite rule. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360449 (https://phabricator.wikimedia.org/T438033) [15:46:26] 10ops-esams, 06SRE, 06DC-Ops, 06Traffic: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859#12490612 (10RobH) We also had another issue iwth a ganeti host in esams where we updated the firmware, it would clear the error, only for the error... [15:46:45] (03CR) 10Clément Goubert: [C:03+1] helm/rest-gw: Remove articledesc rewrite rule. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360449 (https://phabricator.wikimedia.org/T438033) (owner: 10Klausman) [15:47:00] (03PS1) 10Blake: mw-script: disable native k8s sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360457 [15:47:09] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ebernhardson@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360455 (owner: 10Ebernhardson) [15:47:58] (03CR) 10Dpogorzelski: [C:03+1] ml: add vLLM 0.32 image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360334 (https://phabricator.wikimedia.org/T442138) (owner: 10Bartosz Wójtowicz) [15:47:59] (03Merged) 10jenkins-bot: Reapply "cirrus: Send more_like traffic to eqiad" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360455 (owner: 10Ebernhardson) [15:48:02] (03CR) 10Clément Goubert: [C:03+1] mw-script: disable native k8s sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360457 (owner: 10Blake) [15:48:11] (03CR) 10Blake: [C:03+2] mw-script: disable native k8s sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360457 (owner: 10Blake) [15:48:25] !log ebernhardson@deploy1003 Started scap sync-world: Backport for [[gerrit:1360455|Reapply "cirrus: Send more_like traffic to eqiad"]] [15:48:36] (03CR) 10Klausman: [C:03+2] helm/rest-gw: Remove articledesc rewrite rule. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360449 (https://phabricator.wikimedia.org/T438033) (owner: 10Klausman) [15:49:06] ebernhardson: ack, I'll roll out something after you're done. [15:49:55] (03Merged) 10jenkins-bot: mw-script: disable native k8s sidecars [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360457 (owner: 10Blake) [15:50:17] (03PS5) 10Krinkle: MathML+MathJax rollout to phase 5 (Wikipedia small/medium/canary) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311060 (https://phabricator.wikimedia.org/T271001) [15:50:27] (03CR) 10CI reject: [V:04-1] MathML+MathJax rollout to phase 5 (Wikipedia small/medium/canary) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311060 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [15:50:29] (03Merged) 10jenkins-bot: helm/rest-gw: Remove articledesc rewrite rule. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360449 (https://phabricator.wikimedia.org/T438033) (owner: 10Klausman) [15:50:51] (03PS6) 10Krinkle: MathML+MathJax rollout to phase 5 (Wikipedia small/medium/canary) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311060 (https://phabricator.wikimedia.org/T271001) [15:50:58] (03PS5) 10Krinkle: MathML+MathJax rollout to phase 6 (remaining large Wikipedias) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311061 (https://phabricator.wikimedia.org/T271001) [15:51:44] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar, 13Patch-For-Review: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12490655 (10cfoster) @Aklapper just connected them and will add this to our team onboarding doc for next time... [15:51:45] RESOLVED: WidespreadPuppetFailure: Puppet has failed in eqiad - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [15:53:03] !log ebernhardson@deploy1003 ebernhardson: Backport for [[gerrit:1360455|Reapply "cirrus: Send more_like traffic to eqiad"]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:53:47] !log ebernhardson@deploy1003 ebernhardson: Continuing with deployment [15:53:49] (03PS3) 10Majavah: dynamicproxy: Add status option to temporarily lock a proxy [puppet] - 10https://gerrit.wikimedia.org/r/1360447 (https://phabricator.wikimedia.org/T442870) [15:53:49] (03PS4) 10Majavah: openstack: wmcs-webproxy: Support setting the proxy status field [puppet] - 10https://gerrit.wikimedia.org/r/1360448 (https://phabricator.wikimedia.org/T442870) [15:53:49] (03PS2) 10Majavah: openstack: wmcs-webproxy: Display proxy status in list [puppet] - 10https://gerrit.wikimedia.org/r/1360454 (https://phabricator.wikimedia.org/T442870) [15:53:50] (03PS1) 10Majavah: P:wmcs::novaproxy: Show accurate error page for disabled proxies [puppet] - 10https://gerrit.wikimedia.org/r/1360462 (https://phabricator.wikimedia.org/T442870) [15:54:13] !log klausman@deploy1003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [15:54:34] !log klausman@deploy1003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [15:54:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12490686 (10phaultfinder) [15:55:28] (03PS2) 10Majavah: P:wmcs::novaproxy: Show accurate error page for disabled proxies [puppet] - 10https://gerrit.wikimedia.org/r/1360462 (https://phabricator.wikimedia.org/T442870) [15:55:47] (03PS1) 10Dpogorzelski: llm-d: add endpoint-picker and uds-tokenizer images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360463 (https://phabricator.wikimedia.org/T436650) [15:55:52] (03PS1) 10Dpogorzelski: kserve-llm-inference: add router presets for the endpoint picker [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360464 (https://phabricator.wikimedia.org/T436650) [15:55:55] !log klausman@deploy1003 helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [15:56:13] (03CR) 10Cathal Mooney: "Sorry yep I commented on task works ok in testing." [cookbooks] - 10https://gerrit.wikimedia.org/r/1360432 (https://phabricator.wikimedia.org/T442883) (owner: 10Cathal Mooney) [15:56:15] (03CR) 10Cathal Mooney: [C:03+2] sre.network.tls: update calls to Python datetime.not_valid_after: [cookbooks] - 10https://gerrit.wikimedia.org/r/1360432 (https://phabricator.wikimedia.org/T442883) (owner: 10Cathal Mooney) [15:56:19] !log klausman@deploy1003 helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [15:57:01] !log klausman@deploy1003 helmfile [codfw] START helmfile.d/services/rest-gateway: apply [15:57:15] 10SRE-Access-Requests: Add yubikey SSH key for urbanecm - https://phabricator.wikimedia.org/T442901 (10Urbanecm_WMF) 03NEW [15:57:18] (03CR) 10CI reject: [V:04-1] kserve-llm-inference: add router presets for the endpoint picker [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360464 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [15:57:20] !log klausman@deploy1003 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [15:57:51] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter2006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [15:58:14] 10SRE-Access-Requests: Add FIDO-backed SSH keys for kemayo - https://phabricator.wikimedia.org/T442902 (10DLynch) 03NEW [15:58:36] (03PS1) 10DLynch: admin: Add FIDO SSH keys for kemayo [puppet] - 10https://gerrit.wikimedia.org/r/1360467 (https://phabricator.wikimedia.org/T442902) [15:58:51] (03Merged) 10jenkins-bot: sre.network.tls: update calls to Python datetime.not_valid_after: [cookbooks] - 10https://gerrit.wikimedia.org/r/1360432 (https://phabricator.wikimedia.org/T442883) (owner: 10Cathal Mooney) [16:00:05] jhathaway and rzl: #bothumor My software never has bugs. It just develops random features. Rise for Puppet request window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T1600). [16:00:05] No Gerrit patches in the queue for this window AFAICS. [16:01:53] !log ebernhardson@deploy1003 Finished scap sync-world: Backport for [[gerrit:1360455|Reapply "cirrus: Send more_like traffic to eqiad"]] (duration: 13m 28s) [16:02:15] Krinkle: all done, it's free for you [16:03:15] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy SSH key for dreamyjazz [puppet] - 10https://gerrit.wikimedia.org/r/1360444 (https://phabricator.wikimedia.org/T442869) (owner: 10Muehlenhoff) [16:05:23] thx [16:05:25] (03CR) 10Dzahn: "This leads to a "puppet change on every run"-alert. 19_docker-dockerhub-reject is changed on every run because the Docker hub to a differe" [puppet] - 10https://gerrit.wikimedia.org/r/1359404 (https://phabricator.wikimedia.org/T439046) (owner: 10Majavah) [16:06:02] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311060 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [16:06:53] (03Merged) 10jenkins-bot: MathML+MathJax rollout to phase 5 (Wikipedia small/medium/canary) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311060 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [16:07:16] !log krinkle@deploy1003 Started scap sync-world: Backport for [[gerrit:1311060|MathML+MathJax rollout to phase 5 (Wikipedia small/medium/canary) (T271001)]] [16:07:23] T271001: Transition to client-side MathJax SVG rendering as default - https://phabricator.wikimedia.org/T271001 [16:08:11] (03CR) 10Andrew Bogott: [C:03+1] "I am unsure about calling the field 'status' rather than something more direct like 'locked' but I can live with it." [puppet] - 10https://gerrit.wikimedia.org/r/1360447 (https://phabricator.wikimedia.org/T442870) (owner: 10Majavah) [16:09:05] (03CR) 10Majavah: [C:03+2] "Hmm. That implies that this block is not very effective if it docker hub can resolve to different IPs than what it did when creating the b" [puppet] - 10https://gerrit.wikimedia.org/r/1359404 (https://phabricator.wikimedia.org/T439046) (owner: 10Majavah) [16:10:33] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for dreamyjazz - https://phabricator.wikimedia.org/T442869#12490804 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff All done :-) [16:11:37] !log krinkle@deploy1003 krinkle: Backport for [[gerrit:1311060|MathML+MathJax rollout to phase 5 (Wikipedia small/medium/canary) (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:13:16] (03CR) 10Effie Mouzeli: [C:03+1] "Lets be sure we do not need any 'version: 8.5' again ;)" [puppet] - 10https://gerrit.wikimedia.org/r/1357273 (https://phabricator.wikimedia.org/T440321) (owner: 10Kamila Součková) [16:15:03] !log krinkle@deploy1003 krinkle: Continuing with deployment [16:15:12] (03PS1) 10CWilliams: zarcillo: enabling egress to config-master [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360470 (https://phabricator.wikimedia.org/T442904) [16:16:04] (03PS2) 10CWilliams: zarcillo: enabling egress to config-master [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360470 (https://phabricator.wikimedia.org/T442904) [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:16:57] (03PS1) 10Kamila Součková: Revert^3 "mw-parsoid: temporarily override image to PHP 8.5 in eqiad" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360473 [16:18:06] (03PS2) 10DLynch: admin: Add FIDO SSH keys for kemayo [puppet] - 10https://gerrit.wikimedia.org/r/1360467 (https://phabricator.wikimedia.org/T442902) [16:20:09] (03CR) 10Andrew Bogott: [C:03+1] P:wmcs::novaproxy: Show accurate error page for disabled proxies [puppet] - 10https://gerrit.wikimedia.org/r/1360462 (https://phabricator.wikimedia.org/T442870) (owner: 10Majavah) [16:20:20] (03CR) 10Andrew Bogott: [C:03+1] openstack: wmcs-webproxy: Display proxy status in list [puppet] - 10https://gerrit.wikimedia.org/r/1360454 (https://phabricator.wikimedia.org/T442870) (owner: 10Majavah) [16:20:33] (03CR) 10DLynch: "Switched one of them so I wouldn't need to redo attestation later." [puppet] - 10https://gerrit.wikimedia.org/r/1360467 (https://phabricator.wikimedia.org/T442902) (owner: 10DLynch) [16:20:54] !log krinkle@deploy1003 Finished scap sync-world: Backport for [[gerrit:1311060|MathML+MathJax rollout to phase 5 (Wikipedia small/medium/canary) (T271001)]] (duration: 13m 38s) [16:21:00] T271001: Transition to client-side MathJax SVG rendering as default - https://phabricator.wikimedia.org/T271001 [16:22:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [16:23:19] (03CR) 10Andrew Bogott: [C:03+1] openstack: wmcs-webproxy: Support setting the proxy status field [puppet] - 10https://gerrit.wikimedia.org/r/1360448 (https://phabricator.wikimedia.org/T442870) (owner: 10Majavah) [16:25:48] (03PS4) 10Majavah: dynamicproxy: Add status option to temporarily lock a proxy [puppet] - 10https://gerrit.wikimedia.org/r/1360447 (https://phabricator.wikimedia.org/T442870) [16:25:48] (03PS5) 10Majavah: openstack: wmcs-webproxy: Support setting the proxy status field [puppet] - 10https://gerrit.wikimedia.org/r/1360448 (https://phabricator.wikimedia.org/T442870) [16:25:49] (03PS3) 10Majavah: openstack: wmcs-webproxy: Display proxy status in list [puppet] - 10https://gerrit.wikimedia.org/r/1360454 (https://phabricator.wikimedia.org/T442870) [16:25:50] (03PS3) 10Majavah: P:wmcs::novaproxy: Show accurate error page for disabled proxies [puppet] - 10https://gerrit.wikimedia.org/r/1360462 (https://phabricator.wikimedia.org/T442870) [16:26:50] (03PS4) 10Brouberol: admin/data: add spiderpig to the airflow users groups [puppet] - 10https://gerrit.wikimedia.org/r/1360469 (https://phabricator.wikimedia.org/T412941) [16:27:17] (03CR) 10Majavah: dynamicproxy: Add status option to temporarily lock a proxy (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1360447 (https://phabricator.wikimedia.org/T442870) (owner: 10Majavah) [16:27:18] (03CR) 10Brouberol: [V:03+1] "PCC SUCCESS (CORE_DIFF 10): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9576/c" [puppet] - 10https://gerrit.wikimedia.org/r/1360469 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [16:29:08] (03CR) 10Majavah: [C:03+2] dynamicproxy: Add status option to temporarily lock a proxy [puppet] - 10https://gerrit.wikimedia.org/r/1360447 (https://phabricator.wikimedia.org/T442870) (owner: 10Majavah) [16:29:19] (03CR) 10Majavah: [C:03+2] openstack: wmcs-webproxy: Support setting the proxy status field [puppet] - 10https://gerrit.wikimedia.org/r/1360448 (https://phabricator.wikimedia.org/T442870) (owner: 10Majavah) [16:29:28] (03CR) 10Majavah: [C:03+2] openstack: wmcs-webproxy: Display proxy status in list [puppet] - 10https://gerrit.wikimedia.org/r/1360454 (https://phabricator.wikimedia.org/T442870) (owner: 10Majavah) [16:30:26] (03CR) 10Majavah: [C:03+2] P:wmcs::novaproxy: Show accurate error page for disabled proxies [puppet] - 10https://gerrit.wikimedia.org/r/1360462 (https://phabricator.wikimedia.org/T442870) (owner: 10Majavah) [16:30:52] !log brouberol@deploy1003 Started scap deploy-service airflow: Test [16:30:52] !log brouberol@deploy1003 scap deploy-service airflow: [PRODUCTION/CANARY] dse-k8s-eqiad Deploying airflow-test-k8s [16:30:59] !log brouberol@deploy1003 Finished scap deploy-service airflow: Test (duration: 00m 07s) [16:31:05] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [16:31:07] (03CR) 10Kamila Součková: [C:03+2] Revert^3 "mw-parsoid: temporarily override image to PHP 8.5 in eqiad" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360473 (owner: 10Kamila Součková) [16:32:40] (03Merged) 10jenkins-bot: Revert^3 "mw-parsoid: temporarily override image to PHP 8.5 in eqiad" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360473 (owner: 10Kamila Součková) [16:32:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [16:33:55] (03CR) 10Jdlrobson: [C:03+1] Continue to serve config for cached js [extensions/ParserMigration] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360430 (https://phabricator.wikimedia.org/T430970) (owner: 10Arlolra) [16:34:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12490968 (10phaultfinder) [16:38:29] 10SRE-tools, 06Infrastructure-Foundations, 10Packaging: Upgrade prometheus-atlas-exporter - https://phabricator.wikimedia.org/T429672#12490988 (10Lizimedia) a:03Lizimedia [16:39:27] (03CR) 10Cooltey: [C:03+2] wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360403 (owner: 10PipelineBot) [16:40:03] (03PS1) 10Clément Goubert: mediawiki: fix job sidecar [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360479 [16:40:14] volans, federico3, jhathaway, rzl : i'm wondering if I can sneak into this window with a backport to wmf.1 to allow us to continue PHP 8.5 testing on Parsoid today. [16:40:38] nothing in the puppet window but I do have stuff for the infra window [16:40:46] (03PS1) 10C. Scott Ananian: ThumbExtractor: Only pass elements to Zest::matches() for captions [extensions/MultimediaViewer] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360480 (https://phabricator.wikimedia.org/T441878) [16:41:12] (03Merged) 10jenkins-bot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360403 (owner: 10PipelineBot) [16:41:20] ^ that patch, which fixes a crash on PHP 8.5. I might backport it to .wmf.24 as well, just so that I don't have to wait for the train to roll to group 2 [16:41:26] (03CR) 10CI reject: [V:04-1] mediawiki: fix job sidecar [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360479 (owner: 10Clément Goubert) [16:41:29] 10ops-eqiad, 06SRE, 06DC-Ops, 07Essential-Work: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12491003 (10Papaul) @elukey thank you for working on this. Let dc-ops take over this issue and we will report to you or @jhathaway once we have any update since we open a tic... [16:41:47] (03PS1) 10C. Scott Ananian: ThumbExtractor: Only pass elements to Zest::matches() for captions [extensions/MultimediaViewer] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1360481 (https://phabricator.wikimedia.org/T441878) [16:42:00] it would be a spiderpig deploy, so i can self-serve [16:42:45] cscott: mo concerns from my side [16:42:50] *no [16:42:58] (03PS2) 10Clément Goubert: mediawiki: fix job sidecar [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360479 [16:43:47] (03CR) 10Blake: [C:03+1] mediawiki: fix job sidecar [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360479 (owner: 10Clément Goubert) [16:45:22] ok, approval from rzl and volans, representing SRE on call and the owner of this window, is enough? [16:45:32] !log cooltey@deploy1003 helmfile [staging] START helmfile.d/services/wikifeeds: apply [16:46:21] !log cooltey@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [16:46:29] (03CR) 10Andrew Bogott: Add profile for backing up capi worker nodes (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1355194 (https://phabricator.wikimedia.org/T440363) (owner: 10Andrew Bogott) [16:46:54] !log cooltey@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifeeds: apply [16:47:12] !log cooltey@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply [16:47:33] (03CR) 10Dzahn: [C:03+2] ci: Ensure a containerd root beneath the docker data-root [puppet] - 10https://gerrit.wikimedia.org/r/1356455 (https://phabricator.wikimedia.org/T440670) (owner: 10Dduvall) [16:47:37] !log cooltey@deploy1003 helmfile [codfw] START helmfile.d/services/wikifeeds: apply [16:47:50] ok going for it [16:48:01] !log cooltey@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply [16:48:10] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cscott@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360480 (https://phabricator.wikimedia.org/T441878) (owner: 10C. Scott Ananian) [16:48:37] 10ops-esams, 06SRE, 06DC-Ops, 06Traffic: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859#12491017 (10RobH) I need to clear the SEL to see if the error clears up properly, so uploading the system event log here for reference. FYI: it als... [16:48:59] (03CR) 10Blake: [C:03+2] mediawiki: fix job sidecar [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360479 (owner: 10Clément Goubert) [16:49:08] 10ops-esams, 06SRE, 06DC-Ops, 06Traffic: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859#12491019 (10RobH) {F111250042} [16:49:18] (03Merged) 10jenkins-bot: ThumbExtractor: Only pass elements to Zest::matches() for captions [extensions/MultimediaViewer] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360480 (https://phabricator.wikimedia.org/T441878) (owner: 10C. Scott Ananian) [16:49:35] (03CR) 10Clément Goubert: [C:03+1] apache: Configure wikimedia.org redirect in apache, remove from redirects.dat (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1333298 (https://phabricator.wikimedia.org/T427929) (owner: 10Jasmine) [16:49:49] !log cscott@deploy1003 Started scap sync-world: Backport for [[gerrit:1360480|ThumbExtractor: Only pass elements to Zest::matches() for captions (T441878)]] (1.48.0-wmf.1) [16:49:54] T441878: Error: Call to undefined method Dom\Text::hasAttributes() - https://phabricator.wikimedia.org/T441878 [16:50:28] (03PS1) 10Kamila Součková: Revert^2 "mw-parsoid: temporarily override image to PHP 8.5 in eqiad" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360482 [16:51:04] (03Merged) 10jenkins-bot: mediawiki: fix job sidecar [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360479 (owner: 10Clément Goubert) [16:52:13] (03CR) 10Jasmine: apache: Configure wikimedia.org redirect in apache, remove from redirects.dat (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1333298 (https://phabricator.wikimedia.org/T427929) (owner: 10Jasmine) [16:53:56] (03CR) 10Kamila Součková: [C:04-2] "DNM until I've uploaded a PS with the newest image version" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360482 (owner: 10Kamila Součková) [16:53:58] (03CR) 10Dzahn: [C:03+2] "Merged. I synced integration-puppetserver-01 and ran puppet on integration-agent-docker-1070, where /etc/containerd/config.toml now has ro" [puppet] - 10https://gerrit.wikimedia.org/r/1356455 (https://phabricator.wikimedia.org/T440670) (owner: 10Dduvall) [16:54:05] !log cscott@deploy1003 cscott: Backport for [[gerrit:1360480|ThumbExtractor: Only pass elements to Zest::matches() for captions (T441878)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:54:06] (03CR) 10Jasmine: apache: Configure wikimedia.org redirect in apache, remove from redirects.dat (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1333298 (https://phabricator.wikimedia.org/T427929) (owner: 10Jasmine) [16:56:07] (03PS5) 10Andrew Bogott: Add profile for backing up capi worker nodes [puppet] - 10https://gerrit.wikimedia.org/r/1355194 (https://phabricator.wikimedia.org/T440363) [16:58:16] !log cscott@deploy1003 cscott: Continuing with deployment [16:59:16] 10ops-esams, 06SRE, 06DC-Ops, 06Traffic: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859#12491071 (10RobH) 05Open→03Resolved a:03RobH lvs3010 firmware idrac was 7.00.00.00 now 7.30.30.54 bios 1.9.2 now 1.23.1 backplane 6.9... [16:59:22] (03PS1) 10JMeybohm: kubernetes::node: Fix zone detection logic [puppet] - 10https://gerrit.wikimedia.org/r/1360488 (https://phabricator.wikimedia.org/T442874) [17:00:05] bd808: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T1700). [17:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T1700) [17:01:09] (03CR) 10Btullis: admin/data: add spiderpig to the airflow users groups (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1360469 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [17:02:06] (03CR) 10CI reject: [V:04-1] kubernetes::node: Fix zone detection logic [puppet] - 10https://gerrit.wikimedia.org/r/1360488 (https://phabricator.wikimedia.org/T442874) (owner: 10JMeybohm) [17:02:15] i've got a spiderpig deployment in progress, should be done soon. [17:03:26] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.reimage for host lvs3010.esams.wmnet with OS trixie [17:03:32] o/ thanks, lmk and I'll get started after you [17:03:35] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs3010.esams.wmnet (push config only) [17:03:41] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs3010.esams.wmnet (push config only) [17:04:06] !log cscott@deploy1003 Finished scap sync-world: Backport for [[gerrit:1360480|ThumbExtractor: Only pass elements to Zest::matches() for captions (T441878)]] (1.48.0-wmf.1) (duration: 14m 17s) [17:04:11] T441878: Error: Call to undefined method Dom\Text::hasAttributes() - https://phabricator.wikimedia.org/T441878 [17:04:15] (03CR) 10Andrew Bogott: Add profile for backing up capi worker nodes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1355194 (https://phabricator.wikimedia.org/T440363) (owner: 10Andrew Bogott) [17:04:30] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cscott@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1360481 (https://phabricator.wikimedia.org/T441878) (owner: 10C. Scott Ananian) [17:04:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12491115 (10phaultfinder) [17:04:55] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12491116 (10phaultfinder) [17:04:59] !log pmiazga@deploy1003 mwscript-k8s job started: extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=kuwiki --logwiki=metawiki Bruhpedia Estoppel # T442867 [17:05:04] T442867: Unblock stuck global rename of Estoppel - https://phabricator.wikimedia.org/T442867 [17:05:38] (03CR) 10Cathal Mooney: [C:03+1] kubernetes::node: Fix zone detection logic (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1360488 (https://phabricator.wikimedia.org/T442874) (owner: 10JMeybohm) [17:05:58] (03Merged) 10jenkins-bot: ThumbExtractor: Only pass elements to Zest::matches() for captions [extensions/MultimediaViewer] (wmf/1.47.0-wmf.24) - 10https://gerrit.wikimedia.org/r/1360481 (https://phabricator.wikimedia.org/T441878) (owner: 10C. Scott Ananian) [17:06:05] hmm phab just threw AphrontConnectionLostQueryException: #2006: MySQL server has gone away [17:06:21] https://usercontent.irccloud-cdn.com/file/ehvuKf2I/Screenshot%202026-10-29%20at%2017.06.16.png [17:06:26] !log cscott@deploy1003 Started scap sync-world: Backport for [[gerrit:1360481|ThumbExtractor: Only pass elements to Zest::matches() for captions (T441878)]] (1.47.0-wmf.24) [17:07:23] it happened to me too for a moment, and it worked again after a refresh [17:07:40] (03CR) 10CDanis: Add a basic completion script for cookbooks (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/1359941 (https://phabricator.wikimedia.org/T367230) (owner: 10Lizimedia) [17:08:20] (03CR) 10CDanis: Add a basic completion script for cookbooks (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/1359941 (https://phabricator.wikimedia.org/T367230) (owner: 10Lizimedia) [17:09:42] A Troublesome Encounter! [17:09:42] Woe! This request had its journey cut short by unexpected circumstances (Can Not Connect to MySQL). [17:09:58] phab... [17:10:55] !log cscott@deploy1003 cscott: Backport for [[gerrit:1360481|ThumbExtractor: Only pass elements to Zest::matches() for captions (T441878)]] (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [17:11:02] T441878: Error: Call to undefined method Dom\Text::hasAttributes() - https://phabricator.wikimedia.org/T441878 [17:11:55] !log cscott@deploy1003 cscott: Continuing with deployment [17:12:03] (03CR) 10Blake: "Abandoning this, as the root cause of the disconnection was chased down in T442874 by Janis and Cathal, making this sensitivity change unn" [alerts] - 10https://gerrit.wikimedia.org/r/1360350 (owner: 10Blake) [17:12:19] (03Abandoned) 10Blake: bgp: increase trigger time for NodeBGPSessionStatusNotEstabilshed [alerts] - 10https://gerrit.wikimedia.org/r/1360350 (owner: 10Blake) [17:13:54] (03PS2) 10Catrope: admin: Add FIDO SSH keys for catrope [puppet] - 10https://gerrit.wikimedia.org/r/1358290 (https://phabricator.wikimedia.org/T442247) [17:14:47] (03CR) 10Catrope: "Updated with new keys following the new process. Also uploaded attestations for these through the Google Form" [puppet] - 10https://gerrit.wikimedia.org/r/1358290 (https://phabricator.wikimedia.org/T442247) (owner: 10Catrope) [17:15:39] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for catrope - https://phabricator.wikimedia.org/T442247#12491183 (10Catrope) I have regenerated my keys using the new process, uploaded the attestations through the Google Form, and updated the puppet patch with the new keys. [17:16:22] (03PS1) 10Jasmine: httpbb: Add assetlinks.json httpbb test [puppet] - 10https://gerrit.wikimedia.org/r/1360494 [17:17:42] !log cscott@deploy1003 Finished scap sync-world: Backport for [[gerrit:1360481|ThumbExtractor: Only pass elements to Zest::matches() for captions (T441878)]] (1.47.0-wmf.24) (duration: 11m 16s) [17:17:44] (03PS2) 10Jasmine: httpbb: Add assetlinks.json httpbb test [puppet] - 10https://gerrit.wikimedia.org/r/1360494 [17:17:48] T441878: Error: Call to undefined method Dom\Text::hasAttributes() - https://phabricator.wikimedia.org/T441878 [17:18:17] (03PS1) 10Jgreen: Switch fundraising default bastion to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1360495 [17:18:55] rzl: ok, over to you. thanks! [17:19:21] cscott: thanks! [17:19:45] (03PS2) 10JMeybohm: kubernetes::node: Fix zone detection logic [puppet] - 10https://gerrit.wikimedia.org/r/1360488 (https://phabricator.wikimedia.org/T442874) [17:19:59] (03CR) 10Jgreen: [C:03+2] Switch fundraising default bastion to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1360495 (owner: 10Jgreen) [17:20:01] (03PS3) 10Jasmine: httpbb: Add assetlinks.json httpbb test [puppet] - 10https://gerrit.wikimedia.org/r/1360494 (https://phabricator.wikimedia.org/T427929) [17:20:10] !log jgreen@dns1004 START - running authdns-update [17:20:19] (03CR) 10JMeybohm: "All very valid points, thanks @cmooney@wikimedia.org" [puppet] - 10https://gerrit.wikimedia.org/r/1360488 (https://phabricator.wikimedia.org/T442874) (owner: 10JMeybohm) [17:20:42] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [17:20:43] (03CR) 10Clément Goubert: [C:03+1] httpbb: Add assetlinks.json httpbb test [puppet] - 10https://gerrit.wikimedia.org/r/1360494 (https://phabricator.wikimedia.org/T427929) (owner: 10Jasmine) [17:21:04] (03CR) 10RLazarus: [C:03+2] mw-*: Update canary deployments to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360004 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [17:21:48] (03CR) 10RLazarus: [V:03+2 C:03+2] envoy: Update to v1.40.1 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1359939 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [17:22:13] (03CR) 10JMeybohm: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360488 (https://phabricator.wikimedia.org/T442874) (owner: 10JMeybohm) [17:22:30] (03CR) 10Cathal Mooney: [C:03+1] "Looks good, logic checks out should be a no-op on existing hosts I think." [puppet] - 10https://gerrit.wikimedia.org/r/1360488 (https://phabricator.wikimedia.org/T442874) (owner: 10JMeybohm) [17:22:47] !log jgreen@dns1004 END - running authdns-update [17:23:18] (03Merged) 10jenkins-bot: mw-*: Update canary deployments to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360004 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [17:23:58] (03CR) 10CI reject: [V:04-1] kubernetes::node: Fix zone detection logic [puppet] - 10https://gerrit.wikimedia.org/r/1360488 (https://phabricator.wikimedia.org/T442874) (owner: 10JMeybohm) [17:24:15] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.dhcp for host lvs3010.esams.wmnet (remove config only) [17:24:22] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host lvs3010.esams.wmnet (remove config only) [17:26:10] 10ops-eqiad, 06SRE, 06DC-Ops: eqiad: audit mgmt switches - https://phabricator.wikimedia.org/T441671#12491252 (10VRiley-WMF) |Name|Manufacturer|Type|Rack|Checked| |msw-a1-eqiad|NETGEAR|GS748T|A1 (061130.0101)|Yes| |msw-a2-eqiad|NETGEAR|GS748T|A2 (061130.0102)|Yes| |msw-a3-eqiad|NETGEAR|GS748T|A3 (061130.010... [17:27:22] !log vgutierrez@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on lvs3010.esams.wmnet with reason: host reimage [17:28:55] 10ops-eqiad, 06SRE, 06DC-Ops: eqiad: audit mgmt switches - https://phabricator.wikimedia.org/T441671#12491278 (10VRiley-WMF) Hey @ayounsi I completed the check of these switches [17:28:59] (03PS3) 10JMeybohm: kubernetes::node: Fix zone detection logic [puppet] - 10https://gerrit.wikimedia.org/r/1360488 (https://phabricator.wikimedia.org/T442874) [17:29:26] (03PS1) 10Jdlrobson: Skin: Update skin-responsive logic [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360497 (https://phabricator.wikimedia.org/T439273) [17:29:40] (03PS1) 10Jdlrobson: Skin: skin--responsive class no longer varies on user [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360498 (https://phabricator.wikimedia.org/T439273) [17:30:22] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [17:30:53] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360499 [17:31:44] (03PS1) 10Aqu: airflow-main: add phabricator token secret [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360500 (https://phabricator.wikimedia.org/T434047) [17:31:49] !log rzl@deploy1003 Started scap sync-world: 1360004 T436338 [17:31:50] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 02 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360497 (https://phabricator.wikimedia.org/T439273) (owner: 10Jdlrobson) [17:31:53] (03CR) 10JMeybohm: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360488 (https://phabricator.wikimedia.org/T442874) (owner: 10JMeybohm) [17:32:05] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 02 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360498 (https://phabricator.wikimedia.org/T439273) (owner: 10Jdlrobson) [17:32:46] !log rzl@deploy1003 rzl: 1360004 T436338 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [17:32:56] (03CR) 10Slyngshede: [C:03+1] "Verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1359924 (https://phabricator.wikimedia.org/T442741) (owner: 10Xcollazo) [17:33:17] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs3010.esams.wmnet with reason: host reimage [17:34:41] (03CR) 10Clément Goubert: [C:03+1] zarcillo: enabling egress to config-master [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360470 (https://phabricator.wikimedia.org/T442904) (owner: 10CWilliams) [17:35:07] (03CR) 10Dzahn: [V:03+1] "verified new keys out of band and as yubikey only" [puppet] - 10https://gerrit.wikimedia.org/r/1358290 (https://phabricator.wikimedia.org/T442247) (owner: 10Catrope) [17:35:14] !log rzl@deploy1003 rzl: Continuing with deployment [17:35:20] (03CR) 10Dzahn: [V:03+1 C:03+2] admin: Add FIDO SSH keys for catrope [puppet] - 10https://gerrit.wikimedia.org/r/1358290 (https://phabricator.wikimedia.org/T442247) (owner: 10Catrope) [17:36:56] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for catrope - https://phabricator.wikimedia.org/T442247#12491318 (10Dzahn) 05In progress→03Resolved a:03Dzahn puppet change merged! new keys will be deployed on all relevant hosts within the next 30 min. [17:37:35] !log rzl@deploy1003 Finished scap sync-world: 1360004 T436338 (duration: 06m 40s) [17:38:23] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for catrope - https://phabricator.wikimedia.org/T442247#12491325 (10Dzahn) 05Resolved→03Open wait, should this stay open until the old key is removed? [17:38:55] (03CR) 10Blake: [C:03+1] kubernetes::node: Fix zone detection logic [puppet] - 10https://gerrit.wikimedia.org/r/1360488 (https://phabricator.wikimedia.org/T442874) (owner: 10JMeybohm) [17:39:04] (03CR) 10Jasmine: [C:03+2] httpbb: Add assetlinks.json httpbb test [puppet] - 10https://gerrit.wikimedia.org/r/1360494 (https://phabricator.wikimedia.org/T427929) (owner: 10Jasmine) [17:39:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12491342 (10phaultfinder) [17:40:38] (03CR) 10Dzahn: [C:03+1] "this is failing on phab2003. probably needs a follow-up to only do it on active host, similar to other services in the past. will look at " [puppet] - 10https://gerrit.wikimedia.org/r/1359770 (https://phabricator.wikimedia.org/T442177) (owner: 10Jelto) [17:40:40] 06SRE, 06Infrastructure-Foundations, 10netops, 06Traffic: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12491346 (10ssingh) Great work all around, folks! <3 [17:49:49] (03PS1) 10TChin: [eventstreams] Bump to v0.22.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360503 (https://phabricator.wikimedia.org/T383977) [17:53:24] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs3010.esams.wmnet with OS trixie [17:53:34] (03CR) 10Jasmine: [C:03+2] apache: Configure wikimedia.org redirect in apache, remove from redirects.dat [puppet] - 10https://gerrit.wikimedia.org/r/1333298 (https://phabricator.wikimedia.org/T427929) (owner: 10Jasmine) [17:56:10] belatedly, infra window complete 👍 I'll be bumping some more envoys but nothing that conflicts with other deployments [17:57:09] (03PS7) 10Jasmine: apache: Configure wikimedia.org redirect in apache, remove from redirects.dat [puppet] - 10https://gerrit.wikimedia.org/r/1333298 (https://phabricator.wikimedia.org/T427929) [17:57:24] (03PS2) 10TChin: [eventstreams] Bump to v0.22.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360503 (https://phabricator.wikimedia.org/T383977) [17:57:26] (03CR) 10Clément Goubert: [C:03+1] apache: Configure wikimedia.org redirect in apache, remove from redirects.dat [puppet] - 10https://gerrit.wikimedia.org/r/1333298 (https://phabricator.wikimedia.org/T427929) (owner: 10Jasmine) [18:00:04] dduvall and dancy: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for MediaWiki train - Utc-7 Version deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T1800). [18:00:55] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for catrope - https://phabricator.wikimedia.org/T442247#12491461 (10Catrope) >>! In T442247#12491325, @Dzahn wrote: > wait, should this stay open until the old key is removed? Yes let's do that [18:02:33] o/ [18:02:55] (03CR) 10Jasmine: [C:03+2] apache: Configure wikimedia.org redirect in apache, remove from redirects.dat [puppet] - 10https://gerrit.wikimedia.org/r/1333298 (https://phabricator.wikimedia.org/T427929) (owner: 10Jasmine) [18:04:06] quick backport ongoing dancy sorry we're running over on an apache config [18:04:13] well about to be ongoing [18:04:26] no image rebuild at all, should be quick'ish [18:09:11] (03PS2) 10Kamila Součková: Revert^2 "mw-parsoid: temporarily override image to PHP 8.5 in eqiad" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360482 [18:10:23] (03CR) 10Kamila Součková: [C:03+2] "done" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360482 (owner: 10Kamila Součková) [18:10:40] 10ops-esams, 06SRE, 06DC-Ops, 06Traffic: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859#12491542 (10ssingh) Thank you for taking care of this so quickly, @RobH. [18:11:42] !log kamila@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [18:12:05] (03Merged) 10jenkins-bot: Revert^2 "mw-parsoid: temporarily override image to PHP 8.5 in eqiad" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360482 (owner: 10Kamila Součková) [18:12:13] (03PS1) 10CDanis: fundraising_data_import: bad upload file glob match [puppet] - 10https://gerrit.wikimedia.org/r/1360507 (https://phabricator.wikimedia.org/T416948) [18:12:26] !log kamila@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [18:12:30] (03CR) 10CDanis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360507 (https://phabricator.wikimedia.org/T416948) (owner: 10CDanis) [18:12:35] !log kamila@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [18:13:15] !log kamila@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [18:13:32] !log kamila@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [18:14:08] !log kamila@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [18:15:07] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:15:35] 10ops-esams, 06SRE, 06DC-Ops, 06Traffic: lvs3010 unable to boot via PXE with error issue: Comm Error: Backplane 0. - https://phabricator.wikimedia.org/T442859#12491556 (10Vgutierrez) Thx. Host has been reimaged successfully [18:15:58] (03CR) 10CDanis: [C:03+2] fundraising_data_import: bad upload file glob match [puppet] - 10https://gerrit.wikimedia.org/r/1360507 (https://phabricator.wikimedia.org/T416948) (owner: 10CDanis) [18:17:12] (03PS1) 10TrainBranchBot: group2 to 1.48.0-wmf.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360508 (https://phabricator.wikimedia.org/T438223) [18:17:15] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by dduvall@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360508 (https://phabricator.wikimedia.org/T438223) (owner: 10TrainBranchBot) [18:18:02] (03Merged) 10jenkins-bot: group2 to 1.48.0-wmf.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360508 (https://phabricator.wikimedia.org/T438223) (owner: 10TrainBranchBot) [18:18:09] PROBLEM - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:18:33] (03CR) 10Jgleeson: [C:03+1] "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/1360507 (https://phabricator.wikimedia.org/T416948) (owner: 10CDanis) [18:18:47] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:19:47] (03CR) 10MSantos: [V:04-1] "@krinkle@fastmail.com after discussing with Birgit and Olga we decided that we need to do a tech news announcement and deploy it next week" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311061 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [18:26:10] dancy: getting testserver check failures but looking into it [18:27:57] PROBLEM - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:30:34] (03PS4) 10JMeybohm: kubernetes::node: Fix zone detection logic [puppet] - 10https://gerrit.wikimedia.org/r/1360488 (https://phabricator.wikimedia.org/T442874) [18:30:53] Those alerts look related. [18:31:00] (03CR) 10JMeybohm: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360488 (https://phabricator.wikimedia.org/T442874) (owner: 10JMeybohm) [18:31:48] (03PS1) 10Clément Goubert: apache: Fix wikimedia.org redirects [puppet] - 10https://gerrit.wikimedia.org/r/1360512 (https://phabricator.wikimedia.org/T427929) [18:32:24] dduvall: they're probably me [18:32:26] go through [18:32:52] a test is borked because the config that makes it not borked is not correct but also not merged yet [18:33:57] claime: ack, ty! i'll continue [18:39:41] claime: fyi, it's odd because the `docroot/wikimedia.org/.well-known/assetlinks.json` on the deployment server looks right [18:39:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12491708 (10phaultfinder) [18:40:38] dduvall: it breaks because currently it's a 301 and not a 200 [18:40:50] !log dduvall@deploy1003 rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.1 refs T438223 [18:40:56] T438223: 1.48.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T438223 [18:40:58] claime: ah, right. ok [18:41:18] Congrats on completing the crisis train! [18:41:43] (03PS2) 10Clément Goubert: apache: Fix wikimedia.org redirects [puppet] - 10https://gerrit.wikimedia.org/r/1360512 (https://phabricator.wikimedia.org/T427929) [18:42:27] dduvall: and the patch is taking me a minute because I broke redirects.dat and need to reimplement apache rewriterules T_T [18:42:38] see the PS that I just pushed... [18:43:05] (03PS3) 10Clément Goubert: apache: Fix wikimedia.org redirects [puppet] - 10https://gerrit.wikimedia.org/r/1360512 (https://phabricator.wikimedia.org/T427929) [18:44:29] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:45:19] dancy: ty! buena suerte y solidaridad a j.nuche for next week's train :| [18:45:25] claime: ack [18:45:26] haha [18:47:21] (03CR) 10RLazarus: [C:03+1] apache: Fix wikimedia.org redirects (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1360512 (https://phabricator.wikimedia.org/T427929) (owner: 10Clément Goubert) [18:49:58] (03CR) 10Clément Goubert: [C:03+2] apache: Fix wikimedia.org redirects [puppet] - 10https://gerrit.wikimedia.org/r/1360512 (https://phabricator.wikimedia.org/T427929) (owner: 10Clément Goubert) [18:53:52] 06SRE, 06Infrastructure-Foundations: Create nodejs 26 production images - https://phabricator.wikimedia.org/T437509#12491763 (10Jdforrester-WMF) 05Open→03Resolved Let's call this (long) done. [18:54:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12491769 (10phaultfinder) [18:55:22] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [18:57:38] RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:et-1/0/2 (Transport: cr1-eqiad:et-1/1/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [18:57:39] RESOLVED: CoreBGPDown: Core BGP session down between cr1-codfw and cr1-eqiad (208.80.153.220) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqiad&var-bgp_neighbor=cr1-eqiad - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [19:00:26] !log jasmine@deploy1003 Started scap sync-world: Deploying 1360512 T427929 [19:00:33] T427929: Host assetlinks.json on root wikimedia.org domain - https://phabricator.wikimedia.org/T427929 [19:03:38] !log jasmine@deploy1003 jasmine: Deploying 1360512 T427929 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:05:11] PROBLEM - Check unit status of fundraising-data-import on deploy1003 is CRITICAL: CRITICAL: Status of the systemd unit fundraising-data-import https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [19:09:01] !log jasmine@deploy1003 Scap cancelled without rolling back. [19:11:17] (03PS1) 10Ottomata: EventStreamConfig: Add revision_html_feature_counts_change.dev0 stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360516 (https://phabricator.wikimedia.org/T436695) [19:15:00] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12491912 (10phaultfinder) [19:16:38] 10ops-eqiad, 06SRE, 06DC-Ops, 10ServiceOps-Upgrades-Hardware, 06ServiceOps (Next quarter): Q1:rack/setup/install conf101[0-2] - https://phabricator.wikimedia.org/T435426#12491922 (10Jclark-ctr) a:03Jclark-ctr These servers have Arrived [19:16:42] (03PS1) 10Clément Goubert: apache: also fix *.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1360518 (https://phabricator.wikimedia.org/T427929) [19:17:50] (03PS1) 10Ahmon Dancy: listTaskCounts: Do not send negative task counts to stats [extensions/GrowthExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360521 (https://phabricator.wikimedia.org/T394969) [19:18:07] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q1:rack/setup/install puppetserver100[45] and ganeti106[0-4] - https://phabricator.wikimedia.org/T434699#12491940 (10Jclark-ctr) These have Arrived on site [19:22:26] (03PS2) 10Clément Goubert: apache: also fix *.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1360518 (https://phabricator.wikimedia.org/T427929) [19:23:05] (03CR) 10Clément Goubert: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360518 (https://phabricator.wikimedia.org/T427929) (owner: 10Clément Goubert) [19:24:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12491961 (10phaultfinder) [19:26:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:26:25] claime: Ping me when I can deploy a backport (1360521) [19:26:39] (log noise reduction) [19:28:05] (03CR) 10AKhatun: [C:03+1] EventStreamConfig: Add revision_html_feature_counts_change.dev0 stream [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360516 (https://phabricator.wikimedia.org/T436695) (owner: 10Ottomata) [19:28:25] (03PS3) 10Clément Goubert: apache: also fix *.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1360518 (https://phabricator.wikimedia.org/T427929) [19:29:00] (03PS1) 10Catrope: Remove old SSH key for catrope [puppet] - 10https://gerrit.wikimedia.org/r/1360523 (https://phabricator.wikimedia.org/T442247) [19:29:11] dancy: ack [19:29:11] (03PS2) 10Catrope: Remove old SSH key for catrope [puppet] - 10https://gerrit.wikimedia.org/r/1360523 (https://phabricator.wikimedia.org/T442247) [19:29:25] (03CR) 10Jforrester: [C:03+2] chromium-render: Drop --single-process, add tini, up to 2026-10-28-215517-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359960 (https://phabricator.wikimedia.org/T441276) (owner: 10Jforrester) [19:29:46] (03CR) 10Subramanya Sastry: [C:03+1] chromium-render: Drop --single-process, add tini, up to 2026-10-28-215517-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359960 (https://phabricator.wikimedia.org/T441276) (owner: 10Jforrester) [19:31:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:31:19] (03Merged) 10jenkins-bot: chromium-render: Drop --single-process, add tini, up to 2026-10-28-215517-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359960 (https://phabricator.wikimedia.org/T441276) (owner: 10Jforrester) [19:31:24] (03PS4) 10Clément Goubert: apache: also fix *.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1360518 (https://phabricator.wikimedia.org/T427929) [19:31:59] (03PS1) 10DLynch: Remove wgVisualEditorDisableForAnons for enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360524 (https://phabricator.wikimedia.org/T442933) [19:32:14] !log jforrester@deploy1003 helmfile [staging] START helmfile.d/services/proton: apply [19:32:33] (03CR) 10RLazarus: [C:03+1] apache: also fix *.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1360518 (https://phabricator.wikimedia.org/T427929) (owner: 10Clément Goubert) [19:32:37] !log jforrester@deploy1003 helmfile [staging] DONE helmfile.d/services/proton: apply [19:32:40] !log jforrester@deploy1003 helmfile [staging] START helmfile.d/services/proton: apply [19:32:42] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, October 29 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360524 (https://phabricator.wikimedia.org/T442933) (owner: 10DLynch) [19:33:09] jouncebot: nowandnext [19:33:09] For the next 0 hour(s) and 26 minute(s): MediaWiki train - Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T1800) [19:33:09] In 0 hour(s) and 26 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T2000) [19:33:20] !log jforrester@deploy1003 helmfile [staging] DONE helmfile.d/services/proton: apply [19:34:03] !log jforrester@deploy1003 helmfile [eqiad] START helmfile.d/services/proton: apply [19:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:34:50] (03CR) 10Clément Goubert: [C:03+2] apache: also fix *.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1360518 (https://phabricator.wikimedia.org/T427929) (owner: 10Clément Goubert) [19:35:37] !log jforrester@deploy1003 helmfile [eqiad] DONE helmfile.d/services/proton: apply [19:35:42] !log jforrester@deploy1003 helmfile [staging] START helmfile.d/services/proton: apply [19:35:52] !log jforrester@deploy1003 helmfile [staging] DONE helmfile.d/services/proton: apply [19:36:01] !log jforrester@deploy1003 helmfile [codfw] START helmfile.d/services/proton: apply [19:37:15] !log jforrester@deploy1003 helmfile [codfw] DONE helmfile.d/services/proton: apply [19:39:36] claime: dancy is window clear? I have a non urgent mw config change I want to deploy. [19:39:57] ottomata: not clear [19:40:05] ack [19:41:48] (03PS1) 10Jforrester: Revert "chromium-render: Drop --single-process, add tini, up to 2026-10-28-215517-production" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360525 (https://phabricator.wikimedia.org/T441276) [19:41:58] (03CR) 10Jforrester: [C:03+2] Revert "chromium-render: Drop --single-process, add tini, up to 2026-10-28-215517-production" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360525 (https://phabricator.wikimedia.org/T441276) (owner: 10Jforrester) [19:43:14] !log jasmine@deploy1003 Started scap sync-world: Deploying T427929 1360518 [19:43:19] T427929: Host assetlinks.json on root wikimedia.org domain - https://phabricator.wikimedia.org/T427929 [19:45:02] (03Merged) 10jenkins-bot: Revert "chromium-render: Drop --single-process, add tini, up to 2026-10-28-215517-production" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360525 (https://phabricator.wikimedia.org/T441276) (owner: 10Jforrester) [19:46:04] !log jforrester@deploy1003 helmfile [staging] START helmfile.d/services/proton: apply [19:46:24] !log jasmine@deploy1003 jasmine: Deploying T427929 1360518 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:47:10] !log jasmine@deploy1003 jasmine: Continuing with deployment [19:48:16] (03CR) 10Dzahn: [C:03+2] Remove old SSH key for catrope [puppet] - 10https://gerrit.wikimedia.org/r/1360523 (https://phabricator.wikimedia.org/T442247) (owner: 10Catrope) [19:48:42] (03CR) 10Dzahn: [C:03+2] "sure, removing keys is always an easier merge than adding them :)" [puppet] - 10https://gerrit.wikimedia.org/r/1360523 (https://phabricator.wikimedia.org/T442247) (owner: 10Catrope) [19:52:55] !log jasmine@deploy1003 Finished scap sync-world: Deploying T427929 1360518 (duration: 10m 21s) [19:53:02] T427929: Host assetlinks.json on root wikimedia.org domain - https://phabricator.wikimedia.org/T427929 [19:54:06] (03CR) 10Clément Goubert: [C:03+2] "@jkilani@wikimedia.org followup tweaks" [puppet] - 10https://gerrit.wikimedia.org/r/1360518 (https://phabricator.wikimedia.org/T427929) (owner: 10Clément Goubert) [19:54:24] dancy: i'm done, you can go ahead, ping ottomata when you're done [19:54:33] Will do. [19:55:00] (03CR) 10TrainBranchBot: [C:03+2] "Approved by dancy@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360521 (https://phabricator.wikimedia.org/T394969) (owner: 10Ahmon Dancy) [19:56:13] !log jforrester@deploy1003 helmfile [staging] DONE helmfile.d/services/proton: apply [19:56:39] (03Merged) 10jenkins-bot: listTaskCounts: Do not send negative task counts to stats [extensions/GrowthExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360521 (https://phabricator.wikimedia.org/T394969) (owner: 10Ahmon Dancy) [19:56:59] !log dancy@deploy1003 Started scap sync-world: Backport for [[gerrit:1360521|listTaskCounts: Do not send negative task counts to stats (T394969)]] (1.48.0-wmf.1) [19:57:06] T394969: [0.5day timebox] PHP Warning: Stats: got negative value for counter "tasktype_count" - https://phabricator.wikimedia.org/T394969 [19:59:51] (03PS2) 10Jforrester: proton: Bump chart number, as revert didn't take [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360530 [20:00:05] RoanKattouw, urbanecm, TheresNoTime, and cjming: That opportune time for a UTC late backport window deploy is upon us again. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T2000). [20:00:05] Krinkle, Thiemo_WMDE, Neriah, arlolra, and kemayo: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [20:00:10] o/ [20:00:15] hi [20:00:16] (03CR) 10Jforrester: [C:03+2] proton: Bump chart number, as revert didn't take [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360530 (owner: 10Jforrester) [20:00:28] o/ [20:00:38] o/ [20:00:46] i need a deployer... [20:00:46] arlolra: I can batch yours with mine if you like (both mw wmf.1) [20:00:58] sure, thanks! [20:01:09] !log dancy@deploy1003 dancy: Backport for [[gerrit:1360521|listTaskCounts: Do not send negative task counts to stats (T394969)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [20:01:10] !log jforrester@deploy1003 helmfile [staging] START helmfile.d/services/proton: apply [20:01:14] !log jforrester@deploy1003 helmfile [staging] DONE helmfile.d/services/proton: apply [20:01:46] o/ [20:01:49] !log cscott@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [20:02:01] (03Merged) 10jenkins-bot: proton: Bump chart number, as revert didn't take [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360530 (owner: 10Jforrester) [20:02:18] !log cscott@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [20:02:24] !log dancy@deploy1003 dancy: Continuing with deployment [20:02:26] awaiting dancy 's lock first , there's another deploy ongoing still. presumably a train blocker fix [20:03:03] I may postpone mine in that case, since its a performance fi that I'd like to roll everywhere at once to have clearer metrics [20:03:26] (03CR) 10Dzahn: [V:03+1 C:03+1] "can you please update it to only add the yubikey but not the "secretive" key? Then we can go ahead with this. Or I can do it later today." [puppet] - 10https://gerrit.wikimedia.org/r/1358237 (https://phabricator.wikimedia.org/T442092) (owner: 10Rsilvola) [20:03:39] Mine will be done soon. ottomata is next in line. [20:03:43] !log jforrester@deploy1003 helmfile [staging] START helmfile.d/services/proton: apply [20:03:55] !log jforrester@deploy1003 helmfile [staging] DONE helmfile.d/services/proton: apply [20:04:25] (03CR) 10Dzahn: [V:03+1 C:03+2] admin: Add FIDO SSH key for isaranto [puppet] - 10https://gerrit.wikimedia.org/r/1358219 (https://phabricator.wikimedia.org/T442176) (owner: 10Ilias Sarantopoulos) [20:04:28] oof.. dbs overloaded. [20:04:41] !log jforrester@deploy1003 helmfile [eqiad] START helmfile.d/services/proton: apply [20:04:43] PROBLEM - MariaDB Event Scheduler pc7 on pc2017 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [20:04:43] PROBLEM - MariaDB Replica IO: pc7 on pc2017 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:04:44] PROBLEM - MariaDB read only pc7 #page on pc2017 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [20:04:44] PROBLEM - MariaDB Event Scheduler pc5 on pc2015 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [20:05:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:05:08] PROBLEM - MariaDB read only pc5 #page on pc2015 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [20:05:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 6.204% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:05:21] PROBLEM - MariaDB Replica SQL: pc5 on pc2015 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:05:29] PROBLEM - MariaDB Replica IO: pc5 on pc1015 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 1040, Errmsg: error connecting to master repl2024@pc2015.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Too many connections https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:05:50] !log jforrester@deploy1003 helmfile [eqiad] DONE helmfile.d/services/proton: apply [20:05:51] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://proton.svc.codfw.wmnet:4030 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [20:06:00] !log jforrester@deploy1003 helmfile [codfw] START helmfile.d/services/proton: apply [20:06:10] !log dancy@deploy1003 Rolling back deployment [20:06:22] Might make more sense to postpone mine once more. The other patches appear more important. [20:06:30] hi, looking, deployers can you-- thanks :) [20:06:31] PROBLEM - MariaDB Replica SQL: pc7 on pc2017 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:06:35] PROBLEM - MariaDB Event Scheduler pc7 on pc2017 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [20:06:36] PROBLEM - MariaDB read only pc7 #page on pc2017 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [20:06:39] PROBLEM - MariaDB Replica IO: pc7 on pc2017 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:06:41] PROBLEM - MariaDB Replica IO: pc5 on pc2015 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:06:41] PROBLEM - MariaDB Event Scheduler pc5 on pc2015 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [20:06:46] after the rollback please hold off deploying anything else [20:06:51] That really doesn't look like a change that should have caused issues. 😬 [20:07:08] !log jforrester@deploy1003 helmfile [codfw] DONE helmfile.d/services/proton: apply [20:07:10] can always be a coincidence but let's remove variables [20:07:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [20:07:16] Probably not. Just wanna get out of the way. [20:07:16] PROBLEM - MariaDB read only pc5 #page on pc2015 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [20:07:16] PROBLEM - MariaDB Replica SQL: pc5 on pc2015 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:08:15] Ok, I removed mine from the queue for now. Good luck! [20:08:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 3.72% idle #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:08:16] !log cscott@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [20:08:42] !log cscott@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [20:09:24] !log dancy@deploy1003 Failed scap sync-world: Backport for [[gerrit:1360521|listTaskCounts: Do not send negative task counts to stats (T394969)]] (1.48.0-wmf.1) (duration: 12m 25s) [20:09:30] T394969: [0.5day timebox] PHP Warning: Stats: got negative value for counter "tasktype_count" - https://phabricator.wikimedia.org/T394969 [20:09:52] rzl: Wanna grab the scap lock? [20:10:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:10:15] cscott: are you here? [20:10:16] FIRING: [2x] MediaWikiLatencyExceeded: p75 latency high: codfw mw-api-int releases routed via main (k8s) 988.7ms - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [20:10:51] RESOLVED: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://proton.svc.codfw.wmnet:4030 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [20:10:52] cscott: pc hosts are paging, coincident with your mw-parsoid deploys, what's rolling out? [20:10:58] FIRING: ProbeDown: Service text-https:443 has failed probes (http_text-https_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:11:33] PROBLEM - MariaDB Replica Lag: pc5 on pc2015 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:11:33] PROBLEM - MariaDB Event Scheduler pc7 on pc2017 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [20:11:36] PROBLEM - MariaDB read only pc7 #page on pc2017 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [20:11:37] dancy: I'm fine just coordinating in here, but if you think it's a good idea, by all means [20:11:47] PROBLEM - MariaDB Replica IO: pc7 on pc2017 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:12:15] FIRING: [4x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [20:12:22] !log dancy@deploy1003 Locking from deployment [MediaWiki]: Holding until database problems are resolved [20:12:33] PROBLEM - MariaDB Replica SQL: pc7 on pc2017 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:12:39] PROBLEM - MariaDB Replica Lag: pc7 on pc2017 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:12:47] Holding the lock since I haven't reverted the change I was trying to deploy. [20:12:56] ack, thanks [20:13:01] PROBLEM - MariaDB Replica Lag: pc5 on pc1015 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 630.51 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:13:02] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Q1:rack/setup/install mc20[56-73] - https://phabricator.wikimedia.org/T436272#12492212 (10Papaul) [20:13:12] PROBLEM - MariaDB read only pc5 #page on pc2015 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [20:13:29] RECOVERY - MariaDB Replica IO: pc5 on pc1015 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:14:15] PROBLEM - MariaDB Replica SQL: pc5 on pc2015 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:14:33] PROBLEM - MariaDB Replica Lag: pc5 on pc2015 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:14:35] PROBLEM - MariaDB Replica IO: pc5 on pc2015 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:14:49] Raine: do you have state on the mw-parsoid work maybe? [20:15:04] rzl: yes, what did I do? [20:15:05] should I just call cscott's phone? [20:15:07] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [20:15:08] PROBLEM - MariaDB read only pc5 #page on pc2015 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [20:15:15] FIRING: [3x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at codfw: 19.81% idle - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:15:16] RESOLVED: [2x] MediaWikiLatencyExceeded: p75 latency high: codfw mw-api-int releases routed via main (k8s) 1.244s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [20:15:30] Getting a "Cannot access the database: Cannot access the database: Database servers in cluster31 are overloaded. In order to protect application servers, the circuit breaking to databases of this section have been activated. Please try again a few seconds." sometimes right now; I don't see anything on wmstatus so thought I should drop this here [20:15:32] Raine: no idea if anything, just seeing a mw-parsoid deploy at the same time as the pc* host pages, wondering if it's related [20:15:35] PROBLEM - MariaDB Event Scheduler pc5 on pc2015 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [20:15:37] mskmskmsk: we're working on it, thanks :) [20:15:45] brett: I wouldn't call yet [20:15:46] FIRING: [2x] MediaWikiLatencyExceeded: p75 latency high: codfw mw-api-int releases routed via main (k8s) 1.129s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [20:15:48] rzl: kk, thanks [20:15:57] RESOLVED: ProbeDown: Service text-https:443 has failed probes (http_text-https_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:16:08] rzl: lemme look at logs... [20:17:09] actually wait, mw-parsoid is just a testing deployment, right? nothing hits it on the live path? [20:17:14] PROBLEM - MariaDB read only pc5 #page on pc2015 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [20:17:19] god I wish we had clearer names for these, that was a red herring [20:17:31] rzl: the 2 pc sections each have the codfw half showing as down in orchestrator [20:17:31] sorry Raine [20:17:43] PROBLEM - MariaDB Replica Lag: pc7 on pc2017 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:17:43] PROBLEM - MariaDB Replica IO: pc7 on pc2017 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:17:43] PROBLEM - MariaDB Event Scheduler pc7 on pc2017 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [20:17:44] PROBLEM - MariaDB read only pc7 #page on pc2017 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [20:17:49] oh it's just codfw it's fi--- oh wait [20:17:56] cezmunsta: hi! yeah, because it's both I'm assuming that's not a hardware issue, right [20:18:07] rzl: nothing hits mw-parsoid on the live path, but they're currently hammering them [20:18:09] RECOVERY - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [20:18:11] but also it's 2 replicas [20:18:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 4.516% idle #page - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:18:35] PROBLEM - MariaDB Replica SQL: pc7 on pc2017 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:18:47] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [20:18:50] ...but also they're parsing all the things [20:19:08] PROBLEM - MariaDB read only pc5 #page on pc2015 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [20:19:09] rzl: not sure, as not at laptop - we had a couple of boxes crash today [20:19:09] okay, that does sound both like a possible trigger and like something we can forcibly roll back without harming live traffic [20:19:17] cezmunsta: no worries, thank you [20:19:30] rzl: should I tell cscott to murder the test? [20:19:39] Raine: if you're talking to him, yes please [20:20:01] I haven't been getting an answer, so if you don't hear back either, we should go ahead and kill it [20:20:07] yeah, we just started rt testing against PHP 8.5 [20:20:15] RESOLVED: [3x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at codfw: 23.05% idle - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [20:20:20] it should just be hitting mw-experimental-parsoid [20:20:34] ah hi! yeah if that started a little after 20:00, do stop it right away please [20:20:46] RESOLVED: [2x] MediaWikiLatencyExceeded: p75 latency high: codfw mw-api-int releases routed via main (k8s) 1.03s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [20:20:48] it might only be hitting that mediawiki but it's hitting the shared parsercache [20:20:48] cscott: what I'm wondering about is whether it's hitting the DBs too hard for the stuff it's re-parsing [20:21:21] this is affecting live traffic so please turn it off now, and we can discuss more afterward [20:21:25] cscott: kill it [20:21:33] PROBLEM - MariaDB Replica SQL: pc7 on pc2017 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:21:35] PROBLEM - MariaDB Event Scheduler pc5 on pc2015 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [20:21:39] PROBLEM - MariaDB Replica IO: pc7 on pc2017 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:21:58] rzl: ok stopped [20:22:05] thanks <3 and sorry [20:22:08] PROBLEM - MariaDB read only pc5 #page on pc2015 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [20:22:09] thanks! watching to see, if things don't recover we'll know you were innocent [20:22:20] pc1015 - can reach via ssh, mariadb is running [20:22:23] (expecting some delay anyway as outstanding queries continue) [20:22:33] mutante: it is codfw to look at [20:22:37] RECOVERY - MariaDB Event Scheduler pc5 on pc2015 is OK: Version 10.11.16-MariaDB-log, Uptime 36s, read_only: False, event_scheduler: True, 4612.55 QPS, connection latency: 0.021647s, query latency: 0.001027s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [20:22:58] first parse was: [20:22:58] Oct 29 20:08:54 ctt-rt-testing-01 nodejs[1799]: -> commonswiki:Duomo di Verona [20:23:01] RECOVERY - MariaDB Replica Lag: pc5 on pc1015 is OK: OK slave_sql_lag Replication lag: 37.06 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:23:04] cezmunsta: ACK but that also alerted. ^ [20:23:07] yeah, that timing does line up [20:23:09] just the lag [20:23:10] RECOVERY - MariaDB read only pc5 #page on pc2015 is OK: Version 10.11.16-MariaDB-log, Uptime 68s, read_only: False, event_scheduler: True, 4848.15 QPS, connection latency: 0.019053s, query latency: 0.000490s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [20:23:22] mutante: yes, they are co-masters [20:23:23] well 💩 [20:23:59] mw latency and errors are recovering, looks like that fixed it :) [20:24:17] hm, weird. fwiw if there's a place to squirrel this away for future reference, https://www.mediawiki.org/wiki/Parsoid/Round-trip_testing#RT-testing_setup says how to stop an RT run in progress. [20:24:26] starting laptop to check the 2 [20:24:39] this is very weird, these are run regularly, right? [20:24:45] FIRING: CirrusConsumerRerenderFetchErrorRate: cirrus_streaming_updater_consumer_search_eqiad in eqiad (k8s): ... [20:24:45] fetch error (rerenders) rate too high - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=eqiad+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusConsumerRerenderFetchErrorRate [20:24:51] yes, at least once a week, usually more. [20:25:07] and codfw is still php 8.3, so nothing should be different there [20:25:09] usually on a friday, but we got an early start this week since we skipped last week [20:25:12] maybe some kind of config change related to the 8.3<->8.5 back and forth-- oh [20:25:30] eqiad is 8.5, codfw 8.3 -- we were comparing the two [20:25:35] ahh [20:25:42] this was targetting eqiad only because that's the machine that 8.5 yeah, but i think eqiad is our default target as well? [20:25:54] oh, so maybe codfw capacity [20:25:56] yeah and even then I wouldn't expect a difference in impact [20:26:01] pc is smaller in codfw, or not? [20:26:15] they are co-master per section [20:26:17] I think it should be identical, cezmunsta can confirm [20:26:35] 1 in each DC [20:26:35] PROBLEM - MariaDB Replica Lag: pc7 on pc2017 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:27:15] RESOLVED: [4x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [20:27:57] RECOVERY - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [20:28:07] oh actually the default was usually codfw, so this is unusual in targeting eqiad. [20:28:48] but the hardware is not identical in eqiad vs codfw [20:28:53] cscott: Raine please check _sec [20:29:09] pc2011 = R440 pc1012 = R660 f.e. [20:29:19] fwiw, i had also started visual diff testing 8.3 vs 8.5 (and i stopped that as well when I got the slack alert) .. but again it is low volume like rt-testing and we run both multiple times a week without problem. [20:29:45] RESOLVED: CirrusConsumerRerenderFetchErrorRate: cirrus_streaming_updater_consumer_search_eqiad in eqiad (k8s): ... [20:29:50] fetch error (rerenders) rate too high - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=eqiad+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusConsumerRerenderFetchErrorRate [20:30:21] (03PS1) 10Bking: cirrussearch: Tune pool counter rejection alert [alerts] - 10https://gerrit.wikimedia.org/r/1360534 (https://phabricator.wikimedia.org/T442912) [20:30:30] 06SRE, 10SRE-Access-Requests, 06Product Safety and Integrity, 06Security-Team, 13Patch-For-Review: Replace SSH key with hw backed SSH keys for rsilvola - https://phabricator.wikimedia.org/T442092#12492272 (10Rsilvola) a:03Rsilvola Waiting on me to regenerate the key with attestation as per the new proc... [20:30:44] (03PS1) 10Ahmon Dancy: Revert "listTaskCounts: Do not send negative task counts to stats" [extensions/GrowthExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360535 [20:30:52] (03CR) 10Ahmon Dancy: [C:03+2] Revert "listTaskCounts: Do not send negative task counts to stats" [extensions/GrowthExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360535 (owner: 10Ahmon Dancy) [20:30:53] rt testing "usually" runs with "X-Wikimedia-Debug": "k8s-mw-parsoid-codfw" and visual diff testing "usually" runs without an x-wikimedia-debug heading at all (just ?useparsoid=0/1 in the url). Subbu initially started visual diff testing targetting k8s-...-next but realized that was 8.3 and then restarted targetting k8s-mw-parsoid-eqiad like the RT testing (right?) [20:31:05] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [20:31:07] so wait, rt testing was only hitting eqiad now? [20:31:11] pc2015 shows as having lost power (I think) [20:31:17] so how the heck did _codfw_ get happier [20:31:22] effie: sorry i don't know what/where _sec is [20:31:33] cscott: yes. [20:32:36] Raine: yes, *in theory* we were only hitting eqiad. Always possible something got screwed up? [20:32:58] you're using X-W-D header to target it though, right? [20:33:07] so unless something is very wrong, that should work [20:33:07] yes [20:33:20] visualdiff makes 2 requests per test .. one for php8.3 without a X-W-D header, and one for php8.5 with a X-W-D header of k8s-mw-parsoid-eqiad [20:33:21] (03CR) 10CI reject: [V:04-1] cirrussearch: Tune pool counter rejection alert [alerts] - 10https://gerrit.wikimedia.org/r/1360534 (https://phabricator.wikimedia.org/T442912) (owner: 10Bking) [20:33:24] just that: even if you hit mw-debug, it still hit actual parsercache and can overload it [20:34:04] what's the default datacenter these days? is it possible there was high load due to some cross-dc issue? (although we'd been targetting "just" codfw for several months. *Maybe* we haven't had a data center switchover since we started using the new setup?) [20:34:16] we just did the switchover, yeah [20:34:22] cezmunsta: oh for real? that'd be surprising but also explain a lot [20:34:35] RECOVERY - MariaDB Replica IO: pc5 on pc2015 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:34:49] (03PS2) 10Bking: cirrussearch: Tune pool counter rejection alert [alerts] - 10https://gerrit.wikimedia.org/r/1360534 (https://phabricator.wikimedia.org/T442912) [20:35:07] oh man, if this is a "just did the switchover, started RT tests, pc host lost power" kind of page, I should get some lottery tickets [20:35:14] lol [20:35:15] RECOVERY - MariaDB Replica SQL: pc5 on pc2015 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:35:23] we started using yiannis' new k8s setup in early July 2026, have we have a datacenter switchover since then? [20:35:24] criminy. [20:35:31] RECOVERY - MariaDB Replica Lag: pc5 on pc2015 is OK: OK slave_sql_lag Replication lag: 0.28 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:35:33] cscott: nope nopity nope [20:35:36] good point, thank you [20:35:59] pc5 should have now cleared [20:36:03] also, parsercache hosts are co-masters, any write on each will end up in the other dc no matter what [20:36:20] ah, fair point, thanks Amir1 [20:36:52] (03Merged) 10jenkins-bot: Revert "listTaskCounts: Do not send negative task counts to stats" [extensions/GrowthExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360535 (owner: 10Ahmon Dancy) [20:37:15] so since we started using the new k8s rt test setup, we've been targetting only codfw but by chance codfw was the main(?) datacenter that worked fine. We did the first dc switch since the new k8s rt test setup and also started targetting eqiad, and things melted down? [20:37:17] !log dancy@deploy1003 Unlocked for deployment [MediaWiki]: Holding until database problems are resolved (duration: 24m 55s) [20:37:26] i'm not sure it all hangs together [20:37:34] cscott: codfw is the main DC as of yesterday [20:37:39] which means it's receiving more load [20:37:52] but I'm not sure how parsercache being co-masters fits into that [20:37:59] ottomata, krinkle: I'm stepping out of line. I reverted the change I was trying to deploy. /srv/mediawiki-staging is clean. [20:38:01] oh, so we *want* to be targetting the 'backup' DC so that our additional RT test load doesn't add to the main DC? maybe? [20:38:18] too many connections on pc2017 [20:38:18] but we were targetting eqiad so that still doesn't make sense [20:38:19] what is your load? Read or write [20:38:42] but, i am pretty confused ... we've been running rt-testing forever ... it is like 185K test pages that completes over 8 hours .. so how is that really a big load contributor? [20:39:01] yeah, the increment over our insane daily traffic should be tiny. [20:39:06] we do a wt2html, and then a html2wt .. and both are POST requests .. but still. [20:39:07] subbu: we have depooled eqiad, patrick stewart died [20:39:08] before k8s it was a baremetal host, same thing basically in terms of backend services (memc, db) [20:39:17] oncall here, quick checkin -- what's our current state, is pc fully recovered? that's a question for cezmunsta I think [20:39:18] oye! [20:39:24] patrick stewart died? (!) [20:39:42] we need to figure out what happened with the RT test, but in the meantime, if PC is healthy again we can unblock deployment [20:39:43] sorry you have to learn about it through wikimedia outage [20:39:45] i thought php2shell was bad enough [20:39:54] how's flaggedrevs doing? [20:40:07] lets probably take the rt-testing discussion out of this channel to not distract deployments for now. [20:40:22] subbu: thanks <3 let me know if you need anything [20:40:23] do I want to check page id of his article in dewiki against pc hosts that went down? [20:40:29] i don't think patrick stewart was on my list of flagged-revs-using-celebrities [20:40:38] Amir1: good call. [20:40:42] I checked enwiki, it wasn't there [20:41:03] I don't think we have any reason to think Patrick Stewart dying was connected to this outage so let's move that out of -ops for now too, unless there's information to suggest it was :) [20:41:38] Patrick Stewart died ?? [20:41:40] cscott: subbu: there might be something about the rt test that makes it PC miss more than previously, and so you end up populating the caches and incurring cross-dc writes. Just a hunch. [20:41:56] ottomata: you're up next I believe from before the backport window [20:42:02] ottomata: no. wait please [20:42:03] Krinkle: RT tests purge, so they always miss. [20:42:13] (but they have done so for years) [20:42:27] I'm still waiting on an all-clear from DBAs on pc* hosts before we unblock deployments [20:42:29] [20:42:43] rzl: pc5 is OK [20:42:48] i suggest #content-transform-team on slack for the rt testing stuff [20:42:48] cezmunsta: perf, thanks <3 [20:43:15] I can't connect on pc7 Amir: do you know what is up with the extra port? I get not allowed for local IP and denied for FQDN [20:43:24] I would appreciate it if anyone wants to weigh in on the RT tests question over on slack, since I have no idea why they would suddenly be a problem [20:43:48] yeah I see "pc2015 #p.age/MariaDB read only pc5 #p.age" still firing [20:44:09] (and for anyone wondering when you need to spell "p.aging" with a dot, that situation *with* the # sign is the only time) [20:44:29] RECOVERY - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [20:46:00] (03PS1) 10SBassett: Enable DBSC report-only mode on group0 and test2wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360542 (https://phabricator.wikimedia.org/T393212) [20:46:09] rzl: pc2015 read_only = 0 - does app code need to reconnect? [20:46:12] argh sorry, ignore that, I pasted the wrong one [20:46:23] I meant "pc2017 #p.age/MariaDB read only pc7 #p.age" is still firing [20:46:39] (03CR) 10SBassett: [C:04-2] "Hold for config deployment." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360542 (https://phabricator.wikimedia.org/T393212) (owner: 10SBassett) [20:47:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:47:38] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 03 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360542 (https://phabricator.wikimedia.org/T393212) (owner: 10SBassett) [20:47:57] usually it takes a bit for it to redo the check and call it fixed [20:48:09] (03CR) 10Ebernhardson: [C:03+2] cirrussearch: Tune pool counter rejection alert [alerts] - 10https://gerrit.wikimedia.org/r/1360534 (https://phabricator.wikimedia.org/T442912) (owner: 10Bking) [20:48:36] PROBLEM - MariaDB read only pc7 #page on pc2017 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [20:49:14] icinga-wm: yeah exactly, that one [20:50:04] it is overloaded still [20:50:17] I can't connect to it [20:50:24] (pc2017) [20:50:28] Amir will depooling pc7 get rid of the connections and free it? [20:50:40] during dolly parton, it didn't [20:50:55] usually it's better to kill it and let it recover [20:50:59] let me just quickly check [20:51:14] it's all derived data anyway [20:51:31] I will try depool just in case before kill [20:51:35] (03Merged) 10jenkins-bot: cirrussearch: Tune pool counter rejection alert [alerts] - 10https://gerrit.wikimedia.org/r/1360534 (https://phabricator.wikimedia.org/T442912) (owner: 10Bking) [20:51:37] the typical case I've seen in mysql overload events is that the master is spinning on one lock going through 10k requests serially and rollback'ing them because it's past their deadline [20:51:41] PROBLEM - MariaDB Replica IO: pc7 on pc2017 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:51:44] and yeah txns for read requests even [20:51:47] !log cwilliams@cumin1004 START - Cookbook sre.mysql.depool depool pc2017: Maintenance [20:51:48] !log cwilliams@cumin1004 START - Cookbook sre.mysql.parsercache [20:51:49] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [20:51:49] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool pc2017: Maintenance [20:51:56] sounds good [20:52:01] so +1 from me on just killing daemon and restarting [20:52:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:52:10] the latency will go up a bit [20:52:15] ack [20:52:18] they are dropping off [20:52:20] but appservers are healthy [20:52:35] RECOVERY - MariaDB Replica Lag: pc7 on pc2017 is OK: OK slave_sql_lag not a slave https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:52:37] RECOVERY - MariaDB Replica SQL: pc7 on pc2017 is OK: OK slave_sql_state not a slave https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:52:39] and I am in [20:52:39] RECOVERY - MariaDB Replica IO: pc7 on pc2017 is OK: OK slave_io_state not a slave https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [20:52:41] yeah we can eat a little latency right now, if we were near saturation I'd be more worried [20:52:55] cezmunsta: nice job [20:53:50] I'm pushing a restart of the systemd service [20:54:08] 06SRE, 10SRE-Access-Requests: Add yubikey backed SSH keys for joal - https://phabricator.wikimedia.org/T442587#12492345 (10JAllemandou) 05Open→03Declined Closing as this is handled through {T442758} [20:54:21] damn it, that was fault [20:54:30] it should finish soon [20:54:35] PROBLEM - MariaDB Event Scheduler pc7 on pc2017 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [20:54:36] PROBLEM - MariaDB read only pc7 #page on pc2017 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [20:54:45] that's me, give me a second [20:55:00] (I hope it's not paging. I saw a downtime above) [20:55:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [20:55:27] !log cwilliams@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2017.codfw.wmnet with reason: Alerting [20:55:29] Amir1: same alert that was already paging, you're fine [20:55:44] ah thanks [20:55:46] I'd almost rather keep it undowntimed so we know what's up with it :) but either way [20:56:04] let me check, maybe it's not properly depooled [20:56:07] like the last time [20:56:30] oh if we're keeping it depooled then definitely no problem downtiming [20:56:35] okay, it's depooled [20:57:12] rzl: it was because I was going to kill it... but Amir got there first [20:57:34] Amir1: is it stuck? [20:57:51] yeah, systemd restart is too nice [20:57:54] kill -9? [20:58:17] go ahead [20:58:18] in that case Amir1 cezmunsta any objection to resuming normal deployments? even with it depooled, happy to hold off while you keep working, but let us know [20:58:59] I missed what the likely cause was... what was it? [20:59:28] we think a parsoid test job, but we still don't know why it was more impactful today than usual -- possibly switchover related [20:59:43] ack [20:59:48] we won't resume that until we know more, just thinking about the backport window for unrelated stuff [21:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261029T2100) [21:00:15] RESOLVED: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [21:00:26] (continue to hold off on deployments until my all clear, please) [21:00:27] is anyone doing a scap deploy right now? [21:00:31] oh no I will wait [21:00:33] thanks :) [21:00:45] did a kill -9 [21:00:54] now starting the systemd [21:01:56] it's back up now, looks clean [21:02:15] I have started replication [21:02:35] RECOVERY - MariaDB Event Scheduler pc7 on pc2017 is OK: Version 10.11.16-MariaDB-log, Uptime 79s, read_only: False, event_scheduler: True, 16.83 QPS, connection latency: 0.025226s, query latency: 0.001289s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [21:02:38] RECOVERY - MariaDB read only pc7 #page on pc2017 is OK: Version 10.11.16-MariaDB-log, Uptime 81s, read_only: False, event_scheduler: True, 16.91 QPS, connection latency: 0.017882s, query latency: 0.001095s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [21:02:38] awesome [21:02:39] orchestrator looks happy [21:02:48] that means I am happy [21:02:52] sweet [21:02:59] i realize the db hosts being operated on right now are in codfw; last night, a couple of eqiad db hosts were depooled because of problems (db1251, 1228). although unrelated, does that provide any useful context to what might have happened? [21:03:01] 😌 [21:03:08] looking at the timing, it looks like pc2017 started complaining about 4 minutes *before* I kicked off the RT test job. (The first logged helmfile deploy failed to start jobs due to a git misconfiguration and it took me a few minutes to fix that. 2008 is when the first parse actually happened.) [21:03:12] it's RW [21:03:33] bliviero: they crashed, these seem to be connection related [21:03:58] i was just thinking "fewer available overall" > "more load on what's up" [21:04:16] gigantic amount of deadlocks: [21:04:23] https://www.irccloud.com/pastebin/NDBjJwzL/ [21:04:35] bliviero: no, different DB clusters -- good call though [21:04:37] those are old(ish) though [21:04:54] there were >2.5k connections on pc2017 [21:06:20] ah right, thanks for catching that [21:06:42] I could see a bunch of queries when I gained access, they all had the same query hint [21:07:11] subbu and I almost simultaneously kicked off RT testing (20:08:54) and visual diff testing (20:09:53) and the first pc2017 page was ~20:04. So I think I made things worse but i wasn't the one originally responsible... unless pool counter is involved still, but that's a different hat :) [21:07:59] (I pinged some folks on that slack thread for fuller context) [21:08:51] Amir: REPLACE /* MediaWiki\ObjectCache\SqlBagOStuff::modifyTableSpecificBlobsForSet */ INTO `pc207` -- all seemed to be that, any clues from it? [21:09:13] that's the generic part, do you have the full query? [21:09:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12492377 (10phaultfinder) [21:10:18] not, they dropped off before I could look again, let me see if sys has any info [21:10:55] https://www.irccloud.com/pastebin/XXnYZSYW/ [21:11:15] these are the pods that were holding a connection to pc2015 during that outage [21:11:38] used dig -x on the processlist basically [21:11:45] (also who uses + for argument) [21:12:05] hm, yeah and that further exonerates mw-parsoid [21:12:51] https://www.irccloud.com/pastebin/WNT0Q90I/ [21:14:20] that looks like maybe the FR bug is not fixed? [21:15:12] it's way too late for me to check that, and I spent half of today reading libvpx code so my brain refuses to take on more [21:16:03] FR=FlaggedRevs? FR is only dewiki though, that's like 10x lower traffic than enwiki. [21:16:34] yeah, or maybe star trek [21:18:24] hm, mw-api-ext is pretty high there. there is one unplugged hole in poolcounter for flagged revs, but you have to be using the legacy parser and have the postprocessing cache disabled. I had a patch for it but it seemed low priority at the time. It is possible that API traffic for a flagged revs page is exercising the legacy parser enough to cause a problem? [21:19:21] (03PS1) 10Eric Gardner: [WIP] Add Creative Commons license icons for the share card [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1358851 (https://phabricator.wikimedia.org/T439162) [21:19:26] that would surprise me? but I've been surprised before [21:19:48] I don't suppose we have any metrics that would confirm [21:20:06] never mind, that was [ParserOutputAccess: Use the parser cache for stale fallback output (1354187) · Gerrit Code Review](https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1354187) and it's already merged and deployed. [21:20:42] FIRING: [2x] GanetiBGPNoInPrefixes: ganeti2034 is not sending any prefix to lsw1-a4-codfw - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPNoInPrefixes - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPNoInPrefixes [21:22:25] 10ops-eqiad, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12492420 (10wiki_willy) a:03VRiley-WMF Adding ops-eqiad tag and assigning to @VRiley-WMF (cc'ing @Jclark-ctr as well) [21:24:09] Amir I can only seem to find digests and of course the exporter couldn't connect to capture any metrics.. nothing looked too off before the blankety-blank section [21:24:30] rzl: would you prefer it to be repooled or left depooled? [21:24:45] you're the expert :) what's your instinct? [21:25:13] i'd love to get even one page title or cache key out of the logs to rule in/rule things. [21:25:21] *rule out things [21:25:28] (03PS1) 10BCornwall: Revert "mx: use production, rather than aggregate network list" [puppet] - 10https://gerrit.wikimedia.org/r/1360552 [21:26:17] based upon what I can see from pc2015, which seemed to have contention on InnoDB rrow locks [21:26:42] before hand, but seems OK now, I will repool pc7 [21:26:58] sgtm [21:27:38] thank you for the late-night DBAing :( [21:27:58] !log cwilliams@cumin1004 START - Cookbook sre.mysql.pool pool pc2017: Maintenance [21:27:58] !log cwilliams@cumin1004 START - Cookbook sre.mysql.parsercache [21:28:05] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.mysql.parsercache (exit_code=0) [21:28:05] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool pc2017: Maintenance [21:30:16] (03CR) 10BCornwall: [C:03+2] Revert "mx: use production, rather than aggregate network list" [puppet] - 10https://gerrit.wikimedia.org/r/1360552 (owner: 10BCornwall) [21:31:22] Unless we hear otherwise, we'll resume rt-testing / visualdiff testing tomorrow. [21:31:51] (03PS3) 10CWilliams: zarcillo: enabling egress to config-master [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360470 (https://phabricator.wikimedia.org/T442904) [21:32:19] subbu: sounds good to me. sorry for the fire drill, still not sure what the story was here [21:32:33] rzl: are we good to resume deployments now? [21:32:39] but agree with cscott that whatever it was, the tests might have exacerbated it but didn't cause it [21:32:55] maryum: nearly, I want to watch a little longer after that pc7 repool and make sure we're still in good shape [21:33:03] ack, thanks. [21:33:11] rzl: sounds good to me [21:35:40] cezmunsta: I think I'm happy, you? [21:36:09] rzl: connections seem to be staying much lower, so perhaps as you say something exacerbated things and then made unhappy sealions - yes it seems like it is stable [21:37:02] okay, cool [21:37:21] cezmunsta: thanks for being around <3 really appreciate it, and I'm sure m.arostegui will be glad I didn't call him [21:37:45] (03CR) 10Bking: [C:03+1] Revert "mx: use production, rather than aggregate network list" [puppet] - 10https://gerrit.wikimedia.org/r/1360552 (owner: 10BCornwall) [21:38:19] we can resume deployments -- maryum I know ottomata was also waiting, and we didn't get to any of the deployments in the window, either, so unhelpfully I'm going to let you sort out who goes first [21:38:33] ottomata: are you around? [21:38:44] I was planning to do a sync-world [21:39:18] I'm going to go ahead and prepare unless I hear something... [21:44:05] 10SRE-Access-Requests: Add FIDO-backed SSH keys for snowick - https://phabricator.wikimedia.org/T442951 (10SNowick_WMF) 03NEW [21:44:19] if possible, I'd still like to deploy when you all are done [21:44:53] rzl: yw ... now back to fixing my ZFS box :D ... in case needed, it was just "sudo cookbook sre.mysql.depool -r Maintenance pc2017" , which should cover the pair .. also, my tmux session on pc2017 has root connected via socket , at least for now [21:45:18] (03CR) 10CWilliams: [C:03+1] zarcillo: enabling egress to config-master [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360470 (https://phabricator.wikimedia.org/T442904) (owner: 10CWilliams) [21:45:37] (03PS1) 10SBassett: Bump security landing page image version to 2026-10-08-214827 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360557 [21:45:53] cezmunsta: ack, thanks! have a good evening [21:46:16] thanks y'all <3 [21:46:34] cezmunsta: ah, a fellow zfs enthusiast! I just hope you're running the correct OS with it [21:50:51] arlolra: I will let you know when I wrap up [21:51:05] thanks [21:54:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12492614 (10phaultfinder) [21:58:17] (03PS1) 10BCornwall: mx: use production+frack rather than aggregate net [puppet] - 10https://gerrit.wikimedia.org/r/1360559 (https://phabricator.wikimedia.org/T442063) [21:59:49] (03CR) 10BCornwall: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9577/co" [puppet] - 10https://gerrit.wikimedia.org/r/1359145 (https://phabricator.wikimedia.org/T442497) (owner: 10Ayounsi) [22:01:01] (03CR) 10CI reject: [V:04-1] mx: use production+frack rather than aggregate net [puppet] - 10https://gerrit.wikimedia.org/r/1360559 (https://phabricator.wikimedia.org/T442063) (owner: 10BCornwall) [22:01:47] I, too, would like to get a config deploy out if that's still doable today. [22:10:22] scap is still running, hopefully done soon [22:16:35] Quite a slow one. 🙀 [22:18:55] (03PS2) 10BCornwall: mx: use production+frack rather than aggregate net [puppet] - 10https://gerrit.wikimedia.org/r/1360559 (https://phabricator.wikimedia.org/T442063) [22:19:56] (03CR) 10BCornwall: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9578/co" [puppet] - 10https://gerrit.wikimedia.org/r/1359145 (https://phabricator.wikimedia.org/T442497) (owner: 10Ayounsi) [22:21:38] (03CR) 10CI reject: [V:04-1] mx: use production+frack rather than aggregate net [puppet] - 10https://gerrit.wikimedia.org/r/1360559 (https://phabricator.wikimedia.org/T442063) (owner: 10BCornwall) [22:24:46] (03PS3) 10BCornwall: mx: use production+frack rather than aggregate net [puppet] - 10https://gerrit.wikimedia.org/r/1360559 (https://phabricator.wikimedia.org/T442063) [22:25:01] (03PS4) 10BCornwall: mx: use production+frack rather than aggregate net [puppet] - 10https://gerrit.wikimedia.org/r/1360559 (https://phabricator.wikimedia.org/T442063) [22:27:17] !log Deployed security fix for T440122, T439876, T442478, and T442693 [22:27:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:27:32] Kemayo: the floor is yours [22:28:02] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360524 (https://phabricator.wikimedia.org/T442933) (owner: 10DLynch) [22:28:07] maryum: Thanks! [22:28:50] I guess I'll go next [22:28:58] (03Merged) 10jenkins-bot: Remove wgVisualEditorDisableForAnons for enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360524 (https://phabricator.wikimedia.org/T442933) (owner: 10DLynch) [22:29:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12492733 (10phaultfinder) [22:30:31] Ah, we're not in a deployable state. [22:30:42] `Aborting: git is not clean: /srv/patches` [22:31:33] maryum: is this likely to be a side-effect of your just-finished stuff? [22:31:51] kemayo let me take a look [22:32:15] weird I thought I added everything [22:32:19] fixing now [22:33:03] kemayo: sorted now, apologies [22:33:11] maryum: thanks! [22:33:27] arlolra: want me to bundle yours into the same scap, since you're back? [22:33:49] sure [22:34:13] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy1003 using scap backport" [extensions/ParserMigration] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360430 (https://phabricator.wikimedia.org/T430970) (owner: 10Arlolra) [22:34:24] (03CR) 10BCornwall: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9579/co" [puppet] - 10https://gerrit.wikimedia.org/r/1359145 (https://phabricator.wikimedia.org/T442497) (owner: 10Ayounsi) [22:36:22] (03Merged) 10jenkins-bot: Continue to serve config for cached js [extensions/ParserMigration] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360430 (https://phabricator.wikimedia.org/T430970) (owner: 10Arlolra) [22:36:39] !log kemayo@deploy1003 Started scap sync-world: Backport for [[gerrit:1360524|Remove wgVisualEditorDisableForAnons for enwiki (T442933)]], [[gerrit:1360430|Continue to serve config for cached js (T430970)]] (1.48.0-wmf.1) [22:36:47] T442933: When logged out action=edit URLs on enwiki flash WikiEditor before VisualEditor overtakes it - https://phabricator.wikimedia.org/T442933 [22:36:48] T430970: Turn-off "Report a Visual Bug" functionality on all Wikipedias at the end of Q1 26-27 - https://phabricator.wikimedia.org/T430970 [22:48:55] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for isaranto - https://phabricator.wikimedia.org/T442176#12492887 (10Dzahn) 05Stalled→03In progress Your new key has been deployed. Please give it about 30 min from now max until it has been added to all relevant hosts by puppet. Then feel free to veri... [22:49:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12492889 (10phaultfinder) [22:51:31] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for catrope - https://phabricator.wikimedia.org/T442247#12492893 (10Dzahn) 05Open→03Resolved a:03Dzahn [22:58:19] !log kemayo@deploy1003 kemayo, arlolra: Backport for [[gerrit:1360524|Remove wgVisualEditorDisableForAnons for enwiki (T442933)]], [[gerrit:1360430|Continue to serve config for cached js (T430970)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:58:26] T442933: When logged out action=edit URLs on enwiki flash WikiEditor before VisualEditor overtakes it - https://phabricator.wikimedia.org/T442933 [22:58:27] T430970: Turn-off "Report a Visual Bug" functionality on all Wikipedias at the end of Q1 26-27 - https://phabricator.wikimedia.org/T430970 [22:58:55] arlolra: Anything for you to test? [22:59:06] I will take a quick look [23:00:43] Kemayo: lgtm [23:00:51] Great! [23:00:56] !log kemayo@deploy1003 kemayo, arlolra: Continuing with deployment [23:13:27] !log kemayo@deploy1003 Finished scap sync-world: Backport for [[gerrit:1360524|Remove wgVisualEditorDisableForAnons for enwiki (T442933)]], [[gerrit:1360430|Continue to serve config for cached js (T430970)]] (1.48.0-wmf.1) (duration: 36m 48s) [23:13:35] T442933: When logged out action=edit URLs on enwiki flash WikiEditor before VisualEditor overtakes it - https://phabricator.wikimedia.org/T442933 [23:13:35] T430970: Turn-off "Report a Visual Bug" functionality on all Wikipedias at the end of Q1 26-27 - https://phabricator.wikimedia.org/T430970 [23:13:50] thanks Kemayo [23:13:53] np [23:34:25] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:36:19] (03PS1) 10Southparkfan: [BETA HACK] Stop managing swift.yaml from Puppet [puppet] - 10https://gerrit.wikimedia.org/r/1360569 (https://phabricator.wikimedia.org/T436476) [23:37:25] (03CR) 10CI reject: [V:04-1] [BETA HACK] Stop managing swift.yaml from Puppet [puppet] - 10https://gerrit.wikimedia.org/r/1360569 (https://phabricator.wikimedia.org/T436476) (owner: 10Southparkfan) [23:37:44] (03PS2) 10Southparkfan: [BETA HACK] Stop managing swift.yaml from Puppet [puppet] - 10https://gerrit.wikimedia.org/r/1360569 (https://phabricator.wikimedia.org/T436476) [23:40:41] (03CR) 10Dzahn: [V:03+1] "Hi, could you replace this key with a yubikey based key? Do you have one?" [puppet] - 10https://gerrit.wikimedia.org/r/1358308 (https://phabricator.wikimedia.org/T442183) (owner: 10Dzahn) [23:47:25] !log Running `foreachwikiindblist group2.dblist cleanupWatchlistTemporaryAccountRows.php` for T439059 [23:47:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:47:31] T439059: Skip default watchlist additions during temp account creation - https://phabricator.wikimedia.org/T439059 [23:49:16] (03PS1) 10Dzahn: gitlab_runner: partially revert "Use ferm::join_hosts to resolve host names" [puppet] - 10https://gerrit.wikimedia.org/r/1360571 [23:50:23] (03CR) 10CI reject: [V:04-1] gitlab_runner: partially revert "Use ferm::join_hosts to resolve host names" [puppet] - 10https://gerrit.wikimedia.org/r/1360571 (owner: 10Dzahn) [23:51:05] (03PS2) 10Dzahn: gitlab_runner: partially revert "Use ferm::join_hosts to resolve host names" [puppet] - 10https://gerrit.wikimedia.org/r/1360571 (https://phabricator.wikimedia.org/T439046) [23:52:12] (03CR) 10CI reject: [V:04-1] gitlab_runner: partially revert "Use ferm::join_hosts to resolve host names" [puppet] - 10https://gerrit.wikimedia.org/r/1360571 (https://phabricator.wikimedia.org/T439046) (owner: 10Dzahn) [23:53:18] (03PS3) 10Dzahn: gitlab_runner: partially revert "Use ferm::join_hosts to resolve host names" [puppet] - 10https://gerrit.wikimedia.org/r/1360571 (https://phabricator.wikimedia.org/T439046) [23:56:00] (03CR) 10Dzahn: "for the moment just going back to how it was until recently" [puppet] - 10https://gerrit.wikimedia.org/r/1360571 (https://phabricator.wikimedia.org/T439046) (owner: 10Dzahn) [23:56:23] (03CR) 10Dzahn: [C:03+2] gitlab_runner: partially revert "Use ferm::join_hosts to resolve host names" [puppet] - 10https://gerrit.wikimedia.org/r/1360571 (https://phabricator.wikimedia.org/T439046) (owner: 10Dzahn)