[00:00:04] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [00:00:14] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [00:00:19] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [00:00:26] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [00:04:45] RECOVERY - Host db1217 is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms [00:05:45] PROBLEM - MariaDB Replica IO: m2 on db1217 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:05:47] PROBLEM - MariaDB Replica SQL: m5 on db1217 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:05:47] PROBLEM - MariaDB read only m5 on db1217 is CRITICAL: Could not connect to localhost:3325 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [00:05:49] PROBLEM - MariaDB Replica SQL: m3 on db1217 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:05:49] PROBLEM - MariaDB Replica IO: m1 on db1217 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:05:49] PROBLEM - MariaDB read only m3 on db1217 is CRITICAL: Could not connect to localhost:3323 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [00:06:41] PROBLEM - MariaDB Replica SQL: m1 on db1217 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:06:41] PROBLEM - MariaDB Replica IO: m5 on db1217 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:06:41] PROBLEM - mysqld processes on db1217 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [00:06:41] PROBLEM - MariaDB read only m1 on db1217 is CRITICAL: Could not connect to localhost:3321 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [00:06:43] PROBLEM - MariaDB Replica IO: m3 on db1217 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:07:37] PROBLEM - MariaDB Replica SQL: m2 on db1217 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:07:37] PROBLEM - MariaDB read only m2 on db1217 is CRITICAL: Could not connect to localhost:3322 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [00:08:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:09:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12499968 (10phaultfinder) [00:12:47] PROBLEM - MariaDB Replica Lag: m3 on db1217 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:12:49] PROBLEM - MariaDB Replica Lag: m2 on db1217 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:13:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:13:43] PROBLEM - MariaDB Replica Lag: m5 on db1217 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:14:09] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.21 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:14:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:14:37] PROBLEM - MariaDB Replica Lag: m1 on db1217 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:19:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:24:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:26:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [00:27:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:30:00] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:31:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [00:32:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:32:31] FIRING: Traffic bill over quota: Alert for device cr1-magru.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [00:37:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:39:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12499973 (10phaultfinder) [00:40:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:42:12] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1361227 [00:42:12] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1361227 (owner: 10TrainBranchBot) [00:48:38] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1361227 (owner: 10TrainBranchBot) [00:50:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:52:31] RESOLVED: Traffic bill over quota: Alert for device cr1-magru.wikimedia.org - Traffic bill over quota - https://alerts.wikimedia.org/?q=alertname%3DTraffic+bill+over+quota [00:53:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [00:56:00] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:09:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:12:15] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1361228 [01:12:15] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1361228 (owner: 10TrainBranchBot) [01:14:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:19:06] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1361228 (owner: 10TrainBranchBot) [01:29:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:30:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [01:34:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:34:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12499980 (10phaultfinder) [01:38:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:43:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [01:50:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [01:50:23] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [01:54:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12499982 (10phaultfinder) [01:54:54] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12499983 (10phaultfinder) [02:00:05] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261101T0700) [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T0200) [02:00:22] !log mwpresync@deploy2003 Started scap build-images: Publishing wmf/next image [02:08:29] !log mwpresync@deploy2003 Finished scap build-images: Publishing wmf/next image (duration: 08m 07s) [02:09:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12500001 (10phaultfinder) [02:11:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:29:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12500011 (10phaultfinder) [02:31:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:36:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:44:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:49:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:50:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [02:55:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [03:00:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [03:04:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12500029 (10phaultfinder) [03:05:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [03:10:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [03:13:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [03:15:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [03:18:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [03:23:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [03:25:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [03:29:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12500062 (10phaultfinder) [03:31:05] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [03:34:26] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:00:02] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 02 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360437 (https://phabricator.wikimedia.org/T438016) (owner: 10Neriah) [04:49:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12500087 (10phaultfinder) [05:47:46] !log ssastry@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [05:48:27] !log ssastry@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [05:49:48] !log ssastry@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [05:49:51] !log ssastry@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [05:50:23] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:59:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12500118 (10phaultfinder) [06:39:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12500138 (10phaultfinder) [06:40:50] 10SRE-Access-Requests: Add yubikey (hardware) ssh key for kartik - https://phabricator.wikimedia.org/T443182 (10KartikMistry) 03NEW [06:41:21] 10SRE-Access-Requests: Add FIDO-backed SSH keys for kartik - https://phabricator.wikimedia.org/T443182#12500149 (10KartikMistry) [06:53:30] 10ops-eqiad, 06DC-Ops: Unresponsive management for an-worker1187.mgmt:22 - https://phabricator.wikimedia.org/T443184 (10phaultfinder) 03NEW [07:21:41] jouncebot: nowandnext [07:21:42] No deployments scheduled for the next 0 hour(s) and 38 minute(s) [07:21:42] In 0 hour(s) and 38 minute(s): UTC morning backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T0800) [07:23:47] !log arthurtaylor@deploy2003 helmfile [staging] START helmfile.d/services/termbox: apply [07:23:57] !log arthurtaylor@deploy2003 helmfile [staging] DONE helmfile.d/services/termbox: apply [07:24:03] !log arthurtaylor@deploy2003 helmfile [codfw] START helmfile.d/services/termbox: apply [07:24:06] !log arthurtaylor@deploy2003 helmfile [codfw] DONE helmfile.d/services/termbox: apply [07:24:12] !log arthurtaylor@deploy2003 helmfile [eqiad] START helmfile.d/services/termbox: apply [07:24:17] !log arthurtaylor@deploy2003 helmfile [eqiad] DONE helmfile.d/services/termbox: apply [07:24:31] (03CR) 10Arthur taylor: [C:03+2] "deploying" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360854 (https://phabricator.wikimedia.org/T442990) (owner: 10Arthur taylor) [07:24:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12500202 (10phaultfinder) [07:26:12] (03Merged) 10jenkins-bot: termbox: bump to latest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360854 (https://phabricator.wikimedia.org/T442990) (owner: 10Arthur taylor) [07:27:06] !log arthurtaylor@deploy2003 helmfile [staging] START helmfile.d/services/termbox: apply [07:27:50] !log arthurtaylor@deploy2003 helmfile [staging] DONE helmfile.d/services/termbox: apply [07:27:58] !log arthurtaylor@deploy2003 helmfile [eqiad] START helmfile.d/services/termbox: apply [07:28:53] !log arthurtaylor@deploy2003 helmfile [eqiad] DONE helmfile.d/services/termbox: apply [07:29:13] !log arthurtaylor@deploy2003 helmfile [codfw] START helmfile.d/services/termbox: apply [07:29:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12500207 (10phaultfinder) [07:29:52] !log arthurtaylor@deploy2003 helmfile [codfw] DONE helmfile.d/services/termbox: apply [07:31:05] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [07:33:29] PROBLEM - Thanos swift https on thanos-fe2005 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 260 bytes in 1.181 second response time https://wikitech.wikimedia.org/wiki/Thanos [07:34:26] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:36:27] RECOVERY - Thanos swift https on thanos-fe2005 is OK: HTTP OK: HTTP/1.1 200 OK - 279 bytes in 0.180 second response time https://wikitech.wikimedia.org/wiki/Thanos [07:42:51] (03PS1) 10Muehlenhoff: Remove access for elukey [puppet] - 10https://gerrit.wikimedia.org/r/1361316 [07:44:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [07:50:23] (03CR) 10Muehlenhoff: [C:03+2] Remove access for elukey [puppet] - 10https://gerrit.wikimedia.org/r/1361316 (owner: 10Muehlenhoff) [07:54:26] (03PS1) 10Muehlenhoff: Remove router access for Luca [homer/public] - 10https://gerrit.wikimedia.org/r/1361317 (https://phabricator.wikimedia.org/T443187) [07:54:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12500262 (10phaultfinder) [07:58:08] !log jmm@cumin2003 DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Elukey out of all services on: 2438 hosts [08:00:05] Amir1, urbanecm, and awight: Time to do the UTC morning backport window deploy. Don't look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T0800). [08:00:05] WMDE-Fisch and codders: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [08:00:11] o/ [08:00:52] (03PS3) 10Hashar: rubocop: enable Lint/UnneededDisable [puppet] - 10https://gerrit.wikimedia.org/r/1348795 (https://phabricator.wikimedia.org/T441847) [08:02:06] (03PS5) 10Hashar: rubocop: upgrade and target ruby 2.7 [puppet] - 10https://gerrit.wikimedia.org/r/1348797 (https://phabricator.wikimedia.org/T441847) [08:02:21] (03PS4) 10Hashar: ipmi: ruby: stop using then for multiline when statement [puppet] - 10https://gerrit.wikimedia.org/r/1348873 (https://phabricator.wikimedia.org/T441847) [08:04:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [08:04:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12500266 (10phaultfinder) [08:06:03] the loneliest deployment window. [08:06:15] I have a couple of patches that should only affect beta. shall I go ahead? [08:06:25] (beta wikidata, that is) [08:08:15] morning WMDE-Fisch ! [08:08:22] you want to go first? [08:08:27] sorry I'm a bit late [08:08:32] yes give me a sec [08:10:54] (03CR) 10TrainBranchBot: [C:03+2] "Approved by wmde-fisch@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359782 (https://phabricator.wikimedia.org/T440304) (owner: 10WMDE-Fisch) [08:11:15] (03CR) 10Brouberol: [C:03+1] "LG! Thanks" [puppet] - 10https://gerrit.wikimedia.org/r/1359236 (owner: 10Meno25) [08:11:16] (03CR) 10Brouberol: [C:03+2] Update redirected links [puppet] - 10https://gerrit.wikimedia.org/r/1359236 (owner: 10Meno25) [08:11:37] (03PS1) 10Muehlenhoff: Remove LDAP access for cdiggs-ctr [puppet] - 10https://gerrit.wikimedia.org/r/1361320 [08:11:43] (03Merged) 10jenkins-bot: Enable sub-references globally [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359782 (https://phabricator.wikimedia.org/T440304) (owner: 10WMDE-Fisch) [08:12:26] (03CR) 10Brouberol: [C:03+1] kafka-jumbo: sync wdqs partition counts [puppet] - 10https://gerrit.wikimedia.org/r/1360786 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [08:12:27] !log wmde-fisch@deploy2003 Started scap sync-world: Backport for [[gerrit:1359782|Enable sub-references globally (T440304)]] [08:12:30] (03CR) 10Brouberol: [C:03+2] kafka-jumbo: sync wdqs partition counts [puppet] - 10https://gerrit.wikimedia.org/r/1360786 (https://phabricator.wikimedia.org/T435284) (owner: 10Ryan Kemper) [08:12:33] T440304: 0️⃣ Global rollout phase 2 – Sub-ref deployments to remaining WMF wikis - https://phabricator.wikimedia.org/T440304 [08:18:00] (03PS1) 10Filippo Giunchedi: wmcs: add cloudvps root FIDO keys for filippo [puppet] - 10https://gerrit.wikimedia.org/r/1361321 (https://phabricator.wikimedia.org/T443002) [08:18:24] (03CR) 10Muehlenhoff: [C:03+2] Remove LDAP access for cdiggs-ctr [puppet] - 10https://gerrit.wikimedia.org/r/1361320 (owner: 10Muehlenhoff) [08:18:25] did it always take so long to build these images? [08:19:34] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [08:21:06] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [08:21:14] (03PS1) 10Slyngshede: data.yaml Offboarding elliottetzkorn [puppet] - 10https://gerrit.wikimedia.org/r/1361323 [08:22:20] (03CR) 10CI reject: [V:04-1] data.yaml Offboarding elliottetzkorn [puppet] - 10https://gerrit.wikimedia.org/r/1361323 (owner: 10Slyngshede) [08:25:26] codders: Monday is always special, it takes longer. [08:25:38] (03PS2) 10Slyngshede: data.yaml Offboarding elliottetzkorn [puppet] - 10https://gerrit.wikimedia.org/r/1361323 [08:25:49] fun. because of all the patches? will my build(s) also take so long? [08:26:08] No it's just the first time afaik [08:26:26] good to know! [08:27:47] (03PS1) 10Brouberol: global_config: open overlord port to apps running in kubernetes [puppet] - 10https://gerrit.wikimedia.org/r/1361324 [08:29:10] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1361323 (owner: 10Slyngshede) [08:31:03] (03CR) 10Slyngshede: [C:03+2] data.yaml Offboarding elliottetzkorn [puppet] - 10https://gerrit.wikimedia.org/r/1361323 (owner: 10Slyngshede) [08:32:00] (03CR) 10Atsuko: [C:03+1] global_config: open overlord port to apps running in kubernetes [puppet] - 10https://gerrit.wikimedia.org/r/1361324 (owner: 10Brouberol) [08:33:25] !log wmde-fisch@deploy2003 wmde-fisch: Backport for [[gerrit:1359782|Enable sub-references globally (T440304)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [08:33:30] T440304: 0️⃣ Global rollout phase 2 – Sub-ref deployments to remaining WMF wikis - https://phabricator.wikimedia.org/T440304 [08:34:37] (03PS2) 10Brouberol: global_config: open overlord port to apps running in kubernetes [puppet] - 10https://gerrit.wikimedia.org/r/1361324 [08:35:08] (03CR) 10Atsuko: [C:03+1] global_config: open overlord port to apps running in kubernetes [puppet] - 10https://gerrit.wikimedia.org/r/1361324 (owner: 10Brouberol) [08:36:07] !log wmde-fisch@deploy2003 wmde-fisch: Continuing with deployment [08:37:26] (03CR) 10Brouberol: [C:03+2] global_config: open overlord port to apps running in kubernetes [puppet] - 10https://gerrit.wikimedia.org/r/1361324 (owner: 10Brouberol) [08:39:40] 10ops-eqiad, 06SRE, 06cloud-services-team, 10Cloud-VPS, and 2 others: cloudvirt1071 crash, again - https://phabricator.wikimedia.org/T431374#12500307 (10fgiunchedi) I have put the host back in service now, we'll see if it crashes again [08:40:06] (03PS1) 10Brouberol: global_config: open druid-public ports to apps running in kubernetes [puppet] - 10https://gerrit.wikimedia.org/r/1361327 [08:41:15] (03CR) 10Atsuko: [C:03+1] global_config: open druid-public ports to apps running in kubernetes [puppet] - 10https://gerrit.wikimedia.org/r/1361327 (owner: 10Brouberol) [08:41:39] (03CR) 10Brouberol: [C:03+2] global_config: open druid-public ports to apps running in kubernetes [puppet] - 10https://gerrit.wikimedia.org/r/1361327 (owner: 10Brouberol) [08:41:46] (03CR) 10Ayounsi: [C:03+2] Remove router access for Luca [homer/public] - 10https://gerrit.wikimedia.org/r/1361317 (https://phabricator.wikimedia.org/T443187) (owner: 10Muehlenhoff) [08:42:58] (03Merged) 10jenkins-bot: Remove router access for Luca [homer/public] - 10https://gerrit.wikimedia.org/r/1361317 (https://phabricator.wikimedia.org/T443187) (owner: 10Muehlenhoff) [08:44:44] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12500315 (10phaultfinder) [08:44:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12500314 (10phaultfinder) [08:45:12] 10SRE-swift-storage, 10MediaWiki-File-management, 06MediaWiki-Media-Platform-Team: Unknown error while undeleting a file on Wikimedia Commons - https://phabricator.wikimedia.org/T437114#12500318 (10MatthewVernon) @Sreejithk2000 that file appears visible to me...? [08:49:08] !log wmde-fisch@deploy2003 Finished scap sync-world: Backport for [[gerrit:1359782|Enable sub-references globally (T440304)]] (duration: 36m 41s) [08:49:14] T440304: 0️⃣ Global rollout phase 2 – Sub-ref deployments to remaining WMF wikis - https://phabricator.wikimedia.org/T440304 [08:49:14] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [08:49:37] okay. am I good to go? [08:49:41] codders: yes [08:49:50] You can do both at once [08:50:13] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [08:50:20] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for mgerlach - https://phabricator.wikimedia.org/T443038#12500358 (10MGerlach) Thank you @Dzahn . @andrea.denisse : confirmed that ssh access works. you can remove the old key. Thank you. [08:50:59] !log brouberol@deploy2003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [08:51:14] WMDE-Fisch: it's warning me that my patch is not on 1.47.0-wmf.24, but 1.48.0 is everywhere [08:51:18] do you think I need to care? [08:51:31] !log brouberol@deploy2003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [08:51:36] hmm [08:51:48] codders: I would say no [08:52:09] funny that I hadn't had that warning [08:52:12] the warning just says to consider making a backport to that branch [08:52:16] (03PS2) 10Urbanecm: zookeeper: Add slf4j and log4j jars to the CLASSPATH on bullseye [puppet] - 10https://gerrit.wikimedia.org/r/1360972 (https://phabricator.wikimedia.org/T436481) [08:52:20] (03PS3) 10Urbanecm: zookeeper: Add slf4j and log4j jars to the CLASSPATH on bullseye [puppet] - 10https://gerrit.wikimedia.org/r/1360972 (https://phabricator.wikimedia.org/T436481) [08:52:22] yeah - it's because my patch has a depends-on on a patch in 24 [08:52:44] but I think this doesn't affect rollback-safety [08:52:47] so I'll proceed [08:53:24] (03CR) 10TrainBranchBot: [C:03+2] "Approved by arthurtaylor@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355410 (https://phabricator.wikimedia.org/T440832) (owner: 10Mahmoud-abdelsattar) [08:53:56] codders: you could check with Wikibase what that patch is [08:54:12] (03Merged) 10jenkins-bot: Turn on Wikibase hCaptcha flag for Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355410 (https://phabricator.wikimedia.org/T440832) (owner: 10Mahmoud-abdelsattar) [08:54:15] I am wikibase :) [08:54:16] Maybe they forgot to backport it to the newest branch [08:54:25] Ah right :-D [08:54:34] !log arthurtaylor@deploy2003 Started scap sync-world: Backport for [[gerrit:1355410|Turn on Wikibase hCaptcha flag for Beta (T440832)]] [08:54:40] T440832: [hCaptcha] Enable hCaptcha on the Beta only - https://phabricator.wikimedia.org/T440832 [08:56:08] ah. yeah. could be that we didn't backport it [08:56:36] (03PS3) 10Cparle: Update commit message validator dep [puppet] - 10https://gerrit.wikimedia.org/r/1325468 (https://phabricator.wikimedia.org/T427387) [08:56:47] (03CR) 10Urbanecm: [C:03+1] "LGTM, let's accept the header too." [puppet] - 10https://gerrit.wikimedia.org/r/1325468 (https://phabricator.wikimedia.org/T427387) (owner: 10Cparle) [09:00:29] (03PS1) 10Muehlenhoff: thumbor-plugins: Rebuild against latest security updates in Trixie [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1361328 [09:00:55] !log arthurtaylor@deploy2003 arthurtaylor, mahmoud-abdelsattar: Backport for [[gerrit:1355410|Turn on Wikibase hCaptcha flag for Beta (T440832)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [09:01:00] T440832: [hCaptcha] Enable hCaptcha on the Beta only - https://phabricator.wikimedia.org/T440832 [09:01:39] hmm. can't connect to spiderpig... [09:01:48] that's going to make this harder [09:02:01] (03Abandoned) 10Ozge: knative-serving: Allow port 8112 in the activator ingress policy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360372 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [09:02:17] WMDE-Fisch: are you able to connect to spiderpig? [09:02:42] codders: [09:02:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [09:02:47] No [09:03:13] well... interesting [09:06:19] (03PS1) 10Brouberol: Fix one druid-analytics.svc.eqiad.wment A record [dns] - 10https://gerrit.wikimedia.org/r/1361331 [09:07:21] (03CR) 10Atsuko: [C:03+1] Fix one druid-analytics.svc.eqiad.wment A record [dns] - 10https://gerrit.wikimedia.org/r/1361331 (owner: 10Brouberol) [09:07:40] I wonder who the spiderpig-responsibles are [09:07:53] (03CR) 10Brouberol: [C:03+2] Fix one druid-analytics.svc.eqiad.wment A record [dns] - 10https://gerrit.wikimedia.org/r/1361331 (owner: 10Brouberol) [09:08:14] !log brouberol@dns1004 START - running authdns-update [09:10:27] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org [09:10:28] okay. now it's just kinda hanging instead of directly reporting the timeout. maybe it's recovering? [09:10:50] !log brouberol@dns1004 END - running authdns-update [09:11:57] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet [09:12:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [09:12:56] (03PS5) 10Brouberol: admin/data: add spiderpig to the airflow users groups [puppet] - 10https://gerrit.wikimedia.org/r/1360469 (https://phabricator.wikimedia.org/T412941) [09:12:58] (03CR) 10Brouberol: admin/data: add spiderpig to the airflow users groups (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1360469 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [09:14:29] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org [09:15:35] RECOVERY - Postfix SMTP on crm2001 is OK: OK - Certificate crm2001.codfw.wmnet will expire on Mon 30 Nov 2026 08:40:00 AM GMT +0000. https://wikitech.wikimedia.org/wiki/Mail%23Troubleshooting [09:17:10] !log arthurtaylor@deploy2003 arthurtaylor, mahmoud-abdelsattar: Continuing with deployment [09:17:56] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet [09:17:56] all good - back on track [09:18:46] (03PS1) 10Muehlenhoff: Add FIDO keys for santhosh and mpostoronca [puppet] - 10https://gerrit.wikimedia.org/r/1361335 (https://phabricator.wikimedia.org/T442758) [09:19:00] (03PS1) 10Aqu: airflow: grow dev-metrics PVC to 20Gi [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361336 (https://phabricator.wikimedia.org/T433696) [09:20:19] we're running past the end of the deployment window but I still have a patch to go [09:20:26] any objections to me deploying the second patch? [09:21:13] !log jiji@cumin1004 START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-eqiad [09:22:20] (03CR) 10Ayounsi: [C:03+1] Update common.py Importer logic to skip all IPIP/Tunnel ints [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1361004 (owner: 10Cathal Mooney) [09:23:13] jouncebot: nowandnext [09:23:13] No deployments scheduled for the next 1 hour(s) and 36 minute(s) [09:23:14] In 1 hour(s) and 36 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T1100) [09:25:09] !log arthurtaylor@deploy2003 Finished scap sync-world: Backport for [[gerrit:1355410|Turn on Wikibase hCaptcha flag for Beta (T440832)]] (duration: 30m 34s) [09:25:15] T440832: [hCaptcha] Enable hCaptcha on the Beta only - https://phabricator.wikimedia.org/T440832 [09:25:17] (03CR) 10Brouberol: [C:03+1] airflow: grow dev-metrics PVC to 20Gi [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361336 (https://phabricator.wikimedia.org/T433696) (owner: 10Aqu) [09:25:52] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO keys for santhosh and mpostoronca [puppet] - 10https://gerrit.wikimedia.org/r/1361335 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [09:26:10] okay. no objections - I will proceed with the second patch [09:26:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 24.5% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:26:36] (03CR) 10Slyngshede: [C:03+1] Add FIDO keys for santhosh and mpostoronca [puppet] - 10https://gerrit.wikimedia.org/r/1361335 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [09:28:09] (03CR) 10TrainBranchBot: [C:03+2] "Approved by arthurtaylor@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355507 (https://phabricator.wikimedia.org/T440832) (owner: 10Mahmoud-abdelsattar) [09:28:21] (03CR) 10CI reject: [V:04-1] Use hCaptcha test keys on Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355507 (https://phabricator.wikimedia.org/T440832) (owner: 10Mahmoud-abdelsattar) [09:29:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12500506 (10phaultfinder) [09:30:01] (03PS2) 10Arthur taylor: Use hCaptcha test keys on Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355507 (https://phabricator.wikimedia.org/T440832) (owner: 10Mahmoud-abdelsattar) [09:30:07] (03PS6) 10Blake: mediawiki: Update mesh modules and dependents [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338000 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [09:30:18] (03CR) 10CI reject: [V:04-1] mediawiki: Update mesh modules and dependents [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338000 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [09:30:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [09:30:57] (03CR) 10TrainBranchBot: "Approved by arthurtaylor@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355507 (https://phabricator.wikimedia.org/T440832) (owner: 10Mahmoud-abdelsattar) [09:31:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 24.5% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:31:34] (03PS2) 10Dpogorzelski: kserve-llm-inference: add router presets for the endpoint picker [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360464 (https://phabricator.wikimedia.org/T436650) [09:31:43] (03Merged) 10jenkins-bot: Use hCaptcha test keys on Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1355507 (https://phabricator.wikimedia.org/T440832) (owner: 10Mahmoud-abdelsattar) [09:32:22] okay. i'm all done - thanks! [09:34:42] (03PS1) 10Muehlenhoff: Remove Luca from Icinga CGI config [puppet] - 10https://gerrit.wikimedia.org/r/1361341 (https://phabricator.wikimedia.org/T443187) [09:35:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 24.37% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [09:35:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [09:37:43] (03CR) 10Muehlenhoff: [C:03+2] Remove Luca from Icinga CGI config [puppet] - 10https://gerrit.wikimedia.org/r/1361341 (https://phabricator.wikimedia.org/T443187) (owner: 10Muehlenhoff) [09:44:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [09:44:34] (03CR) 10Brouberol: [C:03+2] airflow: grow dev-metrics PVC to 20Gi [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361336 (https://phabricator.wikimedia.org/T433696) (owner: 10Aqu) [09:45:43] (03PS7) 10Clément Goubert: rest-gateway: front wdqs v2 endpoints in shadow mode [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333815 (https://phabricator.wikimedia.org/T436778) (owner: 10Gmodena) [09:46:17] (03CR) 10Btullis: "Looks good from a functional viewpoint." [puppet] - 10https://gerrit.wikimedia.org/r/1360469 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [09:49:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [09:49:53] (03PS1) 10Brouberol: deployment_server: chown the airflow-sre values files from root to spiderpig [puppet] - 10https://gerrit.wikimedia.org/r/1361342 (https://phabricator.wikimedia.org/T412941) [09:49:54] (03PS1) 10Brouberol: deployment_server: enroll all airflow services in spiderpig [puppet] - 10https://gerrit.wikimedia.org/r/1361343 (https://phabricator.wikimedia.org/T412941) [09:50:23] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [09:50:26] !log aqu@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply [09:50:57] !log aqu@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply [09:51:06] (03CR) 10CI reject: [V:04-1] deployment_server: chown the airflow-sre values files from root to spiderpig [puppet] - 10https://gerrit.wikimedia.org/r/1361342 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [09:51:13] (03CR) 10Clément Goubert: [C:03+1] rest-gateway: front wdqs v2 endpoints in shadow mode [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333815 (https://phabricator.wikimedia.org/T436778) (owner: 10Gmodena) [09:52:03] (03PS2) 10Brouberol: deployment_server: enroll all airflow services in spiderpig [puppet] - 10https://gerrit.wikimedia.org/r/1361343 (https://phabricator.wikimedia.org/T412941) [09:52:03] (03PS1) 10Brouberol: deployment_server: chown the airflow-sre values files from root to spiderpig [puppet] - 10https://gerrit.wikimedia.org/r/1361344 (https://phabricator.wikimedia.org/T412941) [09:52:34] (03Abandoned) 10Brouberol: deployment_server: chown the airflow-sre values files from root to spiderpig [puppet] - 10https://gerrit.wikimedia.org/r/1361342 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [09:53:35] !log jiji@cumin1004 END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-eqiad [09:56:53] (03PS6) 10Brouberol: admin/data: add spiderpig to the airflow users groups [puppet] - 10https://gerrit.wikimedia.org/r/1360469 (https://phabricator.wikimedia.org/T412941) [09:56:53] (03PS2) 10Brouberol: deployment_server: chown the airflow-sre values files from root to spiderpig [puppet] - 10https://gerrit.wikimedia.org/r/1361344 (https://phabricator.wikimedia.org/T412941) [09:56:53] (03PS3) 10Brouberol: deployment_server: enroll all airflow services in spiderpig [puppet] - 10https://gerrit.wikimedia.org/r/1361343 (https://phabricator.wikimedia.org/T412941) [09:57:00] (03CR) 10Brouberol: admin/data: add spiderpig to the airflow users groups (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1360469 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [09:57:07] (03CR) 10Brouberol: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1360469 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [09:57:22] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet [09:58:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [09:59:56] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12500721 (10phaultfinder) [10:00:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:01:09] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet [10:03:44] RESOLVED: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:05:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:05:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 23.23% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:06:07] (03CR) 10Jelto: "looks mostly good, one typo and a nit in the comments" [puppet] - 10https://gerrit.wikimedia.org/r/1361343 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [10:07:47] (03PS4) 10Brouberol: deployment_server: enroll all airflow services in spiderpig [puppet] - 10https://gerrit.wikimedia.org/r/1361343 (https://phabricator.wikimedia.org/T412941) [10:08:12] (03CR) 10Brouberol: deployment_server: enroll all airflow services in spiderpig (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1361343 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [10:08:15] (03CR) 10Slyngshede: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1359315 (owner: 10Ayounsi) [10:09:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:11:11] (03CR) 10Ayounsi: [C:03+2] Netbox: add alerting for coherence.DevicesCoherence report [puppet] - 10https://gerrit.wikimedia.org/r/1359315 (owner: 10Ayounsi) [10:11:31] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Q1:rack/setup/install mc20[56-73] - https://phabricator.wikimedia.org/T436272#12500748 (10jijiki) [10:14:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:14:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 24.79% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:19:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:19:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 24.79% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:20:52] (03CR) 10Jelto: [C:03+2] deployment_server: enroll all airflow services in spiderpig [puppet] - 10https://gerrit.wikimedia.org/r/1361343 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [10:21:48] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 02 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360406 (https://phabricator.wikimedia.org/T442733) (owner: 10Sergio Gimeno) [10:22:13] (03CR) 10Jelto: [C:03+1] deployment_server: enroll all airflow services in spiderpig [puppet] - 10https://gerrit.wikimedia.org/r/1361343 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [10:24:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12500797 (10phaultfinder) [10:27:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 22.71% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:29:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:29:57] jouncebot: nowandnext [10:29:57] No deployments scheduled for the next 0 hour(s) and 30 minute(s) [10:29:57] In 0 hour(s) and 30 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T1100) [10:30:29] (03PS1) 10Ladsgroup: Populate LinkCache when linking referenced entities [extensions/Wikibase] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361350 (https://phabricator.wikimedia.org/T428512) [10:30:39] (03CR) 10Ladsgroup: [C:03+2] Populate LinkCache when linking referenced entities [extensions/Wikibase] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361350 (https://phabricator.wikimedia.org/T428512) (owner: 10Ladsgroup) [10:32:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 22.71% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:33:41] 10ops-eqiad, 06DC-Ops, 06ServiceOps: Rerack mc1057, mc1061, mc1069 - https://phabricator.wikimedia.org/T443209 (10jijiki) 03NEW [10:33:42] 10ops-eqiad, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12500842 (10Ladsgroup) [10:34:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:34:40] (03PS2) 10Gmodena: WIP: wdqs2 alerts [alerts] - 10https://gerrit.wikimedia.org/r/1360953 (https://phabricator.wikimedia.org/T436018) [10:35:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 24.44% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:35:23] RESOLVED: CertAlmostExpired: gNMI TLS certificate for lsw1-d2-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [10:37:59] 10ops-eqiad, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12500857 (10CWilliams-WMF) DC-ops please can you check to see if there are any upgrades for this box. MariaDB is stopped. [10:38:35] (03Merged) 10jenkins-bot: Populate LinkCache when linking referenced entities [extensions/Wikibase] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361350 (https://phabricator.wikimedia.org/T428512) (owner: 10Ladsgroup) [10:39:25] (03PS3) 10Gmodena: wdqs: add WDQS v2 (qlever) alerts [alerts] - 10https://gerrit.wikimedia.org/r/1360953 (https://phabricator.wikimedia.org/T436018) [10:40:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 24.12% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:40:29] !log ladsgroup@deploy2003 Started scap sync-world: Backport for [[gerrit:1361350|Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) [10:40:35] T428512: Consider caching revision id of properties in CachingFallbackBasedTermLookup - https://phabricator.wikimedia.org/T428512 [10:44:22] !log ladsgroup@deploy2003 ladsgroup: Backport for [[gerrit:1361350|Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [10:46:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 22.17% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [10:46:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:47:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:49:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12500898 (10phaultfinder) [10:50:15] !log ladsgroup@deploy2003 ladsgroup: Continuing with deployment [10:51:00] !log ayounsi@cumin1004 START - Cookbook sre.network.peering with action 'email' for AS: 396993 [10:51:05] !log ayounsi@cumin1004 END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 396993 [10:51:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [10:51:51] (03PS3) 10Scott French: changeprop: Update mesh modules and dependents [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351523 (https://phabricator.wikimedia.org/T427666) [10:52:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [10:52:42] (03PS4) 10Scott French: changeprop: Support enabling the mesh as a sidecar [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351528 (https://phabricator.wikimedia.org/T434925) [10:52:52] (03PS5) 10Scott French: changeprop: Wire in mesh.networkpolicy.egress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352101 (https://phabricator.wikimedia.org/T434925) [10:54:28] !log ayounsi@cumin1004 START - Cookbook sre.network.peering with action 'email' for AS: 40317 [10:54:47] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 40317 [10:55:04] !log ayounsi@cumin1004 START - Cookbook sre.network.peering with action 'email' for AS: 13359 [10:55:55] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 13359 [10:56:12] !log ladsgroup@deploy2003 Finished scap sync-world: Backport for [[gerrit:1361350|Populate LinkCache when linking referenced entities (T428512)]] (1.48.0-wmf.1) (duration: 15m 43s) [10:56:18] T428512: Consider caching revision id of properties in CachingFallbackBasedTermLookup - https://phabricator.wikimedia.org/T428512 [10:56:32] !log ayounsi@cumin1004 START - Cookbook sre.network.peering with action 'email' for AS: 4788 [10:57:53] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788 [11:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T1100) [11:00:51] 06SRE-OnFire, 14SRE-Sprint-Week-Sustainability-March2023, 06Release-Engineering-Team, 10Scap, and 3 others: Add etcdmirror status check to scap - https://phabricator.wikimedia.org/T317403#12500947 (10JMeybohm) For etcdmirror there are EtcdReplicationDown and EtcdReplicationLag alerts. We don't have anythin... [11:03:17] (03PS3) 10Blake: mediawiki: Update mesh modules and dependents [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361351 [11:05:48] (03CR) 10Btullis: admin/data: add spiderpig to the airflow users groups (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1360469 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [11:09:13] (03PS1) 10Matthias Mullie: Squashed diff to master [extensions/ReaderExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361359 [11:10:13] (03CR) 10Dreamy Jazz: [C:03+1] Disable automatic user page watching on loginwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360928 (https://phabricator.wikimedia.org/T442779) (owner: 10Zabe) [11:11:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 22.05% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:11:22] (03PS5) 10JMeybohm: upgrade-firmware: Allow updates of NonRAID controllers [cookbooks] - 10https://gerrit.wikimedia.org/r/1354829 (https://phabricator.wikimedia.org/T438999) [11:12:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [11:12:46] 06SRE, 06Infrastructure-Foundations, 10netops: Consider blocking IPv4 Options and certain IPv6 extension headers on network edge - https://phabricator.wikimedia.org/T443214 (10cmooney) 03NEW p:05Triage→03Low [11:13:18] 06SRE, 06Infrastructure-Foundations, 10netops: Consider blocking IPv4 Options and certain IPv6 extension headers on network edge - https://phabricator.wikimedia.org/T443214#12501010 (10cmooney) [11:13:36] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 02 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/ReaderExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361359 (owner: 10Matthias Mullie) [11:14:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12501013 (10phaultfinder) [11:15:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 23.01% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:16:56] !log jayme@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main1008.eqiad.wmnet with reason: firmware upgrade [11:17:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [11:17:23] (03PS1) 10Muehlenhoff: Add FIDO keys for joal [puppet] - 10https://gerrit.wikimedia.org/r/1361362 (https://phabricator.wikimedia.org/T442758) [11:19:11] (03CR) 10Clément Goubert: [C:03+1] mediawiki: Update mesh modules and dependents [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361351 (owner: 10Blake) [11:20:49] (03CR) 10Clément Goubert: [C:03+1] changeprop: Update mesh modules and dependents [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351523 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [11:21:41] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO keys for joal [puppet] - 10https://gerrit.wikimedia.org/r/1361362 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [11:23:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [11:23:30] (03PS6) 10JMeybohm: upgrade-firmware: Allow updates of NonRAID controllers [cookbooks] - 10https://gerrit.wikimedia.org/r/1354829 (https://phabricator.wikimedia.org/T438999) [11:23:34] (03CR) 10Clément Goubert: [C:03+1] changeprop: Support enabling the mesh as a sidecar [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351528 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [11:24:07] (03PS1) 10Muehlenhoff: Remove legacy key for santhosh [puppet] - 10https://gerrit.wikimedia.org/r/1361364 (https://phabricator.wikimedia.org/T442758) [11:26:51] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy key for santhosh [puppet] - 10https://gerrit.wikimedia.org/r/1361364 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [11:27:09] !log jayme@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1008.eqiad.wmnet [11:27:10] (03CR) 10Clément Goubert: [C:03+1] changeprop: Wire in mesh.networkpolicy.egress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1352101 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [11:27:52] !log jayme@cumin1004 START - Cookbook sre.hosts.reboot-single for host kafka-main1008.eqiad.wmnet [11:28:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [11:28:31] claime: I know it's mw infra window but if it's not causing issues, can I backport a change? https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1358412 [11:29:25] Amir1: sure i think it's fine [11:29:43] (03CR) 10Ladsgroup: [C:03+2] Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358412 (https://phabricator.wikimedia.org/T440885) (owner: 10Krinkle) [11:29:54] thanks. That should reduce the write rate on apcu a lot [11:30:10] that's the top writer of apcu in prod [11:30:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 22.43% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [11:31:05] FIRING: KubernetesCalicoDown: wikikube-worker2337.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2337.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [11:32:58] re-downtiming that worker [11:33:03] !log cgoubert@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2337.codfw.wmnet with reason: hardware/networking issues [11:33:09] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps: hw troubleshooting: power-on for wikikube-worker2337.codfw.wmnet - https://phabricator.wikimedia.org/T442714#12501129 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=3973a5d8-049d-4380-92da-18681e1b9ef5) set by cgoubert@cumin2003 for 7 days, 0... [11:33:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [11:33:40] 10SRE-swift-storage, 06Data-Persistence, 10Prod-Kubernetes, 06ServiceOps, and 5 others: Fix thumbor discovery records and make swift use them - https://phabricator.wikimedia.org/T397618#12501131 (10Ladsgroup) Just a note: The proper fix for this is to get rid of circular service dependency between thumbor... [11:34:26] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:34:41] Amir1: ping me when you're done because i want to reboot some nodes that affect deployment (wikikube-ctrl, dragonfly, chartmuseum) [11:34:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [11:34:56] sure [11:35:13] I already +2'ed that patch, it's being merged [11:35:17] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy2003 using scap backport" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358412 (https://phabricator.wikimedia.org/T440885) (owner: 10Krinkle) [11:35:26] Amir1: ack [11:35:45] !log jiji@cumin1004 START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-eqiad [11:36:07] (03Merged) 10jenkins-bot: Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1358412 (https://phabricator.wikimedia.org/T440885) (owner: 10Krinkle) [11:36:21] !log ladsgroup@deploy2003 Started scap sync-world: Backport for [[gerrit:1358412|Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) [11:36:27] T440885: Reduce apcu writes on GET requests - https://phabricator.wikimedia.org/T440885 [11:37:47] (03PS1) 10Vgutierrez: varnish: add local ESI development environment [puppet] - 10https://gerrit.wikimedia.org/r/1361368 [11:38:53] (03CR) 10CI reject: [V:04-1] varnish: add local ESI development environment [puppet] - 10https://gerrit.wikimedia.org/r/1361368 (owner: 10Vgutierrez) [11:39:01] !log dropped links tables from db1160 (s4 master in eqiad): T437278 [11:39:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:39:07] T437278: Drop unneeded tables from x4 and s4 - https://phabricator.wikimedia.org/T437278 [11:39:34] (03CR) 10Btullis: [C:03+2] dse-k8s-wdqs-test1001: Leave partition 4 empty in the preseed recipe [puppet] - 10https://gerrit.wikimedia.org/r/1358204 (https://phabricator.wikimedia.org/T425653) (owner: 10Btullis) [11:39:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [11:40:23] !log ladsgroup@deploy2003 ladsgroup, krinkle: Backport for [[gerrit:1358412|Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:40:27] (03PS1) 10Audrey Penven: Show copyright warning in MEX [extensions/Wikibase] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361369 (https://phabricator.wikimedia.org/T441916) [11:40:27] !log jayme@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-main1008.eqiad.wmnet [11:40:28] !log jayme@cumin1004 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1008.eqiad.wmnet [11:41:09] !log jayme@cumin1004 START - Cookbook sre.hosts.remove-downtime for kafka-main1008.eqiad.wmnet [11:41:10] !log jayme@cumin1004 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1008.eqiad.wmnet [11:41:24] !log ladsgroup@deploy2003 ladsgroup, krinkle: Continuing with deployment [11:41:31] (03PS7) 10JMeybohm: upgrade-firmware: Allow updates of NonRAID controllers [cookbooks] - 10https://gerrit.wikimedia.org/r/1354829 (https://phabricator.wikimedia.org/T438999) [11:41:55] (03PS2) 10Matthias Mullie: Squashed diff to master [extensions/ReaderExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361359 [11:43:59] (03CR) 10CI reject: [V:04-1] Squashed diff to master [extensions/ReaderExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361359 (owner: 10Matthias Mullie) [11:44:12] !log jiji@cumin1004 END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-eqiad [11:44:28] (03CR) 10Btullis: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1361344 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [11:44:47] (03CR) 10JMeybohm: "Verified by upgrading Non-RAID HBA firmware on kafka-main1008 (https://phabricator.wikimedia.org/T438999)" [cookbooks] - 10https://gerrit.wikimedia.org/r/1354829 (https://phabricator.wikimedia.org/T438999) (owner: 10JMeybohm) [11:45:19] (03CR) 10Brouberol: admin/data: add spiderpig to the airflow users groups (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1360469 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [11:47:17] !log ladsgroup@deploy2003 Finished scap sync-world: Backport for [[gerrit:1358412|Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885)]] (1.48.0-wmf.1) (duration: 10m 56s) [11:47:23] T440885: Reduce apcu writes on GET requests - https://phabricator.wikimedia.org/T440885 [11:47:27] (03PS2) 10Vgutierrez: varnish: add local ESI development environment [puppet] - 10https://gerrit.wikimedia.org/r/1361368 [11:47:28] (03Abandoned) 10JavierMonton: stream: pageview-trending-relative [deployment-charts] - 10https://gerrit.wikimedia.org/r/1329232 (https://phabricator.wikimedia.org/T431555) (owner: 10JavierMonton) [11:47:34] (03CR) 10Blake: [C:03+2] mediawiki: Update mesh modules and dependents [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361351 (owner: 10Blake) [11:48:10] !log jmm@cumin2003 START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS trixie [11:48:19] 06SRE, 06Infrastructure-Foundations: Upgrade rpki hosts to Trixie - https://phabricator.wikimedia.org/T438122#12501161 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2003 for host rpki1001.eqiad.wmnet with OS trixie [11:48:30] (03CR) 10CI reject: [V:04-1] varnish: add local ESI development environment [puppet] - 10https://gerrit.wikimedia.org/r/1361368 (owner: 10Vgutierrez) [11:49:39] !log jmm@cumin2003 START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (push config only) [11:49:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12501173 (10phaultfinder) [11:49:45] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (push config only) [11:49:48] (03CR) 10Klausman: [C:03+1] llm-d: add endpoint-picker and uds-tokenizer images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360463 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [11:50:12] (03Merged) 10jenkins-bot: mediawiki: Update mesh modules and dependents [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361351 (owner: 10Blake) [11:51:24] (03PS3) 10Matthias Mullie: Squashed diff to master [extensions/ReaderExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361359 [11:51:34] (03PS3) 10Vgutierrez: varnish: add local ESI development environment [puppet] - 10https://gerrit.wikimedia.org/r/1361368 [11:55:37] !log jmm@cumin2003 START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (remove config only) [11:55:43] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (remove config only) [11:56:00] !log blake@deploy2003 Started scap sync-world: Non-build deploy to bump chart version (T434925) [11:56:05] T434925: Adopt the service mesh in changeprop / changeprop-jobqueue for diversion to Pretrain - https://phabricator.wikimedia.org/T434925 [11:56:42] (03CR) 10Cathal Mooney: [C:03+1] Remove router access for Luca [homer/public] - 10https://gerrit.wikimedia.org/r/1361317 (https://phabricator.wikimedia.org/T443187) (owner: 10Muehlenhoff) [11:56:54] :'( [11:57:14] (03CR) 10Cathal Mooney: [C:03+2] Update common.py Importer logic to skip all IPIP/Tunnel ints [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1361004 (owner: 10Cathal Mooney) [11:57:16] claime: I'm done [11:57:24] Amir1: ty [11:57:33] !log jmm@cumin2003 START - Cookbook sre.hosts.downtime for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage [11:57:37] jouncebot: nowandnext [11:57:37] For the next 0 hour(s) and 2 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T1100) [11:57:37] In 2 hour(s) and 2 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T1400) [11:59:00] (03Merged) 10jenkins-bot: Update common.py Importer logic to skip all IPIP/Tunnel ints [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/1361004 (owner: 10Cathal Mooney) [11:59:39] !log blake@deploy2003 Finished scap sync-world: Non-build deploy to bump chart version (T434925) (duration: 07m 48s) [12:00:04] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:00:14] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:00:19] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:00:29] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:00:32] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [12:00:41] !log cgoubert@cumin2003 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad [12:00:43] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [12:00:45] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet [12:00:47] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet [12:01:23] (03PS1) 10Audrey Penven: Set license acknowledgement cookie for MEX cypress tests [extensions/EntitySchema] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361372 (https://phabricator.wikimedia.org/T441916) [12:01:31] (03PS1) 10Audrey Penven: Fix incomplete settings in NewEntitySchemaTest [extensions/EntitySchema] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361373 (https://phabricator.wikimedia.org/T441916) [12:01:42] FIRING: JobUnavailable: Reduced availability for job routinator in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:02:08] (03CR) 10Blake: [C:03+2] changeprop: Update mesh modules and dependents [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351523 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [12:02:24] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] Fix incomplete settings in NewEntitySchemaTest [extensions/EntitySchema] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361373 (https://phabricator.wikimedia.org/T441916) (owner: 10Audrey Penven) [12:02:57] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage [12:03:22] !log cmooney@cumin1004 START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary [12:03:50] (03PS1) 10Audrey Penven: Set license acknowledgement cookie for MEX cypress tests [extensions/WikibaseLexeme] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361378 (https://phabricator.wikimedia.org/T441916) [12:03:59] (03PS1) 10Audrey Penven: Set license acknowledgement cookie for MEX cypress tests [extensions/Score] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361379 (https://phabricator.wikimedia.org/T441916) [12:04:07] (03Merged) 10jenkins-bot: changeprop: Update mesh modules and dependents [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351523 (https://phabricator.wikimedia.org/T427666) (owner: 10Scott French) [12:04:24] !log jiji@cumin1004 START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet [12:04:34] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [phabricator/translations] (wmf/stable) - 10https://gerrit.wikimedia.org/r/1361381 (owner: 10L10n-bot) [12:04:49] !log jiji@cumin1004 START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet [12:04:56] !log jayme@cumin1004 START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-eqiad [12:05:34] !log blake@deploy2003 helmfile [staging] START helmfile.d/services/changeprop: apply [12:05:39] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 02 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/EntitySchema] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361372 (https://phabricator.wikimedia.org/T441916) (owner: 10Audrey Penven) [12:05:51] !log blake@deploy2003 helmfile [staging] DONE helmfile.d/services/changeprop: apply [12:06:05] !log blake@deploy2003 helmfile [codfw] START helmfile.d/services/changeprop: apply [12:06:11] !log blake@deploy2003 helmfile [codfw] DONE helmfile.d/services/changeprop: apply [12:06:17] !log blake@deploy2003 helmfile [eqiad] START helmfile.d/services/changeprop: apply [12:06:19] (03PS1) 10Muehlenhoff: Add FIDO key for arthurtaylor [puppet] - 10https://gerrit.wikimedia.org/r/1361387 (https://phabricator.wikimedia.org/T442763) [12:06:22] !log blake@deploy2003 helmfile [eqiad] DONE helmfile.d/services/changeprop: apply [12:06:36] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 02 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/EntitySchema] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361373 (https://phabricator.wikimedia.org/T441916) (owner: 10Audrey Penven) [12:06:37] cmooney@cumin1004 update-extras (PID 250520) is awaiting input [12:06:41] 06SRE, 06Infrastructure-Foundations, 10netops: Consider blocking IPv4 Options and certain IPv6 extension headers on network edge - https://phabricator.wikimedia.org/T443214#12501246 (10cmooney) [12:07:43] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 02 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/Score] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361379 (https://phabricator.wikimedia.org/T441916) (owner: 10Audrey Penven) [12:07:55] (03PS1) 10Ozge: ml-services: Remove qwen36-27b-experiments from llm [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361390 (https://phabricator.wikimedia.org/T442150) [12:08:05] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet [12:08:07] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet [12:08:13] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet [12:08:15] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet [12:08:22] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 02 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/WikibaseLexeme] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361378 (https://phabricator.wikimedia.org/T441916) (owner: 10Audrey Penven) [12:08:38] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 02 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploy" [extensions/Wikibase] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361369 (https://phabricator.wikimedia.org/T441916) (owner: 10Audrey Penven) [12:09:38] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for arthurtaylor [puppet] - 10https://gerrit.wikimedia.org/r/1361387 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [12:09:41] !log jiji@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet [12:09:44] !log jiji@cumin1004 START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet [12:10:15] (03CR) 10Ladsgroup: "I'm hoping to push a change today but it's not merged yet (Ia7c16ca955d87) Is it important to push this asap?" [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1361328 (owner: 10Muehlenhoff) [12:11:42] (03CR) 10Muehlenhoff: "Then simply go ahead with your patch and if there's any blockers to it, we can still merge this one tomorrow." [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1361328 (owner: 10Muehlenhoff) [12:12:24] !log jiji@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet [12:12:26] !log jiji@cumin1004 START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet [12:14:19] (03PS1) 10JMeybohm: k8s.pool-depool-node: Fix repool message [cookbooks] - 10https://gerrit.wikimedia.org/r/1361391 (https://phabricator.wikimedia.org/T440696) [12:14:24] (03CR) 10Btullis: [C:04-1] "I am uncomfortable with this change as it is, because I believe that currently anyone in the `deployment` group is able to sudo to the `sp" [puppet] - 10https://gerrit.wikimedia.org/r/1360469 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [12:14:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12501429 (10phaultfinder) [12:14:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [12:14:54] !log jayme@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet [12:14:58] !log jayme@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet [12:14:59] !log jiji@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet [12:15:20] 10ops-eqiad, 06DC-Ops, 10Prod-Kubernetes, 06ServiceOps, and 2 others: Eqiad worker nodes should be connected to 10G NICs - https://phabricator.wikimedia.org/T440696#12501473 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin1004 pool for host wikikube-worker1250.eqiad... [12:15:33] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet [12:15:34] (03CR) 10Btullis: [C:04-1] "I am uncomfortable with this change as it is, because I believe that currently anyone in the `deployment` group is able to sudo to the `sp" [puppet] - 10https://gerrit.wikimedia.org/r/1361344 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [12:15:35] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet [12:15:38] (03CR) 10Blake: [C:03+2] changeprop: Support enabling the mesh as a sidecar [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351528 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [12:15:41] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet [12:15:44] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet [12:16:12] (03CR) 10Btullis: [C:04-1] "Accidentally resolved this." [puppet] - 10https://gerrit.wikimedia.org/r/1360469 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [12:17:13] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki1001.eqiad.wmnet with OS trixie [12:17:23] 06SRE, 06Infrastructure-Foundations: Upgrade rpki hosts to Trixie - https://phabricator.wikimedia.org/T438122#12501489 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2003 for host rpki1001.eqiad.wmnet with OS trixie completed: - rpki1001 (**PASS**) - Downtimed on Icinga/Alert... [12:17:25] (03Merged) 10jenkins-bot: changeprop: Support enabling the mesh as a sidecar [deployment-charts] - 10https://gerrit.wikimedia.org/r/1351528 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [12:18:16] !log blake@deploy2003 helmfile [staging] START helmfile.d/services/changeprop: apply [12:18:46] !log blake@deploy2003 helmfile [staging] DONE helmfile.d/services/changeprop: apply [12:18:53] !log blake@deploy2003 helmfile [codfw] START helmfile.d/services/changeprop: apply [12:19:11] !log cgoubert@cumin2003 conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw [12:19:29] !log blake@deploy2003 helmfile [codfw] DONE helmfile.d/services/changeprop: apply [12:19:33] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet [12:19:35] !log blake@deploy2003 helmfile [eqiad] START helmfile.d/services/changeprop: apply [12:19:54] !log jiji@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet [12:20:07] !log blake@deploy2003 helmfile [eqiad] DONE helmfile.d/services/changeprop: apply [12:23:21] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet [12:23:23] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet [12:23:29] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1005.eqiad.wmnet [12:23:31] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1005.eqiad.wmnet [12:23:32] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet [12:23:41] !log cgoubert@cumin2003 conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw [12:23:56] !log cgoubert@cumin2003 conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad [12:24:17] (03CR) 10Btullis: [C:03+2] apereo_cas: Add a client_roles parameter for services [puppet] - 10https://gerrit.wikimedia.org/r/1352484 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [12:24:18] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet [12:24:25] PROBLEM - Postfix SMTP on mx-in1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mail%23Troubleshooting [12:24:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [12:25:15] RECOVERY - Postfix SMTP on mx-in1001 is OK: OK - Certificate mx-in1001.wikimedia.org will expire on Wed 30 Dec 2026 10:29:06 AM GMT +0000. https://wikitech.wikimedia.org/wiki/Mail%23Troubleshooting [12:27:24] !log jiji@cumin1004 START - Cookbook sre.hosts.reboot-single for host mc-wf2001.codfw.wmnet [12:28:20] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet [12:29:09] 06SRE, 06Infrastructure-Foundations: Upgrade rpki hosts to Trixie - https://phabricator.wikimedia.org/T438122#12501552 (10MoritzMuehlenhoff) [12:29:23] !log cgoubert@cumin2003 conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad [12:29:57] !log blake@deploy2003 helmfile [staging] START helmfile.d/services/changeprop: apply [12:30:08] !log blake@deploy2003 helmfile [staging] DONE helmfile.d/services/changeprop: apply [12:30:16] !log blake@deploy2003 helmfile [eqiad] START helmfile.d/services/changeprop: apply [12:30:25] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet [12:30:27] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet [12:30:31] !log blake@deploy2003 helmfile [eqiad] DONE helmfile.d/services/changeprop: apply [12:30:33] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1006.eqiad.wmnet [12:30:35] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1006.eqiad.wmnet [12:30:38] !log blake@deploy2003 helmfile [codfw] START helmfile.d/services/changeprop: apply [12:30:43] bjensen: tell me when you're done, I'm gonna reboot dragonfly nodes and that may block deployments [12:31:01] !log blake@deploy2003 helmfile [codfw] DONE helmfile.d/services/changeprop: apply [12:31:03] !log cmooney@cumin1004 END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary [12:31:42] RESOLVED: JobUnavailable: Reduced availability for job routinator in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [12:33:36] claime: i'm done for now, currently observing [12:33:50] 10SRE-SLO, 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)): page_change SLO windows - https://phabricator.wikimedia.org/T438054#12501581 (10APizzata-WMF) Hey @RLazarus I have managed to extract daily data points for the metric and will apply the same methodology applied in the [[ https://gitlab.wi... [12:33:52] ack, shouldn't take too long [12:33:56] !log cgoubert@deploy2003 Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot [12:34:03] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance [12:34:05] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1157 (T441633)', diff saved to https://phabricator.wikimedia.org/P97046 and previous config saved to /var/cache/conftool/dbconfig/20261102-123404-ladsgroup.json [12:34:17] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet [12:34:46] !log jiji@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2001.codfw.wmnet [12:34:49] !log jiji@cumin1004 START - Cookbook sre.hosts.reboot-single for host mc-wf2002.codfw.wmnet [12:37:22] !log cgoubert@cumin2003 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet [12:37:24] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet [12:37:24] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad [12:37:34] !log cmooney@cumin1004 START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox [12:38:04] !log cmooney@cumin1004 END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox [12:38:06] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet [12:40:57] !log jiji@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2002.codfw.wmnet [12:41:50] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1157 (T441633)', diff saved to https://phabricator.wikimedia.org/P97048 and previous config saved to /var/cache/conftool/dbconfig/20261102-124149-ladsgroup.json [12:43:17] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet [12:44:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12501628 (10phaultfinder) [12:44:53] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12501629 (10phaultfinder) [12:45:58] (03PS1) 10Muehlenhoff: Remove repo sync definition for Routinator on Bookworm [puppet] - 10https://gerrit.wikimedia.org/r/1361408 (https://phabricator.wikimedia.org/T438122) [12:46:08] !log jayme@cumin1004 END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad [12:47:16] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet [12:47:45] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org [12:48:17] !log cgoubert@deploy2003 Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 14m 21s) [12:50:20] (03PS1) 10Muehlenhoff: Add FIDO key for kartik [puppet] - 10https://gerrit.wikimedia.org/r/1361409 (https://phabricator.wikimedia.org/T443182) [12:51:52] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97049 and previous config saved to /var/cache/conftool/dbconfig/20261102-125151-ladsgroup.json [12:52:08] (03CR) 10KartikMistry: [C:03+1] Add FIDO key for kartik [puppet] - 10https://gerrit.wikimedia.org/r/1361409 (https://phabricator.wikimedia.org/T443182) (owner: 10Muehlenhoff) [12:52:24] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for kartik [puppet] - 10https://gerrit.wikimedia.org/r/1361409 (https://phabricator.wikimedia.org/T443182) (owner: 10Muehlenhoff) [12:52:48] (03CR) 10Kevin Bazira: [C:03+1] ml-services: Remove qwen36-27b-experiments from llm [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361390 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [12:53:26] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org [12:53:28] FIRING: KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [12:53:54] (03CR) 10Ozge: [C:03+2] ml-services: Remove qwen36-27b-experiments from llm [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361390 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [12:55:36] (03Merged) 10jenkins-bot: ml-services: Remove qwen36-27b-experiments from llm [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361390 (https://phabricator.wikimedia.org/T442150) (owner: 10Ozge) [12:56:17] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org [12:58:28] RESOLVED: KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [12:59:42] FIRING: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:00:18] (03PS1) 10Mvolz: admin: Add FIDO SSH key for mvolz [puppet] - 10https://gerrit.wikimedia.org/r/1361410 (https://phabricator.wikimedia.org/T442756) [13:01:12] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12501720 (10Papaul) a:03VRiley-WMF @CWilliams-WMF yes there are upgrades for the node BIOS: 1.23 IDRAC: 7..30 [13:01:25] (03CR) 10CI reject: [V:04-1] admin: Add FIDO SSH key for mvolz [puppet] - 10https://gerrit.wikimedia.org/r/1361410 (https://phabricator.wikimedia.org/T442756) (owner: 10Mvolz) [13:01:54] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97050 and previous config saved to /var/cache/conftool/dbconfig/20261102-130153-ladsgroup.json [13:02:06] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org [13:02:52] !log jayme@cumin1004 START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-codfw [13:04:42] RESOLVED: JobUnavailable: Reduced availability for job atlas_exporter in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:05:09] !log cwilliams@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1217.eqiad.wmnet with reason: Maintenance [13:05:22] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12501734 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=23a8d1cc-c554-4e07-8429-27379adacb76) set by cwilliams@cumin1004 for 3 days, 0:00:00 on 1 host(s) and their services with reason: Main... [13:05:22] (03PS1) 10Muehlenhoff: Add site.pp/preseed entries for new puppet servers and Ganeti nodes in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1361411 (https://phabricator.wikimedia.org/T434699) [13:05:33] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12501737 (10CWilliams-WMF) @Papaul great, thanks for confirming. The box is ready for those to be applied when ready, I have added downtime. [13:07:47] !log ozge@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [13:07:53] !log ozge@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . [13:08:00] (03CR) 10CI reject: [V:04-1] Add site.pp/preseed entries for new puppet servers and Ganeti nodes in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1361411 (https://phabricator.wikimedia.org/T434699) (owner: 10Muehlenhoff) [13:08:15] 06SRE, 10Wikimedia-Mailing-lists, 07Chinese-Sites: Alternative mailing lists for Chinese ArbCom - https://phabricator.wikimedia.org/T442488#12501745 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup https://lists.wikimedia.org/postorius/lists/wikipedia-zh-arbcom-b.lists.wikimedia.org https://lists.wiki... [13:09:54] (03PS2) 10Muehlenhoff: Add site.pp/preseed entries for new puppet servers and Ganeti nodes in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1361411 (https://phabricator.wikimedia.org/T434699) [13:11:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:11:43] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12501758 (10VRiley-WMF) Will log in and apply updates soon [13:11:56] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1157 (T441633)', diff saved to https://phabricator.wikimedia.org/P97051 and previous config saved to /var/cache/conftool/dbconfig/20261102-131155-ladsgroup.json [13:12:03] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance [13:12:06] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97052 and previous config saved to /var/cache/conftool/dbconfig/20261102-131205-ladsgroup.json [13:12:32] (03CR) 10CI reject: [V:04-1] Add site.pp/preseed entries for new puppet servers and Ganeti nodes in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1361411 (https://phabricator.wikimedia.org/T434699) (owner: 10Muehlenhoff) [13:15:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:17:27] (03PS3) 10Muehlenhoff: Add site.pp/preseed entries for new puppet servers and Ganeti nodes in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1361411 (https://phabricator.wikimedia.org/T434699) [13:20:44] RESOLVED: RipeAtlasAnchorUnreachable: ipv6 ping to eqsin RIPE Atlas anchor: failures over threshold for measurement 95145506 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [13:21:15] (03CR) 10Ladsgroup: [C:03+1] "confirmed oob" [puppet] - 10https://gerrit.wikimedia.org/r/1361107 (https://phabricator.wikimedia.org/T442758) (owner: 10Addshore) [13:21:39] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97054 and previous config saved to /var/cache/conftool/dbconfig/20261102-132138-ladsgroup.json [13:21:54] (03CR) 10Muehlenhoff: [C:03+2] admin: Add new HW backed SSH key for addshore [puppet] - 10https://gerrit.wikimedia.org/r/1361107 (https://phabricator.wikimedia.org/T442758) (owner: 10Addshore) [13:22:24] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet [13:23:18] (03CR) 10Btullis: [C:03+2] idp_test: Accept wmf_s3_login requests only from the Ceph admin hosts [puppet] - 10https://gerrit.wikimedia.org/r/1352485 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [13:23:24] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12501828 (10VRiley-WMF) Starting with BIOS update 1.23.1 [13:25:25] (03PS2) 10Muehlenhoff: admin: Add FIDO SSH key for mvolz [puppet] - 10https://gerrit.wikimedia.org/r/1361410 (https://phabricator.wikimedia.org/T442756) (owner: 10Mvolz) [13:26:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [13:26:14] !log jmm@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet [13:31:41] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97055 and previous config saved to /var/cache/conftool/dbconfig/20261102-133140-ladsgroup.json [13:33:12] (03CR) 10Muehlenhoff: [C:03+1] "Verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1361410 (https://phabricator.wikimedia.org/T442756) (owner: 10Mvolz) [13:33:16] (03CR) 10Muehlenhoff: [C:03+2] admin: Add FIDO SSH key for mvolz [puppet] - 10https://gerrit.wikimedia.org/r/1361410 (https://phabricator.wikimedia.org/T442756) (owner: 10Mvolz) [13:33:18] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12501860 (10VRiley-WMF) BIOS completed. Starting IDRAC [13:36:40] (03PS1) 10Muehlenhoff: Add FIDO key for jforrester [puppet] - 10https://gerrit.wikimedia.org/r/1361421 (https://phabricator.wikimedia.org/T442758) [13:38:39] (03CR) 10Btullis: [C:03+2] idp_test: Remove required_groups from the wmf_s3_login client [puppet] - 10https://gerrit.wikimedia.org/r/1352486 (https://phabricator.wikimedia.org/T435602) (owner: 10Btullis) [13:39:13] (03PS2) 10Muehlenhoff: Add FIDO key for jforrester [puppet] - 10https://gerrit.wikimedia.org/r/1361421 (https://phabricator.wikimedia.org/T442758) [13:39:25] (03CR) 10Klausman: [V:03+2 C:03+2] ml: add vLLM 0.32 image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360334 (https://phabricator.wikimedia.org/T442138) (owner: 10Bartosz Wójtowicz) [13:41:16] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for jforrester [puppet] - 10https://gerrit.wikimedia.org/r/1361421 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [13:41:22] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12501892 (10cmooney) [13:41:43] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97056 and previous config saved to /var/cache/conftool/dbconfig/20261102-134143-ladsgroup.json [13:43:36] (03PS4) 10Gmodena: wdqs: add WDQS v2 (qlever) alerts [alerts] - 10https://gerrit.wikimedia.org/r/1360953 (https://phabricator.wikimedia.org/T436018) [13:44:05] (03CR) 10Dpogorzelski: [C:03+2] llm-d: add endpoint-picker and uds-tokenizer images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360463 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [13:44:52] !log cmooney@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1251.eqiad.wmnet [13:45:44] !log cmooney@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1251.eqiad.wmnet [13:45:55] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12501916 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by cmooney@cumin1004 depool for host wikikube-worker12... [13:46:03] 06SRE, 10SRE-Access-Requests: Requesting access to Analytics Data Lake for Federico Ceratto - https://phabricator.wikimedia.org/T443235 (10FCeratto-WMF) 03NEW [13:46:06] 06SRE, 10MediaWiki-General, 06ServiceOps: Introduce a global load shedding level - https://phabricator.wikimedia.org/T443236 (10Ladsgroup) 03NEW [13:46:22] (03CR) 10CI reject: [V:04-1] wdqs: add WDQS v2 (qlever) alerts [alerts] - 10https://gerrit.wikimedia.org/r/1360953 (https://phabricator.wikimedia.org/T436018) (owner: 10Gmodena) [13:46:48] 06SRE, 10MediaWiki-General, 06ServiceOps, 07Epic: Introduce a global load shedding level - https://phabricator.wikimedia.org/T443236#12501938 (10Ladsgroup) [13:47:08] (03PS1) 10Muehlenhoff: Add FIDO key for tarrow [puppet] - 10https://gerrit.wikimedia.org/r/1361425 (https://phabricator.wikimedia.org/T442763) [13:47:38] (03PS2) 10Muehlenhoff: Add FIDO key for tarrow [puppet] - 10https://gerrit.wikimedia.org/r/1361425 (https://phabricator.wikimedia.org/T442763) [13:48:48] (03CR) 10CI reject: [V:04-1] Add FIDO key for tarrow [puppet] - 10https://gerrit.wikimedia.org/r/1361425 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [13:49:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12501970 (10phaultfinder) [13:49:45] !log aqu@deploy2003 Started deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe] [13:50:32] !log aqu@deploy2003 Finished deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe] (duration: 00m 47s) [13:50:37] !log aqu@deploy2003 Started deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe] [13:51:46] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97057 and previous config saved to /var/cache/conftool/dbconfig/20261102-135145-ladsgroup.json [13:51:51] (03CR) 10Slyngshede: [C:04-1] Add FIDO key for tarrow (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1361425 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [13:52:03] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance [13:53:50] (03PS3) 10Muehlenhoff: Add FIDO key for tarrow [puppet] - 10https://gerrit.wikimedia.org/r/1361425 (https://phabricator.wikimedia.org/T442763) [13:53:55] (03CR) 10Muehlenhoff: Add FIDO key for tarrow (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1361425 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [13:54:36] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12502041 (10VRiley-WMF) IDRAC is completed with BIOS. This should be all set @CWilliams-WMF [13:55:23] !log aqu@deploy2003 Finished deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe] (duration: 04m 46s) [13:55:48] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for tarrow [puppet] - 10https://gerrit.wikimedia.org/r/1361425 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [13:56:31] 06SRE, 10SRE-Access-Requests: Requesting access to Analytics Data Lake for Federico Ceratto - https://phabricator.wikimedia.org/T443235#12502048 (10FCeratto-WMF) [13:57:43] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance [13:57:45] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1198 (T441633)', diff saved to https://phabricator.wikimedia.org/P97059 and previous config saved to /var/cache/conftool/dbconfig/20261102-135744-ladsgroup.json [13:57:53] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12502075 (10CWilliams-WMF) Thanks for the quick response, @VRiley-WMF! I will get MariaDB running again so that the box can catch up. [13:58:16] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12502076 (10CWilliams-WMF) a:05VRiley-WMF→03CWilliams-WMF [13:59:03] !log jayme@cumin1004 END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-codfw [14:00:05] Lucas_WMDE, urbanecm, and TheresNoTime: That opportune time for a UTC afternoon backport window deploy is upon us again. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T1400). [14:00:05] yerdua_wmde, vs1, sergi0, and matthiasmullie: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [14:00:12] o/ [14:00:24] hi [14:00:24] o/ [14:00:27] I'm here [14:00:34] o/ [14:00:37] I would very much like to deploy yerdua_wmde’s changes first, because that resolves some bad legal liability in Wikidata [14:00:47] unfortunately they’ll take a while due to a full l10n cache rebuild :S [14:00:56] but I’ll still proceed unless someone shouts otherwise [14:01:45] actually, I see matthiasmullie’s change also includes l10n changes… [14:01:55] would it make sense to do that together? [14:01:59] matthiasmullie: how risky is your backport? [14:02:03] Lucas_WMDE: any chance you could combine with mine? Also takes a i18n rebuild, and supposed to be pretty safe (nothing actively in use) [14:02:11] hehe, great minds ^^ [14:02:18] alright then let’s do that [14:02:37] !log cmooney@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1251.eqiad.wmnet [14:02:39] !log cmooney@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1251.eqiad.wmnet [14:02:45] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12502118 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by cmooney@cumin1004 pool for host wikikube-worker1251... [14:03:04] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy2003 using scap backport" [extensions/EntitySchema] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361372 (https://phabricator.wikimedia.org/T441916) (owner: 10Audrey Penven) [14:03:05] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy2003 using scap backport" [extensions/EntitySchema] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361373 (https://phabricator.wikimedia.org/T441916) (owner: 10Audrey Penven) [14:03:05] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy2003 using scap backport" [extensions/Score] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361379 (https://phabricator.wikimedia.org/T441916) (owner: 10Audrey Penven) [14:03:06] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy2003 using scap backport" [extensions/WikibaseLexeme] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361378 (https://phabricator.wikimedia.org/T441916) (owner: 10Audrey Penven) [14:03:07] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy2003 using scap backport" [extensions/Wikibase] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361369 (https://phabricator.wikimedia.org/T441916) (owner: 10Audrey Penven) [14:03:11] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy2003 using scap backport" [extensions/ReaderExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361359 (owner: 10Matthias Mullie) [14:04:07] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1198 (T441633)', diff saved to https://phabricator.wikimedia.org/P97060 and previous config saved to /var/cache/conftool/dbconfig/20261102-140406-ladsgroup.json [14:04:31] (03Merged) 10jenkins-bot: Set license acknowledgement cookie for MEX cypress tests [extensions/EntitySchema] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361372 (https://phabricator.wikimedia.org/T441916) (owner: 10Audrey Penven) [14:04:43] (03Merged) 10jenkins-bot: Fix incomplete settings in NewEntitySchemaTest [extensions/EntitySchema] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361373 (https://phabricator.wikimedia.org/T441916) (owner: 10Audrey Penven) [14:04:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12502121 (10phaultfinder) [14:04:45] (03Merged) 10jenkins-bot: Set license acknowledgement cookie for MEX cypress tests [extensions/Score] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361379 (https://phabricator.wikimedia.org/T441916) (owner: 10Audrey Penven) [14:04:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12502122 (10phaultfinder) [14:05:09] (03Merged) 10jenkins-bot: Set license acknowledgement cookie for MEX cypress tests [extensions/WikibaseLexeme] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361378 (https://phabricator.wikimedia.org/T441916) (owner: 10Audrey Penven) [14:06:01] !log cmooney@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1252.eqiad.wmnet [14:06:12] !log sukhe@puppetserver1001 conftool action : set/pooled=no; selector: name=dns7001.* [reason: testing pdns-rec 5.4.9] [14:07:24] !log cmooney@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1252.eqiad.wmnet [14:07:31] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12502140 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by cmooney@cumin1004 depool for host wikikube-worker12... [14:07:56] (03PS1) 10Kamila Součková: deployment_server/mw-*: switch next and migration to PHP 8.5 [puppet] - 10https://gerrit.wikimedia.org/r/1361437 (https://phabricator.wikimedia.org/T439955) [14:09:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12502147 (10phaultfinder) [14:09:39] (03Merged) 10jenkins-bot: Show copyright warning in MEX [extensions/Wikibase] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361369 (https://phabricator.wikimedia.org/T441916) (owner: 10Audrey Penven) [14:09:41] (03Merged) 10jenkins-bot: Squashed diff to master [extensions/ReaderExperiments] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361359 (owner: 10Matthias Mullie) [14:09:58] Lukas_WMDE: Could you ping me when you are done? [14:10:04] !log lucaswerkmeister-wmde@deploy2003 Started scap sync-world: Backport for [[gerrit:1361372|Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373|Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379|Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378|Set license acknowledgement cookie f [14:10:04] or MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361369|Show copyright warning in MEX (T441916)]] (1.48.0-wmf.1), [[gerrit:1361359|Squashed diff to master]] (1.48.0-wmf.1) [14:10:11] T441916: [MEX] Display terms and conditions when publishing edits - https://phabricator.wikimedia.org/T441916 [14:10:12] ok [14:11:05] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: name=dns7001.* [reason: [end] testing pdns-rec 5.4.9] [14:14:09] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97061 and previous config saved to /var/cache/conftool/dbconfig/20261102-141408-ladsgroup.json [14:15:29] (03CR) 10Kamila Součková: "This is now handled in scap:" [puppet] - 10https://gerrit.wikimedia.org/r/1357273 (https://phabricator.wikimedia.org/T440321) (owner: 10Kamila Součková) [14:16:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:21:27] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12502209 (10VRiley-WMF) @Marostegui is this okay to proceed with? We can update this unit today. [14:21:29] (03PS1) 10Kamila Součková: Revert "shellbox: revert all back to 8.3 flavors" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361446 [14:22:08] (03PS1) 10Mahmoud-abdelsattar: Stop exempting item/property creation from hCaptcha [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361447 (https://phabricator.wikimedia.org/T440832) [14:22:39] (03PS1) 10Muehlenhoff: Add FIDO key for wmde-leszek [puppet] - 10https://gerrit.wikimedia.org/r/1361448 (https://phabricator.wikimedia.org/T442763) [14:24:12] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97062 and previous config saved to /var/cache/conftool/dbconfig/20261102-142411-ladsgroup.json [14:24:18] image build finally finished [14:24:32] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for wmde-leszek [puppet] - 10https://gerrit.wikimedia.org/r/1361448 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [14:24:46] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12502248 (10Marostegui) >>! In T442800#12502209, @VRiley-WMF wrote: > @Marostegui is this okay to proceed with? We can update this unit today. Yes please go ahead. [14:24:50] (03CR) 10JMeybohm: [C:04-1] "I think this needs some form of protection against running it with action "reboot" on stacked control planes (suggest `sre.k8s.reboot-node" [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) (owner: 10Brouberol) [14:25:28] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12502255 (10VRiley-WMF) Thank you, starting now. [14:26:24] (03CR) 10Btullis: [C:04-1] admin/data: add spiderpig to the airflow users groups (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1360469 (https://phabricator.wikimedia.org/T412941) (owner: 10Brouberol) [14:26:58] (03CR) 10Brouberol: "Can you clarify what you're calling "stacked control planes"? I'm not sure I follow" [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) (owner: 10Brouberol) [14:27:15] FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 22.33% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [14:28:48] (03CR) 10Ssingh: "Hi Dawid. Just as an FYI, helmfile reviews are outside of the purview of Traffic, in case you were blocked on us. Sorry for not being clea" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1359259 (https://phabricator.wikimedia.org/T441912) (owner: 10Dpogorzelski) [14:29:24] 10SRE-SLO, 10Citoid, 06Editing-team, 07Sustainability (Incident Followup): Create probe for citoid that tests whether requests can access the outside internet - https://phabricator.wikimedia.org/T381372#12502279 (10Mvolz) @CDanis Did you need help deciding which urls to use for a probe? Historically we u... [14:29:37] !log lucaswerkmeister-wmde@deploy2003 mlitn, audreypenven, lucaswerkmeister-wmde: Backport for [[gerrit:1361372|Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373|Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379|Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378|Set license ackno [14:29:38] wledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361369|Show copyright warning in MEX (T441916)]] (1.48.0-wmf.1), [[gerrit:1361359|Squashed diff to master]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:29:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12502282 (10phaultfinder) [14:29:43] T441916: [MEX] Display terms and conditions when publishing edits - https://phabricator.wikimedia.org/T441916 [14:29:48] yerdua_wmde, matthiasmullie: please test! [14:29:59] testing! [14:30:15] (03PS1) 10Klausman: hiera/docker/ml-build: add machinery for UID mappings [puppet] - 10https://gerrit.wikimedia.org/r/1361450 (https://phabricator.wikimedia.org/T442138) [14:30:22] Lucas_WMDE: LGTM [14:31:07] (03PS1) 10Jgiannelos: noop: Force image rebuild [software/tegola] (wmf/v0.19.x) - 10https://gerrit.wikimedia.org/r/1361458 [14:31:39] (03PS1) 10Jelto: deployment_server::deploy_service::service_catalog: add wdqs-next [puppet] - 10https://gerrit.wikimedia.org/r/1361459 (https://phabricator.wikimedia.org/T412941) [14:32:30] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] llm-d: add endpoint-picker and uds-tokenizer images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1360463 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [14:33:01] Lucas_WMDE confirmed that it works [14:33:54] (03PS1) 10Muehlenhoff: Remove legacy key for brennen [puppet] - 10https://gerrit.wikimedia.org/r/1361461 (https://phabricator.wikimedia.org/T442761) [14:34:10] (03CR) 10Bartosz Wójtowicz: hiera/docker/ml-build: add machinery for UID mappings (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1361450 (https://phabricator.wikimedia.org/T442138) (owner: 10Klausman) [14:34:14] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1198 (T441633)', diff saved to https://phabricator.wikimedia.org/P97063 and previous config saved to /var/cache/conftool/dbconfig/20261102-143413-ladsgroup.json [14:34:19] !log lucaswerkmeister-wmde@deploy2003 mlitn, audreypenven, lucaswerkmeister-wmde: Continuing with deployment [14:34:21] alright, thanks! [14:34:32] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance [14:34:53] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance [14:34:55] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1212 (T441633)', diff saved to https://phabricator.wikimedia.org/P97064 and previous config saved to /var/cache/conftool/dbconfig/20261102-143454-ladsgroup.json [14:35:24] (03PS2) 10Klausman: hiera/docker/ml-build: add machinery for UID mappings [puppet] - 10https://gerrit.wikimedia.org/r/1361450 (https://phabricator.wikimedia.org/T442138) [14:35:39] (03CR) 10Klausman: hiera/docker/ml-build: add machinery for UID mappings (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1361450 (https://phabricator.wikimedia.org/T442138) (owner: 10Klausman) [14:35:51] (03CR) 10Klausman: [V:03+2 C:03+2] hiera/docker/ml-build: add machinery for UID mappings [puppet] - 10https://gerrit.wikimedia.org/r/1361450 (https://phabricator.wikimedia.org/T442138) (owner: 10Klausman) [14:36:00] !log cmooney@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet [14:36:02] !log cmooney@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet [14:36:14] 10SRE-swift-storage, 10Thumbor: Gradually drop all thumbnails as a one-off clean up - https://phabricator.wikimedia.org/T379942#12502306 (10Ladsgroup) Update: The deletion is still ongoing, there is a bit left but I also made one last run that is done much faster (probably done in a couple of days at most). It... [14:36:16] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12502307 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by cmooney@cumin1004 pool for host wikikube-worker1252... [14:39:21] (03PS1) 10Rsilvola: miscweb: Deploy new security landing page [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361463 (https://phabricator.wikimedia.org/T436543) [14:40:18] (03PS3) 10JMeybohm: kubernetes-generic: Alert on coredns failing uptream health checks [alerts] - 10https://gerrit.wikimedia.org/r/1355559 (https://phabricator.wikimedia.org/T439575) [14:40:53] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1212 (T441633)', diff saved to https://phabricator.wikimedia.org/P97065 and previous config saved to /var/cache/conftool/dbconfig/20261102-144052-ladsgroup.json [14:41:06] (03PS5) 10Slyngshede: CAS 8.0.3 [software/cas-overlay-template] - 10https://gerrit.wikimedia.org/r/1346564 (https://phabricator.wikimedia.org/T439509) [14:41:55] (03PS1) 10AOkoth: devtools: setting up deploy host [puppet] - 10https://gerrit.wikimedia.org/r/1361464 [14:42:51] (03CR) 10JMeybohm: "I've added a runbook for this but I'd like to keep it non task (at least initially) since this is likely to be a side-effect/consequence o" [alerts] - 10https://gerrit.wikimedia.org/r/1355559 (https://phabricator.wikimedia.org/T439575) (owner: 10JMeybohm) [14:42:55] (03PS1) 10Muehlenhoff: Add FIDO keys for jnuche [puppet] - 10https://gerrit.wikimedia.org/r/1361465 (https://phabricator.wikimedia.org/T442761) [14:43:35] (03CR) 10AOkoth: [C:03+2] devtools: setting up deploy host [puppet] - 10https://gerrit.wikimedia.org/r/1361464 (owner: 10AOkoth) [14:44:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12502397 (10phaultfinder) [14:47:03] !log lucaswerkmeister-wmde@deploy2003 Finished scap sync-world: Backport for [[gerrit:1361372|Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361373|Fix incomplete settings in NewEntitySchemaTest (T441916)]] (1.48.0-wmf.1), [[gerrit:1361379|Set license acknowledgement cookie for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361378|Set license acknowledgement cookie [14:47:03] for MEX cypress tests (T441916)]] (1.48.0-wmf.1), [[gerrit:1361369|Show copyright warning in MEX (T441916)]] (1.48.0-wmf.1), [[gerrit:1361359|Squashed diff to master]] (1.48.0-wmf.1) (duration: 36m 59s) [14:47:10] T441916: [MEX] Display terms and conditions when publishing edits - https://phabricator.wikimedia.org/T441916 [14:47:14] vs1: I’m done deploying [14:47:38] (03CR) 10JMeybohm: [C:04-1] "Oh, sorry. The control planes which co-locate etcd and k8s apiserver (https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters/Add_or_remov" [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) (owner: 10Brouberol) [14:47:51] also ping sergi0 I suppose [14:48:30] (03CR) 10Blake: [C:03+1] "Looks good to me, thanks for adding the runbook 😊" [alerts] - 10https://gerrit.wikimedia.org/r/1355559 (https://phabricator.wikimedia.org/T439575) (owner: 10JMeybohm) [14:48:32] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12502447 (10VRiley-WMF) This has been completed @Marostegui let's hope that's all it was. Thanks! [14:48:45] I'm ready [14:48:48] thanks Lucas_WMDE , @vs1 are you going next? [14:49:18] (03CR) 10JMeybohm: [C:04-1] "That's imprecise given there is another role following this approach and others might appear. Basically it's nodes that include `profile::" [cookbooks] - 10https://gerrit.wikimedia.org/r/1359220 (https://phabricator.wikimedia.org/T420588) (owner: 10Brouberol) [14:49:34] depends on Lucas [14:49:36] vs1: do you need a deployer? [14:49:45] yes [14:49:46] I was wondering why you only wanted to be pinged :P [14:49:47] ok ^^ [14:49:54] I don’t think I can review all those changes today [14:50:01] but maybe the throttle exemption [14:50:04] as that’s more time-critical [14:50:55] Throttle exemption happen on 7 Nov so it should go first [14:50:55] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97066 and previous config saved to /var/cache/conftool/dbconfig/20261102-145054-ladsgroup.json [14:51:01] (03CR) 10Lucas Werkmeister (WMDE): Lift IP cap for edit-a-thon (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361187 (https://phabricator.wikimedia.org/T443033) (owner: 10Tryvix1509) [14:51:07] vs1: ^ comment [14:51:45] sergi0: we can probably deploy your change together with vs1’s? looks fairly unrisky to me [14:51:57] yes, go for it [14:53:10] (03PS1) 10Muehlenhoff: Remove legacy key for joal [puppet] - 10https://gerrit.wikimedia.org/r/1361468 (https://phabricator.wikimedia.org/T442758) [14:53:45] (03CR) 10Tryvix1509: Lift IP cap for edit-a-thon (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361187 (https://phabricator.wikimedia.org/T443033) (owner: 10Tryvix1509) [14:54:31] (03CR) 10Lucas Werkmeister (WMDE): ukwiki: limit "user" namespace to only registered users (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360437 (https://phabricator.wikimedia.org/T438016) (owner: 10Neriah) [14:55:07] (03PS5) 10Dzahn: rsyslog: remove lookup entries for php7.0-fpm [puppet] - 10https://gerrit.wikimedia.org/r/1341351 (https://phabricator.wikimedia.org/T435393) (owner: 10Southparkfan) [14:56:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:56:35] (03CR) 10Lucas Werkmeister (WMDE): Lift IP cap for edit-a-thon (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361187 (https://phabricator.wikimedia.org/T443033) (owner: 10Tryvix1509) [14:57:23] (03CR) 10Lucas Werkmeister (WMDE): [C:03+1] eswiki: Add protect user right to botadmin group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361148 (https://phabricator.wikimedia.org/T443124) (owner: 10Tryvix1509) [14:57:24] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy key for joal [puppet] - 10https://gerrit.wikimedia.org/r/1361468 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [14:57:45] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361148 (https://phabricator.wikimedia.org/T443124) (owner: 10Tryvix1509) [14:57:46] (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360406 (https://phabricator.wikimedia.org/T442733) (owner: 10Sergio Gimeno) [14:58:35] (03Merged) 10jenkins-bot: eswiki: Add protect user right to botadmin group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361148 (https://phabricator.wikimedia.org/T443124) (owner: 10Tryvix1509) [14:58:39] (03Merged) 10jenkins-bot: GrowthExperiments: enable SpecialHome on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360406 (https://phabricator.wikimedia.org/T442733) (owner: 10Sergio Gimeno) [14:58:55] !log lucaswerkmeister-wmde@deploy2003 Started scap sync-world: Backport for [[gerrit:1361148|eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406|GrowthExperiments: enable SpecialHome on testwiki (T442733)]] [14:59:05] T443124: Add the protect user right to the botadmin group on Spanish Wikipedia - https://phabricator.wikimedia.org/T443124 [14:59:05] T442733: Release Special:Home in testwiki - https://phabricator.wikimedia.org/T442733 [14:59:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12502503 (10phaultfinder) [15:00:58] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97067 and previous config saved to /var/cache/conftool/dbconfig/20261102-150057-ladsgroup.json [15:02:10] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH keys for aikochou - https://phabricator.wikimedia.org/T442367#12502544 (10Dzahn) @andrea.denisse Aiko will update with a new key once she has obtained a yubikey. currently waiting for that. [15:03:17] (03PS1) 10Dpogorzelski: llm-d: build the endpoint picker with Go 1.26 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1361471 (https://phabricator.wikimedia.org/T436650) [15:03:20] !log lucaswerkmeister-wmde@deploy2003 sgimeno, lucaswerkmeister-wmde, tryvix1509: Backport for [[gerrit:1361148|eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406|GrowthExperiments: enable SpecialHome on testwiki (T442733)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:03:22] (03PS3) 10Dpogorzelski: kserve-llm-inference: add router presets for the endpoint picker [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360464 (https://phabricator.wikimedia.org/T436650) [15:03:31] vs1: please test the eswiki change [15:03:34] sergi0: please test the testwiki change [15:03:44] on it [15:04:36] (03CR) 10Klausman: [C:03+1] kserve-llm-inference: add router presets for the endpoint picker [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360464 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [15:04:53] Lucas_WMDE: looking good from my end [15:05:00] (03CR) 10Jgiannelos: [C:03+2] noop: Force image rebuild [software/tegola] (wmf/v0.19.x) - 10https://gerrit.wikimedia.org/r/1361458 (owner: 10Jgiannelos) [15:05:05] ok [15:05:28] Lukas_WMDE: eswiki check OK. [15:05:33] !log lucaswerkmeister-wmde@deploy2003 sgimeno, lucaswerkmeister-wmde, tryvix1509: Continuing with deployment [15:06:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:07:19] (03CR) 10Jaime Nuche: [C:03+1] Add FIDO keys for jnuche [puppet] - 10https://gerrit.wikimedia.org/r/1361465 (https://phabricator.wikimedia.org/T442761) (owner: 10Muehlenhoff) [15:07:31] (03CR) 10Dpogorzelski: [C:03+2] llm-d: build the endpoint picker with Go 1.26 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1361471 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [15:07:33] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] llm-d: build the endpoint picker with Go 1.26 [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1361471 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [15:10:34] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12502604 (10CWilliams-WMF) Thanks @VRiley-WMF ! [15:11:00] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1212 (T441633)', diff saved to https://phabricator.wikimedia.org/P97068 and previous config saved to /var/cache/conftool/dbconfig/20261102-151059-ladsgroup.json [15:11:02] (03CR) 10Btullis: [C:03+1] "Looks good, thanks." [puppet] - 10https://gerrit.wikimedia.org/r/1361459 (https://phabricator.wikimedia.org/T412941) (owner: 10Jelto) [15:11:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:11:18] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance [15:11:20] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1223 (T441633)', diff saved to https://phabricator.wikimedia.org/P97069 and previous config saved to /var/cache/conftool/dbconfig/20261102-151119-ladsgroup.json [15:11:40] (03PS1) 10Hnowlan: swagger: Make alerting message more useful [alerts] - 10https://gerrit.wikimedia.org/r/1361473 (https://phabricator.wikimedia.org/T442710) [15:12:18] (03PS1) 10Atsuko: trafficserver: increase timeouts for airflow-dumps [puppet] - 10https://gerrit.wikimedia.org/r/1361474 (https://phabricator.wikimedia.org/T443252) [15:12:40] (03CR) 10Atsuko: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1361474 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [15:13:23] (03CR) 10Dpogorzelski: [C:03+2] kserve-llm-inference: add router presets for the endpoint picker [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360464 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [15:13:31] !log lucaswerkmeister-wmde@deploy2003 Finished scap sync-world: Backport for [[gerrit:1361148|eswiki: Add protect user right to botadmin group (T443124)]], [[gerrit:1360406|GrowthExperiments: enable SpecialHome on testwiki (T442733)]] (duration: 14m 35s) [15:13:39] T443124: Add the protect user right to the botadmin group on Spanish Wikipedia - https://phabricator.wikimedia.org/T443124 [15:13:40] T442733: Release Special:Home in testwiki - https://phabricator.wikimedia.org/T442733 [15:13:49] !log UTC afternoon backport+config window done [15:13:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:14:00] Lucas_WMDE: thank you! [15:14:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12502653 (10Jhancock.wm) a:03Jhancock.wm moved a server to a different channel at 9:14am local time. holding to see if alert clears. [15:15:22] (03Merged) 10jenkins-bot: kserve-llm-inference: add router presets for the endpoint picker [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360464 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [15:15:31] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12502659 (10CWilliams-WMF) a:05VRiley-WMF→03CWilliams-WMF I have started replication to give it a little workout. [15:17:17] Thanks Lucas [15:17:27] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1223 (T441633)', diff saved to https://phabricator.wikimedia.org/P97070 and previous config saved to /var/cache/conftool/dbconfig/20261102-151726-ladsgroup.json [15:17:30] (03CR) 10Majavah: [C:03+1] wmcs root keys: add FIDO keys for volans [puppet] - 10https://gerrit.wikimedia.org/r/1360965 (https://phabricator.wikimedia.org/T443002) (owner: 10Volans) [15:17:32] !log aqu@deploy2003 Started deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6] [15:17:35] (03CR) 10Majavah: [C:03+1] wmcs: add cloudvps root FIDO keys for filippo [puppet] - 10https://gerrit.wikimedia.org/r/1361321 (https://phabricator.wikimedia.org/T443002) (owner: 10Filippo Giunchedi) [15:17:39] (03PS1) 10Gkyziridis: ml-services: place jina-embeddings on 24G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361475 (https://phabricator.wikimedia.org/T443256) [15:18:17] !log aqu@deploy2003 Finished deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6] (duration: 00m 44s) [15:18:30] PROBLEM - Host ldap-replica2007 is DOWN: PING CRITICAL - Packet loss = 100% [15:19:08] PROBLEM - Host dse-k8s-etcd2001 is DOWN: PING CRITICAL - Packet loss = 100% [15:19:08] PROBLEM - Host dse-k8s-ctrl2001 is DOWN: PING CRITICAL - Packet loss = 100% [15:19:08] PROBLEM - Host ml-etcd2002 is DOWN: PING CRITICAL - Packet loss = 100% [15:19:12] (03CR) 10Btullis: trafficserver: increase timeouts for airflow-dumps (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1361474 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [15:19:16] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c4-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442765#12502733 (10Jhancock.wm) a:03Jhancock.wm changed one server to a different channel at 9:18am local time. waiting to see if it resolves. [15:19:28] PROBLEM - Host ganeti2037 is DOWN: PING CRITICAL - Packet loss = 100% [15:19:56] PROBLEM - Host schema2003 is DOWN: PING CRITICAL - Packet loss = 100% [15:20:02] PROBLEM - Host aux-k8s-worker2004 is DOWN: PING CRITICAL - Packet loss = 100% [15:20:02] PROBLEM - Host tcp-proxy2001 is DOWN: PING CRITICAL - Packet loss = 100% [15:20:23] huh [15:20:31] what's this [15:20:50] FIRING: ProbeDown: Service ganeti2037:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:20:59] (03PS2) 10Atsuko: trafficserver: increase timeouts for airflow-dumps [puppet] - 10https://gerrit.wikimedia.org/r/1361474 (https://phabricator.wikimedia.org/T443252) [15:21:07] (03CR) 10Atsuko: trafficserver: increase timeouts for airflow-dumps (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1361474 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [15:21:08] that's C4 [15:21:09] (03CR) 10Atsuko: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1361474 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [15:22:02] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - sophroid_4252: Servers aux-k8s-worker2003.codfw.wmnet, aux-k8s-worker2005.codfw.wmnet, aux-k8s-worker2002.codfw.wmnet, aux-k8s-worker2009.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [15:22:02] PROBLEM - PyBal backends health check on lvs2013 is CRITICAL: PYBAL CRITICAL - CRITICAL - sophroid_4252: Servers aux-k8s-worker2007.codfw.wmnet, aux-k8s-worker2008.codfw.wmnet, aux-k8s-worker2003.codfw.wmnet, aux-k8s-worker2002.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [15:22:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:23:04] (03CR) 10Btullis: [C:03+1] "Looks good to me." [puppet] - 10https://gerrit.wikimedia.org/r/1361474 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [15:23:20] FIRING: KubernetesCalicoDown: dse-k8s-ctrl2001.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s-dse&var-instance=dse-k8s-ctrl2001.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:24:02] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [15:24:40] !log aqu@deploy2003 Started deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6] [15:24:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12502825 (10phaultfinder) [15:26:06] FIRING: [2x] KubernetesCalicoDown: aux-k8s-worker2004.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:26:07] (03PS1) 10Jgiannelos: maintenance: Fix python dependencies [software/tegola] (wmf/v0.19.x) - 10https://gerrit.wikimedia.org/r/1361476 [15:27:09] (03CR) 10Jgiannelos: [C:03+2] maintenance: Fix python dependencies [software/tegola] (wmf/v0.19.x) - 10https://gerrit.wikimedia.org/r/1361476 (owner: 10Jgiannelos) [15:27:15] FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:27:29] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97072 and previous config saved to /var/cache/conftool/dbconfig/20261102-152728-ladsgroup.json [15:27:33] (03CR) 10Vgutierrez: [C:03+1] trafficserver: increase timeouts for airflow-dumps [puppet] - 10https://gerrit.wikimedia.org/r/1361474 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [15:28:02] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [15:28:06] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [15:28:31] (03CR) 10Atsuko: [C:03+2] trafficserver: increase timeouts for airflow-dumps [puppet] - 10https://gerrit.wikimedia.org/r/1361474 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [15:28:40] FIRING: KubernetesRsyslogDown: rsyslog on dse-k8s-worker2006:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=dse-k8s-worker2006 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [15:28:40] (03CR) 10CI reject: [V:04-1] maintenance: Fix python dependencies [software/tegola] (wmf/v0.19.x) - 10https://gerrit.wikimedia.org/r/1361476 (owner: 10Jgiannelos) [15:28:42] FIRING: JobUnavailable: Reduced availability for job tcp_proxy in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:28:46] RECOVERY - Host ganeti2037 is UP: PING OK - Packet loss = 0%, RTA = 31.63 ms [15:28:55] (03PS3) 10Tryvix1509: Lift IP cap for edit-a-thon [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361187 (https://phabricator.wikimedia.org/T443033) [15:29:30] 10SRE-SLO, 10Citoid, 06Editing-team, 07Sustainability (Incident Followup): Create probe for citoid that tests whether requests can access the outside internet - https://phabricator.wikimedia.org/T381372#12502858 (10CDanis) Sorry @Mvolz , a few naive questions: * Do we have logs or other data to answer thi... [15:29:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12502860 (10phaultfinder) [15:29:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12502861 (10phaultfinder) [15:30:00] FIRING: [8x] NodeBGPSessionStatusNotEstablished: Kubernetes node aux-k8s-worker2004 has a BGP session which is not in the 'established' state. [15:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T1530) [15:30:27] (03PS1) 10Dpogorzelski: kserve-llmisvc-resources: add a Gateway for LLMInferenceService routes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361478 (https://phabricator.wikimedia.org/T436650) [15:30:32] RECOVERY - Host dse-k8s-ctrl2001 is UP: PING OK - Packet loss = 0%, RTA = 31.61 ms [15:30:34] RECOVERY - Host dse-k8s-etcd2001 is UP: PING OK - Packet loss = 0%, RTA = 30.53 ms [15:30:46] RECOVERY - Host ldap-replica2007 is UP: PING OK - Packet loss = 0%, RTA = 31.94 ms [15:30:50] RESOLVED: ProbeDown: Service ganeti2037:1811 has failed probes (tcp_ganeti_noded_ip4) - https://wikitech.wikimedia.org/wiki/Ganeti - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [15:30:58] RECOVERY - Host ml-etcd2002 is UP: PING OK - Packet loss = 0%, RTA = 30.91 ms [15:31:05] FIRING: [2x] KubernetesCalicoDown: aux-k8s-worker2004.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:31:10] !log aqu@deploy2003 Finished deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6] (duration: 06m 29s) [15:31:19] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12502876 (10Jhancock.wm) i fear we're gonna have to move some servers out of this rack to keep it from alerting. trying to figure out what's the easiest. [15:31:24] (03CR) 10Lucas Werkmeister (WMDE): Lift IP cap for edit-a-thon (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361187 (https://phabricator.wikimedia.org/T443033) (owner: 10Tryvix1509) [15:31:30] RECOVERY - Host aux-k8s-worker2004 is UP: PING OK - Packet loss = 0%, RTA = 32.10 ms [15:31:30] RECOVERY - Host schema2003 is UP: PING OK - Packet loss = 0%, RTA = 30.77 ms [15:31:30] RECOVERY - Host tcp-proxy2001 is UP: PING OK - Packet loss = 0%, RTA = 30.75 ms [15:31:37] (03Abandoned) 10Jgiannelos: noop: Force image rebuild [software/tegola] (wmf/v0.19.x) - 10https://gerrit.wikimedia.org/r/1361458 (owner: 10Jgiannelos) [15:32:15] RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:33:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:33:20] RESOLVED: [2x] KubernetesCalicoDown: aux-k8s-worker2004.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [15:33:42] RESOLVED: JobUnavailable: Reduced availability for job tcp_proxy in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:34:26] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:35:00] RESOLVED: [8x] NodeBGPSessionStatusNotEstablished: Kubernetes node aux-k8s-worker2004 has a BGP session which is not in the 'established' state. [15:35:28] (03PS1) 10Jgiannelos: maintenance: Fix python dependencies. Bump base images. [software/tegola] (wmf/v0.19.x) - 10https://gerrit.wikimedia.org/r/1361481 [15:35:57] (03Abandoned) 10Jgiannelos: maintenance: Fix python dependencies [software/tegola] (wmf/v0.19.x) - 10https://gerrit.wikimedia.org/r/1361476 (owner: 10Jgiannelos) [15:36:06] (03Abandoned) 10Jgiannelos: maintenance: Fix python dependencies. Bump base images. [software/tegola] (wmf/v0.19.x) - 10https://gerrit.wikimedia.org/r/1361481 (owner: 10Jgiannelos) [15:36:39] (03CR) 10Jelto: [C:03+2] deployment_server::deploy_service::service_catalog: add wdqs-next [puppet] - 10https://gerrit.wikimedia.org/r/1361459 (https://phabricator.wikimedia.org/T412941) (owner: 10Jelto) [15:37:00] (03CR) 10JavierMonton: [C:03+1] [eventstreams] Bump to v0.22.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360503 (https://phabricator.wikimedia.org/T383977) (owner: 10TChin) [15:37:32] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97073 and previous config saved to /var/cache/conftool/dbconfig/20261102-153731-ladsgroup.json [15:37:57] (03CR) 10CDanis: [C:03+2] Update commit message validator dep [puppet] - 10https://gerrit.wikimedia.org/r/1325468 (https://phabricator.wikimedia.org/T427387) (owner: 10Cparle) [15:38:04] (03PS1) 10Jgiannelos: maintenance: Fix python dependencies. Bump base images. [software/tegola] (wmf/v0.19.x) - 10https://gerrit.wikimedia.org/r/1361483 [15:38:06] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [15:38:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:38:27] 10ops-eqiad, 06DC-Ops: upload and install the latest firmware on db1217 - https://phabricator.wikimedia.org/T443244#12502946 (10Aklapper) [15:38:33] 10SRE-SLO, 10Citoid, 06Editing-team, 07Sustainability (Incident Followup): Create probe for citoid that tests whether requests can access the outside internet - https://phabricator.wikimedia.org/T381372#12502948 (10Mvolz) >>! In T381372#12502858, @CDanis wrote: > Sorry @Mvolz , a few naive questions: > >... [15:38:40] RESOLVED: KubernetesRsyslogDown: rsyslog on dse-k8s-worker2006:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=dse-k8s-worker2006 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [15:39:16] (03CR) 10Jelto: [C:03+1] "lgtm" [cookbooks] - 10https://gerrit.wikimedia.org/r/1361391 (https://phabricator.wikimedia.org/T440696) (owner: 10JMeybohm) [15:39:32] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12502952 (10Papaul) @Marostegui @wiki_willy i looked more into the issue after this server just crash and there are 2 possible solutions i am seeing here . Even after the firmware were updated on the other serv... [15:40:45] (03CR) 10Jelto: [C:03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/1359987 (https://phabricator.wikimedia.org/T439046) (owner: 10Dzahn) [15:41:41] (03CR) 10Jgiannelos: [C:03+2] maintenance: Fix python dependencies. Bump base images. [software/tegola] (wmf/v0.19.x) - 10https://gerrit.wikimedia.org/r/1361483 (owner: 10Jgiannelos) [15:42:50] (03Merged) 10jenkins-bot: maintenance: Fix python dependencies. Bump base images. [software/tegola] (wmf/v0.19.x) - 10https://gerrit.wikimedia.org/r/1361483 (owner: 10Jgiannelos) [15:43:28] (03CR) 10CDanis: [C:03+1] Add a basic completion script for cookbooks [software/spicerack] - 10https://gerrit.wikimedia.org/r/1359941 (https://phabricator.wikimedia.org/T367230) (owner: 10Lizimedia) [15:45:21] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps, 10ServiceOps-Upgrades-Hardware: Q1:rack/setup/install mc20[56-73] - https://phabricator.wikimedia.org/T436272#12503009 (10Jhancock.wm) @jijiki we can't do some of the racks you listed cause of power constraints, and one is a dedicated cloud rack. this was my or... [15:46:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:47:26] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12503016 (10CWilliams-WMF) @Papaul interesting, we should probably check across all of the DB nodes, so I will create a ticket for our queue - it looks like something that can be easily done against the iDRAC. [15:47:34] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1223 (T441633)', diff saved to https://phabricator.wikimedia.org/P97074 and previous config saved to /var/cache/conftool/dbconfig/20261102-154733-ladsgroup.json [15:47:52] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance [15:47:57] PROBLEM - Host es2036 #page is DOWN: PING CRITICAL - Packet loss = 100% [15:48:10] !ack [15:48:10] 8722 (ACKED) Host es2036 (paged) [15:48:21] (03CR) 10TChin: [C:03+2] [eventstreams] Bump to v0.22.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360503 (https://phabricator.wikimedia.org/T383977) (owner: 10TChin) [15:48:43] (03CR) 10CDanis: "Is this really a noop on prod?" [puppet] - 10https://gerrit.wikimedia.org/r/1357239 (https://phabricator.wikimedia.org/T441785) (owner: 10Majavah) [15:49:18] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: Check server profile of DB instances - https://phabricator.wikimedia.org/T443264 (10CWilliams-WMF) 03NEW [15:49:24] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12503051 (10Papaul) @CWilliams-WMF We will have to reboot the node after the change [15:49:58] (03Merged) 10jenkins-bot: [eventstreams] Bump to v0.22.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360503 (https://phabricator.wikimedia.org/T383977) (owner: 10TChin) [15:50:06] !log volans@cumin1004 START - Cookbook sre.mysql.depool depool es2036: Host crashed [15:50:18] !log volans@cumin1004 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2036: Host crashed [15:51:07] (03PS1) 10A-pizzata: eventgate: Add possible_event_dt_fields and new version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361488 (https://phabricator.wikimedia.org/T440968) [15:51:15] FIRING: [8x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [15:51:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [15:51:17] !log tchin@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply [15:51:49] !log tchin@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply [15:52:53] (03CR) 10Ottomata: [C:03+1] eventgate: Add possible_event_dt_fields and new version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361488 (https://phabricator.wikimedia.org/T440968) (owner: 10A-pizzata) [15:53:34] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1265.eqiad.wmnet with reason: Maintenance [15:54:19] !log tchin@deploy2003 helmfile [staging] START helmfile.d/services/eventstreams: apply [15:54:46] !log tchin@deploy2003 helmfile [staging] DONE helmfile.d/services/eventstreams: apply [15:54:59] volans: does es2036 need a look, I see that you have depooled it? [15:55:30] cezmunsta: it crashed, I was doing some basic check of HW logs and opening a task before pinging you [15:55:40] most likely yes will need a look once back up [15:55:52] ack [15:56:04] !log tchin@deploy2003 helmfile [eqiad] START helmfile.d/services/eventstreams: apply [15:56:15] FIRING: [8x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [15:56:38] !log tchin@deploy2003 helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply [15:57:11] !log tchin@deploy2003 helmfile [codfw] START helmfile.d/services/eventstreams: apply [15:57:18] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Maintenance [15:57:20] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1272 (T441633)', diff saved to https://phabricator.wikimedia.org/P97076 and previous config saved to /var/cache/conftool/dbconfig/20261102-155719-ladsgroup.json [15:57:40] !log tchin@deploy2003 helmfile [codfw] DONE helmfile.d/services/eventstreams: apply [16:00:06] (03CR) 10Ottomata: [C:03+2] eventgate: Add possible_event_dt_fields and new version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361488 (https://phabricator.wikimedia.org/T440968) (owner: 10A-pizzata) [16:00:10] cezmunsta: https://phabricator.wikimedia.org/T443268 host is booting now [16:00:17] +1 [16:00:45] RECOVERY - Host es2036 #page is UP: PING OK - Packet loss = 0%, RTA = 31.92 ms [16:00:47] (03PS1) 10JavierMonton: ssh-key: new yubikey [puppet] - 10https://gerrit.wikimedia.org/r/1361491 [16:01:28] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1272 (T441633)', diff saved to https://phabricator.wikimedia.org/P97077 and previous config saved to /var/cache/conftool/dbconfig/20261102-160127-ladsgroup.json [16:01:58] (03Merged) 10jenkins-bot: eventgate: Add possible_event_dt_fields and new version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361488 (https://phabricator.wikimedia.org/T440968) (owner: 10A-pizzata) [16:02:09] (03CR) 10CI reject: [V:04-1] ssh-key: new yubikey [puppet] - 10https://gerrit.wikimedia.org/r/1361491 (owner: 10JavierMonton) [16:02:14] RECOVERY - haproxy failover on dbproxy1022 is OK: OK check_failover servers up 2 down 0: https://wikitech.wikimedia.org/wiki/HAProxy [16:02:18] RECOVERY - haproxy failover on dbproxy1024 is OK: OK check_failover servers up 2 down 0: https://wikitech.wikimedia.org/wiki/HAProxy [16:02:24] PROBLEM - MariaDB Event Scheduler es6 on es2036 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [16:02:29] PROBLEM - MariaDB Replica IO: es6 #page on es2036 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:02:29] PROBLEM - MariaDB read only es6 on es2036 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [16:02:32] PROBLEM - MariaDB Events es6 on es2036 is CRITICAL: CRITICAL - Failed to query events: ERROR 2002 (HY000): Cant connect to local server through socket /run/mysqld/mysqld.sock (2) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [16:02:44] RECOVERY - MariaDB read only m1 on db1217 is OK: Version 10.11.16-MariaDB-log, Uptime 52s, read_only: True, event_scheduler: True, 25.17 QPS, connection latency: 0.009035s, query latency: 0.000288s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [16:02:45] !ack [16:02:45] 8723 (ACKED) es2036 (paged)/MariaDB Replica IO: es6 (paged) [16:03:12] RECOVERY - haproxy failover on dbproxy1025 is OK: OK check_failover servers up 2 down 0: https://wikitech.wikimedia.org/wiki/HAProxy [16:03:20] RECOVERY - haproxy failover on dbproxy1026 is OK: OK check_failover servers up 2 down 0: https://wikitech.wikimedia.org/wiki/HAProxy [16:03:26] RECOVERY - haproxy failover on dbproxy1023 is OK: OK check_failover servers up 2 down 0: https://wikitech.wikimedia.org/wiki/HAProxy [16:03:40] RECOVERY - MariaDB read only m2 on db1217 is OK: Version 10.11.16-MariaDB-log, Uptime 54s, read_only: True, event_scheduler: True, 14.43 QPS, connection latency: 0.017665s, query latency: 0.000697s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [16:03:42] RECOVERY - mysqld processes on db1217 is OK: PROCS OK: 4 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [16:03:43] PROBLEM - mysqld processes #page on es2036 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [16:03:43] PROBLEM - MariaDB Replica SQL: es6 #page on es2036 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:03:48] RECOVERY - MariaDB read only m5 on db1217 is OK: Version 10.11.16-MariaDB-log, Uptime 16s, read_only: True, event_scheduler: True, 25.22 QPS, connection latency: 0.009598s, query latency: 0.000347s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [16:03:50] RECOVERY - MariaDB read only m3 on db1217 is OK: Version 10.11.16-MariaDB-log, Uptime 37s, read_only: True, event_scheduler: True, 13.50 QPS, connection latency: 0.008463s, query latency: 0.000319s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [16:04:01] !ack [16:04:02] 8724 (ACKED) es2036 (paged)/mysqld processes (paged) [16:04:02] 8725 (ACKED) es2036 (paged)/MariaDB Replica SQL: es6 (paged) [16:04:04] let me downtime it too [16:04:12] RECOVERY - haproxy failover on dbproxy1028 is OK: OK check_failover servers up 2 down 0: https://wikitech.wikimedia.org/wiki/HAProxy [16:04:16] RECOVERY - haproxy failover on dbproxy1027 is OK: OK check_failover servers up 2 down 0: https://wikitech.wikimedia.org/wiki/HAProxy [16:04:26] RECOVERY - haproxy failover on dbproxy1029 is OK: OK check_failover servers up 2 down 0: https://wikitech.wikimedia.org/wiki/HAProxy [16:04:55] !log volans@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2036.codfw.wmnet with reason: host crashed [16:05:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [16:05:42] RECOVERY - MariaDB Replica IO: m5 on db1217 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:05:42] RECOVERY - MariaDB Replica SQL: m1 on db1217 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:05:46] RECOVERY - MariaDB Replica IO: m3 on db1217 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:05:46] RECOVERY - MariaDB Replica IO: m2 on db1217 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:05:48] RECOVERY - MariaDB Replica IO: m1 on db1217 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:06:15] FIRING: [8x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [16:06:28] RECOVERY - MariaDB Replica SQL: m2 on db1217 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:06:46] RECOVERY - MariaDB Replica Lag: m5 on db1217 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:06:46] RECOVERY - MariaDB Replica SQL: m5 on db1217 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:06:48] RECOVERY - MariaDB Replica SQL: m3 on db1217 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:08:45] FIRING: [3x] CirrusConsumerRerenderFetchErrorRate: cirrus_streaming_updater_consumer_cloudelastic_eqiad in eqiad (k8s): fetch error (rerenders) rate too high - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusConsumerRerenderFetchErrorRate [16:08:48] RECOVERY - MariaDB Replica Lag: m2 on db1217 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:09:53] 06SRE, 07SecTeam-Processed: Build/deploy new Flink OCI (docker) image - https://phabricator.wikimedia.org/T443032#12503229 (10sbassett) [16:11:30] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97078 and previous config saved to /var/cache/conftool/dbconfig/20261102-161129-ladsgroup.json [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:12:36] (03PS2) 10JavierMonton: ssh-key: new yubikey [puppet] - 10https://gerrit.wikimedia.org/r/1361491 [16:13:15] FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [16:14:14] 06SRE, 10SRE-Access-Requests, 06Product Safety and Integrity, 06Security-Team, 07SecTeam-Processed: Request IRC #mediawiki_security access for Novem Linguae - https://phabricator.wikimedia.org/T442747#12503296 (10sbassett) Ping @Rsilvola for the ostensible approval (or not) for this request. Though PSI... [16:14:32] 06SRE, 10SRE-Access-Requests, 06Product Safety and Integrity, 06Security-Team, 07SecTeam-Processed: Request IRC #mediawiki_security access for Novem Linguae - https://phabricator.wikimedia.org/T442747#12503297 (10sbassett) [16:15:41] (03CR) 10BCornwall: [V:03+1 C:03+2] varnish: Fix thumb.wm.o normalization to match upload.wm.o [puppet] - 10https://gerrit.wikimedia.org/r/1341936 (https://phabricator.wikimedia.org/T425216) (owner: 10Krinkle) [16:15:46] RECOVERY - MariaDB Replica Lag: m3 on db1217 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:15:52] (03PS1) 10Muehlenhoff: Remove Luca from Icinga contact group [puppet] - 10https://gerrit.wikimedia.org/r/1361495 (https://phabricator.wikimedia.org/T443187) [16:16:06] jouncebot: nowandnext [16:16:06] No deployments scheduled for the next 0 hour(s) and 13 minute(s) [16:16:06] In 0 hour(s) and 13 minute(s): Wikimedia Portals Update (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T1630) [16:16:42] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:18:15] RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate [16:18:22] !log cmooney@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1253.eqiad.wmnet [16:18:30] (03CR) 10BCornwall: [V:03+1 C:03+2] "PCC SUCCESS (CORE_DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9587/co" [puppet] - 10https://gerrit.wikimedia.org/r/1341936 (https://phabricator.wikimedia.org/T425216) (owner: 10Krinkle) [16:18:32] RECOVERY - MariaDB Events es6 on es2036 is OK: OK - All 4 events in ops database are ENABLED https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [16:18:43] RECOVERY - mysqld processes #page on es2036 is OK: PROCS OK: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [16:19:10] !log cmooney@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1253.eqiad.wmnet [16:19:17] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12503324 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by cmooney@cumin1004 depool for host wikikube-worker12... [16:19:19] (03CR) 10BCornwall: [V:03+2 C:03+2] varnish: Refactor normalize-thumbnail-url.vtc [puppet] - 10https://gerrit.wikimedia.org/r/1333533 (https://phabricator.wikimedia.org/T425216) (owner: 10Krinkle) [16:19:26] RECOVERY - MariaDB Event Scheduler es6 on es2036 is OK: Version 10.11.16-MariaDB-log, Uptime 87s, read_only: True, event_scheduler: True, 12.22 QPS, connection latency: 0.017279s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [16:19:30] RECOVERY - MariaDB read only es6 on es2036 is OK: Version 10.11.16-MariaDB-log, Uptime 91s, read_only: True, event_scheduler: True, 21.62 QPS, connection latency: 0.021505s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [16:20:31] RECOVERY - MariaDB Replica IO: es6 #page on es2036 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:21:33] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97079 and previous config saved to /var/cache/conftool/dbconfig/20261102-162132-ladsgroup.json [16:21:42] RESOLVED: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:22:37] (03CR) 10Brennen Bearnes: [C:03+1] Remove legacy key for brennen [puppet] - 10https://gerrit.wikimedia.org/r/1361461 (https://phabricator.wikimedia.org/T442761) (owner: 10Muehlenhoff) [16:22:44] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for kemayo - https://phabricator.wikimedia.org/T442902#12503358 (10DLynch) @Dzahn All seems to work, you can remove the old key when you want. [16:24:05] (03PS1) 10Muehlenhoff: Add FIDO keys for ssastry and mlitn [puppet] - 10https://gerrit.wikimedia.org/r/1361497 (https://phabricator.wikimedia.org/T442758) [16:24:43] RECOVERY - MariaDB Replica SQL: es6 #page on es2036 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:27:15] RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at codfw: 24.44% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [16:27:37] (03PS1) 10Majavah: P:wmcs::instance: Remove a root key [puppet] - 10https://gerrit.wikimedia.org/r/1361502 (https://phabricator.wikimedia.org/T443002) [16:28:56] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy key for brennen [puppet] - 10https://gerrit.wikimedia.org/r/1361461 (https://phabricator.wikimedia.org/T442761) (owner: 10Muehlenhoff) [16:30:05] jan_drewniak: Time to snap out of that daydream and deploy Wikimedia Portals Update. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T1630). [16:30:28] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO keys for ssastry and mlitn [puppet] - 10https://gerrit.wikimedia.org/r/1361497 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [16:30:49] (03CR) 10Majavah: [C:03+2] P:wmcs::instance: Remove a root key [puppet] - 10https://gerrit.wikimedia.org/r/1361502 (https://phabricator.wikimedia.org/T443002) (owner: 10Majavah) [16:31:01] brett: ok to merge your two varnish patches along? [16:31:14] moritzm: if it prompts, mine is ok to merge as well [16:31:17] moritzm: Go for it, thanks! Was waiting for it to free up :) [16:31:35] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1272 (T441633)', diff saved to https://phabricator.wikimedia.org/P97080 and previous config saved to /var/cache/conftool/dbconfig/20261102-163134-ladsgroup.json [16:31:52] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1280.eqiad.wmnet with reason: Maintenance [16:31:55] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1280 (T441633)', diff saved to https://phabricator.wikimedia.org/P97081 and previous config saved to /var/cache/conftool/dbconfig/20261102-163154-ladsgroup.json [16:31:56] ok, merging [16:32:06] taavi: your's wasn't in the list yet [16:32:22] ack, i'll merge it separately after yours is done then [16:32:32] RECOVERY - MariaDB Replica Lag: m1 on db1217 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [16:32:43] !log cmooney@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet [16:32:45] !log cmooney@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet [16:32:52] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12503420 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by cmooney@cumin1004 pool for host wikikube-worker1253... [16:32:55] (03CR) 10Tiziano Fogli: [C:03+2] tcpircbot: replace newlines with spaces in PRIVMSG (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1342704 (https://phabricator.wikimedia.org/T438269) (owner: 10Tiziano Fogli) [16:33:06] welp [16:33:23] tappof: ok to merge yours? [16:33:33] wow, everybody's joining the party [16:33:34] taavi: yes, thx [16:33:45] doing [16:35:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [16:35:33] (03PS1) 10Ottomata: otto - add yubikey ssh public key; also remove unused key [puppet] - 10https://gerrit.wikimedia.org/r/1361503 (https://phabricator.wikimedia.org/T442758) [16:36:03] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1280 (T441633)', diff saved to https://phabricator.wikimedia.org/P97082 and previous config saved to /var/cache/conftool/dbconfig/20261102-163602-ladsgroup.json [16:39:55] (03CR) 10Tiziano Fogli: [C:03+2] icinga-sunsetting: Fix typo in --debug help text [software] - 10https://gerrit.wikimedia.org/r/1356661 (https://phabricator.wikimedia.org/T441466) (owner: 10Esof-1974) [16:40:11] (03PS2) 10AikoChou: admin: Add FIDO SSH key for aikochou [puppet] - 10https://gerrit.wikimedia.org/r/1358720 (https://phabricator.wikimedia.org/T442367) [16:40:59] (03Merged) 10jenkins-bot: icinga-sunsetting: Fix typo in --debug help text [software] - 10https://gerrit.wikimedia.org/r/1356661 (https://phabricator.wikimedia.org/T441466) (owner: 10Esof-1974) [16:41:55] (03CR) 10Muehlenhoff: [C:03+1] "Verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1361491 (owner: 10JavierMonton) [16:42:00] (03CR) 10Muehlenhoff: [C:03+2] ssh-key: new yubikey [puppet] - 10https://gerrit.wikimedia.org/r/1361491 (owner: 10JavierMonton) [16:42:18] cezmunsta: the 3 open incidents for db1262 from few days ago can be resolved?? [16:43:29] volans: I presume so, yes [16:43:34] thanks! [16:46:06] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97083 and previous config saved to /var/cache/conftool/dbconfig/20261102-164605-ladsgroup.json [16:48:15] 06SRE, 10MediaWiki-General, 06ServiceOps, 07Epic: Introduce a global load shedding level - https://phabricator.wikimedia.org/T443236#12503592 (10Bugreporter) > stop all bots To prevent tools and scripts who does not respect maxlag (e.g. in Wikidata) we may also rate limit edits to 8/minute for every users... [16:49:39] 06SRE, 06DBA, 10MediaWiki-Action-API, 06Traffic: internal_api_error_DBConnectionError on some en/ja/pl/zh projects - https://phabricator.wikimedia.org/T443274#12503620 (10AlexisJazz) [16:53:23] (03CR) 10Majavah: "This is removing an absented file with the comment of `TODO: remove once absented`. So not a NOOP, but harmless cleanup that should happen" [puppet] - 10https://gerrit.wikimedia.org/r/1357239 (https://phabricator.wikimedia.org/T441785) (owner: 10Majavah) [16:53:29] 06SRE, 06DBA, 10MediaWiki-Action-API, 06Traffic: internal_api_error_DBConnectionError on some en/ja/pl/zh projects - https://phabricator.wikimedia.org/T443274#12503655 (10DamianZaremba) Cluebot III on enwiki has also been seeing this intermittently. First instance was (during a page edit call); ` 2026-11-... [16:53:45] FIRING: [3x] CirrusConsumerRerenderFetchErrorRate: cirrus_streaming_updater_consumer_cloudelastic_eqiad in eqiad (k8s): fetch error (rerenders) rate too high - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusConsumerRerenderFetchErrorRate [16:56:03] (03PS1) 10Ladsgroup: Temporarily increase circuit breaking [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361508 (https://phabricator.wikimedia.org/T443274) [16:56:08] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97084 and previous config saved to /var/cache/conftool/dbconfig/20261102-165607-ladsgroup.json [16:56:15] (03CR) 10Filippo Giunchedi: [C:03+2] wmcs: add cloudvps root FIDO keys for filippo [puppet] - 10https://gerrit.wikimedia.org/r/1361321 (https://phabricator.wikimedia.org/T443002) (owner: 10Filippo Giunchedi) [16:57:46] 06SRE, 06DBA, 10MediaWiki-Action-API, 06Traffic, 13Patch-For-Review: internal_api_error_DBConnectionError on some en/ja/pl/zh projects - https://phabricator.wikimedia.org/T443274#12503700 (10Ladsgroup) We are on one dc and one of the three replicas for the ES has gone down due to hardware issues leaving... [16:58:00] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361508 (https://phabricator.wikimedia.org/T443274) (owner: 10Ladsgroup) [16:58:45] RESOLVED: [2x] CirrusConsumerRerenderFetchErrorRate: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s): fetch error (rerenders) rate too high - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusConsumerRerenderFetchErrorRate [16:58:57] (03Merged) 10jenkins-bot: Temporarily increase circuit breaking [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361508 (https://phabricator.wikimedia.org/T443274) (owner: 10Ladsgroup) [16:59:14] !log ladsgroup@deploy2003 Started scap sync-world: Backport for [[gerrit:1361508|Temporarily increase circuit breaking (T443274)]] [16:59:20] T443274: internal_api_error_DBConnectionError on some en/ja/pl/zh projects - https://phabricator.wikimedia.org/T443274 [16:59:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12503713 (10phaultfinder) [17:00:05] thcipriani: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for Release Engineering Clinic Duty . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T1700). [17:00:28] oh good. I didn't update the calendar. [17:01:15] FIRING: [8x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [17:02:07] looks like Release Engineering Clinic Duty this week is dduvall (but it's me on the calendar because I didn't update the secondary asignee) [17:03:29] !log ladsgroup@deploy2003 ladsgroup: Backport for [[gerrit:1361508|Temporarily increase circuit breaking (T443274)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [17:06:07] !log ladsgroup@deploy2003 ladsgroup: Continuing with deployment [17:06:10] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1280 (T441633)', diff saved to https://phabricator.wikimedia.org/P97085 and previous config saved to /var/cache/conftool/dbconfig/20261102-170609-ladsgroup.json [17:06:15] FIRING: [7x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [17:06:28] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1289.eqiad.wmnet with reason: Maintenance [17:06:30] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1289 (T441633)', diff saved to https://phabricator.wikimedia.org/P97086 and previous config saved to /var/cache/conftool/dbconfig/20261102-170629-ladsgroup.json [17:07:27] !log cwilliams@cumin1004 START - Cookbook sre.mysql.pool pool db1228: Maintenance [17:07:39] !log cwilliams@cumin1004 END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1228: Maintenance [17:09:26] (03PS1) 10CWilliams: db1228: enabling notifications [puppet] - 10https://gerrit.wikimedia.org/r/1361511 (https://phabricator.wikimedia.org/T442815) [17:09:47] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12503833 (10phaultfinder) [17:10:38] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1289 (T441633)', diff saved to https://phabricator.wikimedia.org/P97087 and previous config saved to /var/cache/conftool/dbconfig/20261102-171037-ladsgroup.json [17:10:52] (03CR) 10CWilliams: [C:03+2] db1228: enabling notifications [puppet] - 10https://gerrit.wikimedia.org/r/1361511 (https://phabricator.wikimedia.org/T442815) (owner: 10CWilliams) [17:11:15] FIRING: [8x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [17:12:53] !log ladsgroup@deploy2003 Finished scap sync-world: Backport for [[gerrit:1361508|Temporarily increase circuit breaking (T443274)]] (duration: 13m 39s) [17:12:59] T443274: internal_api_error_DBConnectionError on some en/ja/pl/zh projects - https://phabricator.wikimedia.org/T443274 [17:15:09] something seems to be happening; is the main phab issue for this T443274 or something else (there seems to be a ton of duplicates, so I'm not sure what's the original) [17:16:03] mskmskmsk: I think so yes given what I'm seeing is circuit breaking, and Amir1 linked his patch to that task [17:16:15] RESOLVED: [8x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [17:16:37] mskmskmsk: something happening in a good way or bad way? :D [17:16:51] the patch should be deployed by now [17:17:02] I'm working on a better fix (should have done it long time ago) [17:17:41] Amir1: first signs are positive, error rate is going down on mw-api-ext [17:17:50] (03PS1) 10Atsuko: Revert "trafficserver: increase timeouts for airflow-dumps" [puppet] - 10https://gerrit.wikimedia.org/r/1361512 [17:18:03] (03CR) 10Atsuko: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1361512 (owner: 10Atsuko) [17:18:50] Amir1: not sure; the DB error would only appear on very large pages (e.g. WP:FAC) but seems to have gone away now [17:19:28] cool cool [17:19:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12503934 (10phaultfinder) [17:19:48] claime: how is your mediawiki? wanna merge https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1361513 ? [17:20:22] Amir1: wdym my mediawiki? if you mean my ability to review a patch to mediawiki core, bad :P [17:20:30] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12503938 (10cmooney) @JMeybohm if you get adventurous I recorded moving one of them just now: https://drive.google.com/file/d/1cUe7VoRNQF... [17:20:40] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97088 and previous config saved to /var/cache/conftool/dbconfig/20261102-172039-ladsgroup.json [17:20:58] "bad" is a social construct [17:21:56] i.e. "bad" is invention of big tech firms to sell us more AI [17:23:59] (03PS1) 10Jgiannelos: tegola: Bump latest image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361514 [17:28:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12503965 (10Jhancock.wm) 11:13am [17:30:42] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97089 and previous config saved to /var/cache/conftool/dbconfig/20261102-173041-ladsgroup.json [17:30:47] 10ops-codfw, 06SRE, 06DC-Ops, 06ServiceOps: hw troubleshooting: power-on for wikikube-worker2337.codfw.wmnet - https://phabricator.wikimedia.org/T442714#12503974 (10Jhancock.wm) server found with power to the psus, but does not boot when physical power button is pressed. unplugged the power cables and drai... [17:32:38] !log otto@deploy2003 Started scap deploy-service eventgate-analytics: Time to persist metric - T440968 [17:32:38] !log otto@deploy2003 scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics [17:32:45] T440968: mediawiki.page_change.v1 SLO - Develop the freshness metric - https://phabricator.wikimedia.org/T440968 [17:34:12] (03PS1) 10Ladsgroup: rdbms: Allow overriding of load monitor per lb cluster [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361519 (https://phabricator.wikimedia.org/T443274) [17:34:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12504006 (10phaultfinder) [17:36:35] 06SRE, 10SRE-Access-Requests, 06Data-Engineering, 06Data-Engineering-Radar: Requesting access to internal data platform for cfoster - https://phabricator.wikimedia.org/T442648#12504018 (10mpopov) Approving @cfoster's membership in `analytics-product-users` Also should be added as primary approver with me... [17:37:20] (03PS1) 10Ladsgroup: Adjust circuit breaking thresholds [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361521 (https://phabricator.wikimedia.org/T443274) [17:39:03] (03CR) 10Jgiannelos: [C:03+2] tegola: Bump latest image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361514 (owner: 10Jgiannelos) [17:40:28] (03Merged) 10jenkins-bot: tegola: Bump latest image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361514 (owner: 10Jgiannelos) [17:40:44] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1289 (T441633)', diff saved to https://phabricator.wikimedia.org/P97090 and previous config saved to /var/cache/conftool/dbconfig/20261102-174043-ladsgroup.json [17:41:02] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [17:43:19] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361521 (https://phabricator.wikimedia.org/T443274) (owner: 10Ladsgroup) [17:43:19] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy2003 using scap backport" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361519 (https://phabricator.wikimedia.org/T443274) (owner: 10Ladsgroup) [17:44:10] (03Merged) 10jenkins-bot: Adjust circuit breaking thresholds [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361521 (https://phabricator.wikimedia.org/T443274) (owner: 10Ladsgroup) [17:44:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12504051 (10phaultfinder) [17:44:47] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 2 others: Move eligible eqiad row A and B k8s worker nodes to 10G links - https://phabricator.wikimedia.org/T443064#12504052 (10cmooney) [17:45:26] !log Investigating spiderpig [17:45:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:46:20] !log jgiannelos@deploy1003 helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply [17:46:41] !log jgiannelos@deploy1003 helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply [17:47:23] (03Merged) 10jenkins-bot: rdbms: Allow overriding of load monitor per lb cluster [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361519 (https://phabricator.wikimedia.org/T443274) (owner: 10Ladsgroup) [17:47:42] !log ladsgroup@deploy2003 Started scap sync-world: Backport for [[gerrit:1361521|Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519|rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) [17:47:48] T443274: internal_api_error_DBConnectionError on some en/ja/pl/zh projects - https://phabricator.wikimedia.org/T443274 [17:48:14] !log otto@deploy2003 helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply [17:49:03] !log otto@deploy2003 helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply [17:49:21] !log otto@deploy2003 helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply [17:49:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12504073 (10phaultfinder) [17:50:07] !log otto@deploy2003 helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply [17:50:24] Restarted spiderpig [17:51:47] !log ladsgroup@deploy2003 ladsgroup: Backport for [[gerrit:1361521|Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519|rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [17:52:06] (03PS1) 10AKhatun: Prefetch the LAC revision_feature_counts artifact on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361525 (https://phabricator.wikimedia.org/T436695) [17:52:39] !log jgiannelos@deploy1003 helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply [17:53:18] !log jgiannelos@deploy1003 helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply [17:53:31] !log jgiannelos@deploy1003 helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply [17:53:39] !log ladsgroup@deploy2003 ladsgroup: Continuing with deployment [17:54:08] !log jgiannelos@deploy1003 helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply [17:54:28] (03CR) 10Btullis: [C:03+1] Revert "trafficserver: increase timeouts for airflow-dumps" [puppet] - 10https://gerrit.wikimedia.org/r/1361512 (owner: 10Atsuko) [17:54:43] (03CR) 10Atsuko: [C:03+2] Revert "trafficserver: increase timeouts for airflow-dumps" [puppet] - 10https://gerrit.wikimedia.org/r/1361512 (owner: 10Atsuko) [17:55:44] (03CR) 10Klausman: [C:03+1] kserve-llmisvc-resources: add a Gateway for LLMInferenceService routes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361478 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [17:57:46] (03PS2) 10Neriah: ukwiki: limit "user" namespace to only registered users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360437 (https://phabricator.wikimedia.org/T438016) [17:58:53] (03CR) 10Ottomata: [C:03+1] Prefetch the LAC revision_feature_counts artifact on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361525 (https://phabricator.wikimedia.org/T436695) (owner: 10AKhatun) [17:58:59] (03CR) 10Rsilvola: [C:03+2] miscweb: Deploy new security landing page [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361463 (https://phabricator.wikimedia.org/T436543) (owner: 10Rsilvola) [17:59:29] (03CR) 10Rsilvola: [V:03+2 C:03+2] miscweb: Deploy new security landing page [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361463 (https://phabricator.wikimedia.org/T436543) (owner: 10Rsilvola) [17:59:33] !log ladsgroup@deploy2003 Finished scap sync-world: Backport for [[gerrit:1361521|Adjust circuit breaking thresholds (T443274)]], [[gerrit:1361519|rdbms: Allow overriding of load monitor per lb cluster (T443274)]] (1.48.0-wmf.1) (duration: 11m 51s) [17:59:39] T443274: internal_api_error_DBConnectionError on some en/ja/pl/zh projects - https://phabricator.wikimedia.org/T443274 [17:59:43] (03CR) 10Neriah: ukwiki: limit "user" namespace to only registered users (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360437 (https://phabricator.wikimedia.org/T438016) (owner: 10Neriah) [18:00:04] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T1800) [18:00:05] ryankemper: #bothumor My software never has bugs. It just develops random features. Rise for Wikidata Query Service weekly deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T1800). [18:01:28] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Degraded RAID on an-presto1006 - https://phabricator.wikimedia.org/T441831#12504159 (10BTullis) OK, sounds good to me. I've added the drive as a global hot spare. ` btullis@an-presto1006:~$ sudo perccli64 /c0/eall/sall show | grep... [18:01:38] 06SRE, 06DBA, 10MediaWiki-Action-API, 06MediaWiki-API-Platform-Team, and 3 others: internal_api_error_DBConnectionError on some en/ja/pl/zh projects - https://phabricator.wikimedia.org/T443274#12504161 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup [18:02:05] (03PS1) 10Btullis: raid: Treat hot spare drives as healthy in the broadcom check [puppet] - 10https://gerrit.wikimedia.org/r/1361526 (https://phabricator.wikimedia.org/T441831) [18:02:43] 10ops-codfw, 06DC-Ops, 06ServiceOps: Degraded RAID on wikikube-worker2285 - https://phabricator.wikimedia.org/T441252#12504184 (10Jhancock.wm) @JMeybohm i was on vacation. drive has been swapped. but i didn't see it come up. lmk if you see it. if not i'll need to power down the server and reseat the interna... [18:02:50] !log jgiannelos@deploy1003 helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply [18:02:55] !log jgiannelos@deploy1003 helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply [18:02:59] !log jgiannelos@deploy1003 helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply [18:03:03] !log jgiannelos@deploy1003 helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply [18:03:08] !log jgiannelos@deploy1003 helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply [18:03:11] !log jgiannelos@deploy1003 helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply [18:03:17] o/ [18:03:26] Amir1: are you still scapping? [18:03:41] nope [18:03:52] sweet, grabbing the infra window for some envoy stuff [18:03:56] I read it as "scrapping" at first [18:04:06] well, that I assume you're always doing [18:04:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12504200 (10phaultfinder) [18:04:56] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12504201 (10phaultfinder) [18:05:14] (03PS2) 10RLazarus: mw-*: Update all deployments to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360005 [18:05:18] xD [18:07:46] (03CR) 10RLazarus: [C:03+2] mw-*: Update all deployments to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360005 (owner: 10RLazarus) [18:07:56] rzl: let @akhatun and i know when you are done, we were about to do a mw config [18:08:19] okay :) I might need this whole window but I'll let you know [18:08:27] okay, or, perhaps it is just safe to proceed? [18:08:28] if your thing is an emergency I can back out though! [18:08:28] mw config only? [18:08:32] no, do not please [18:08:34] not emergency [18:08:35] okay [18:08:54] after this deploy we should be okay to coexist but I'll let you know [18:08:58] k ty [18:09:32] (03Merged) 10jenkins-bot: mw-*: Update all deployments to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360005 (owner: 10RLazarus) [18:11:51] !log rzl@deploy2003 Started scap sync-world: https://gerrit.wikimedia.org/r/1360005 T436338 [18:15:09] (03PS2) 10RLazarus: mw-videoscaler: Update to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360003 (https://phabricator.wikimedia.org/T436338) [18:16:28] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Degraded RAID controller on an-presto1015 - https://phabricator.wikimedia.org/T442340#12504296 (10BTullis) a:05Jclark-ctr→03BTullis I'll reopen this and assign it to myself, as unfortunately rebuilding the array is quite a... [18:18:43] !log rzl@deploy2003 Finished scap sync-world: https://gerrit.wikimedia.org/r/1360005 T436338 (duration: 07m 48s) [18:18:53] (03CR) 10RLazarus: [C:03+2] mw-videoscaler: Update to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360003 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [18:18:58] (03CR) 10RLazarus: [C:03+2] rest-gateway: Update to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360002 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [18:19:46] ottomata, akhatun: I'm going to do a couple more deployments in this window but then should wrap it up early, I'll still let you know [18:20:18] (03Merged) 10jenkins-bot: mw-videoscaler: Update to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360003 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [18:20:32] (03Merged) 10jenkins-bot: rest-gateway: Update to Envoy 1.40.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360002 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [18:21:01] 10ops-codfw, 06DC-Ops, 06ServiceOps: Degraded RAID on wikikube-worker2285 - https://phabricator.wikimedia.org/T441252#12504330 (10JMeybohm) 05Open→03Resolved a:03JMeybohm There was some mdadm superblock on the "new" device which got detected so the new disk would not pop up as usual in the OS: ` r... [18:22:04] (03PS1) 10PipelineBot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361531 [18:23:40] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [18:24:00] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [18:24:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [18:24:54] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/rest-gateway: apply [18:25:11] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [18:26:45] (03PS1) 10Jforrester: Abstractwiki: Define virtual-abstractwiki DB pointer [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361533 (https://phabricator.wikimedia.org/T443295) [18:29:13] !log rzl@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply [18:29:33] !log rzl@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply [18:29:52] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Degraded RAID controller on an-presto1015 - https://phabricator.wikimedia.org/T442340#12504401 (10BTullis) OK, we can see that the `/srv` volume has gone read-only too. ` Oct 26 00:00:11 an-presto1015 logrotate[2275634]: error... [18:30:14] !log btullis@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-presto1015.eqiad.wmnet with reason: Recreate data RAID without failed disk [18:30:23] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Degraded RAID controller on an-presto1015 - https://phabricator.wikimedia.org/T442340#12504403 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=069bb161-202f-4424-8220-987b5cf53a53) set by btullis@cumin1004... [18:30:31] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Degraded RAID controller on an-presto1015 - https://phabricator.wikimedia.org/T442340#12504408 (10BTullis) 05Resolved→03Open [18:31:29] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply [18:31:42] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply [18:35:09] ottomata, akhatun: okay, infra window finished :) [18:35:27] (03CR) 10RLazarus: [C:03+2] kubernetes: Update default envoy version to 1.40.1 [puppet] - 10https://gerrit.wikimedia.org/r/1360006 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [18:35:31] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Degraded RAID controller on an-presto1015 - https://phabricator.wikimedia.org/T442340#12504490 (10BTullis) Unmounting and removing the existing device. ` btullis@an-presto1015:~$ sudo umount /srv btullis@an-presto1015:~$ sudo dms... [18:36:12] !log rsilvola@deploy2003 helmfile [codfw] START helmfile.d/services/miscweb: apply [18:36:35] !log rsilvola@deploy2003 helmfile [codfw] DONE helmfile.d/services/miscweb: apply [18:36:42] !log rsilvola@deploy2003 helmfile [eqiad] START helmfile.d/services/miscweb: apply [18:37:07] !log rsilvola@deploy2003 helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [18:39:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [18:40:12] FIRING: [2x] ProbeDown: Service people1005:30443 has failed probes (http_security_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:41:07] Starting a scap deployment from host of https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1361525 @ottomata [18:41:42] Sorry, spiderpig is back, so will use that [18:41:57] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Degraded RAID controller on an-presto1015 - https://phabricator.wikimedia.org/T442340#12504560 (10BTullis) Creating the new LV. ` btullis@an-presto1015:~$ sudo perccli64 /c0 add vd type=raid10 drives=64:0-4,64:6-10 wb ra CLI Versi... [18:42:45] (03CR) 10TrainBranchBot: [C:03+2] "Approved by akhatun@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361525 (https://phabricator.wikimedia.org/T436695) (owner: 10AKhatun) [18:43:33] (03Merged) 10jenkins-bot: Prefetch the LAC revision_feature_counts artifact on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1361525 (https://phabricator.wikimedia.org/T436695) (owner: 10AKhatun) [18:43:49] !log akhatun@deploy2003 Started scap sync-world: Backport for [[gerrit:1361525|Prefetch the LAC revision_feature_counts artifact on all wikipedias]] [18:44:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12504591 (10phaultfinder) [18:47:57] !log akhatun@deploy2003 akhatun: Backport for [[gerrit:1361525|Prefetch the LAC revision_feature_counts artifact on all wikipedias]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [18:48:41] !log akhatun@deploy2003 akhatun: Continuing with deployment [18:49:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12504609 (10phaultfinder) [18:50:52] akhatun: I have a new release of Scap to deploy when you're done. [18:51:15] (03PS1) 10Southparkfan: swift: Remove bullseye backend nodes for deployment-prep [puppet] - 10https://gerrit.wikimedia.org/r/1361535 (https://phabricator.wikimedia.org/T436476) [18:52:20] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Degraded RAID controller on an-presto1015 - https://phabricator.wikimedia.org/T442340#12504618 (10BTullis) Check that the new drive comes up as `/dev/sdb` ` btullis@an-presto1015:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOU... [18:52:46] ok [18:54:32] !log akhatun@deploy2003 Finished scap sync-world: Backport for [[gerrit:1361525|Prefetch the LAC revision_feature_counts artifact on all wikipedias]] (duration: 10m 43s) [18:54:50] @Dancy: I'm done [18:54:54] Thanks! [18:54:59] 10ops-eqiad, 06SRE, 06DC-Ops, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): Degraded RAID controller on an-presto1015 - https://phabricator.wikimedia.org/T442340#12504662 (10BTullis) 05Open→03Resolved All looks OK now. ` btullis@an-presto1015:~$ sudo /usr/local/lib/nagios/plugins/get-raid-status-b... [18:55:06] !log dancy@deploy2003 Installing scap version "4.310.0" for 2 host(s) [18:57:00] !log dancy@deploy2003 Installation of scap version "4.310.0" completed for 2 hosts [18:59:38] Done w/ scap deployment. [19:04:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442614#12504792 (10phaultfinder) [19:07:51] (03CR) 10MSantos: [C:03+1] MathML+MathJax rollout to phase 6 (remaining large Wikipedias) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311061 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [19:10:55] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 02 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311061 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [19:17:05] PROBLEM - Host mr1-eqsin.oob is DOWN: PING CRITICAL - Packet loss = 100% [19:19:32] (03CR) 10Volans: [C:03+2] wmcs root keys: add FIDO keys for volans [puppet] - 10https://gerrit.wikimedia.org/r/1360965 (https://phabricator.wikimedia.org/T443002) (owner: 10Volans) [19:20:50] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 02 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [extensions/ReadingLists] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360551 (https://phabricator.wikimedia.org/T440916) (owner: 10Jdlrobson) [19:21:06] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, November 02 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-i" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359878 (https://phabricator.wikimedia.org/T441705) (owner: 10Jdlrobson) [19:22:07] RECOVERY - Host mr1-eqsin.oob is UP: PING OK - Packet loss = 0%, RTA = 238.37 ms [19:29:01] jouncebot: nowandnext [19:29:01] No deployments scheduled for the next 1 hour(s) and 30 minute(s) [19:29:01] In 1 hour(s) and 30 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T2100) [19:29:36] updating envoys throughout wikikube staging, here comes 57 helmfile deploys :) [19:29:53] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/apertium: apply [19:30:22] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/apertium: apply [19:33:19] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/chart-renderer: apply [19:33:35] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/chart-renderer: apply [19:34:15] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [19:34:23] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [19:34:26] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:34:29] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/citoid: apply [19:34:36] 10ops-codfw, 06DC-Ops: Alert for device ps1-a7-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T443335 (10phaultfinder) 03NEW [19:34:39] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12505114 (10phaultfinder) [19:34:42] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/citoid: apply [19:34:48] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply [19:35:00] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply [19:35:08] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/cxserver: apply [19:35:25] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/cxserver: apply [19:35:32] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/data-gateway: apply [19:35:44] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/data-gateway: apply [19:35:50] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/developer-portal: apply [19:35:59] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/developer-portal: apply [19:36:04] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/device-analytics: apply [19:36:16] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/device-analytics: apply [19:36:22] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/echostore: apply [19:36:33] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/echostore: apply [19:36:39] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/edit-analytics: apply [19:36:53] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/edit-analytics: apply [19:36:57] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/editcheck-headless: apply [19:37:14] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply [19:37:31] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/editor-analytics: apply [19:37:44] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/editor-analytics: apply [19:37:49] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/eventgate-analytics: apply [19:38:16] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply [19:39:01] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/eventstreams: apply [19:39:23] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/eventstreams: apply [19:39:40] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/geo-analytics: apply [19:39:49] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/geo-analytics: apply [19:39:55] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/kartotherian: apply [19:40:07] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/kartotherian: apply [19:40:20] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/lac-lambdas: apply [19:40:35] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply [19:40:56] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/linked-artifacts: apply [19:41:05] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply [19:41:31] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/linkrecommendation: apply [19:41:43] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply [19:41:51] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/machinetranslation: apply [19:44:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12505223 (10phaultfinder) [19:45:30] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/machinetranslation: apply [19:45:41] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/media-analytics: apply [19:45:51] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/media-analytics: apply [19:46:56] (03PS1) 10WMDE-leszek: Remove non-hardware ssh key for wmde-leszek [puppet] - 10https://gerrit.wikimedia.org/r/1361542 (https://phabricator.wikimedia.org/T442763) [19:47:35] rsilvola-wmf: I see you deployed miscweb in eqiad and codfw but not staging, was that intentional? if I deploy it in staging will that break you? [19:48:17] oops it's late for Riku -- cc sbassett in case you know ^ [19:49:01] if I don't hear, I'll go ahead, I expect it's probably fine but just making sure [19:50:46] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/mobileapps: apply [19:51:06] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/mobileapps: apply [19:51:48] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/page-analytics: apply [19:52:00] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/page-analytics: apply [19:52:04] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/proton: apply [19:52:15] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/proton: apply [19:52:35] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/push-notifications: apply [19:52:46] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/push-notifications: apply [19:53:15] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/recommendation-api: apply [19:53:26] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/recommendation-api: apply [19:53:33] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [19:53:39] (03PS1) 10Addshore: admin: Add second new HW backed SSH key for addshore & rm old [puppet] - 10https://gerrit.wikimedia.org/r/1361545 (https://phabricator.wikimedia.org/T442758) [19:53:44] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [19:53:58] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/sessionstore: apply [19:54:09] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/sessionstore: apply [19:54:16] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: apply [19:54:27] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: apply [19:54:34] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-constraints: apply [19:54:47] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply [19:54:55] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-media: apply [19:55:10] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-media: apply [19:56:14] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: apply [19:56:25] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply [19:57:01] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-video: apply [19:57:27] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-video: apply [19:57:33] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply [19:57:57] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply [19:58:05] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/termbox: apply [19:58:33] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/termbox: apply [19:58:38] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/thumbor: apply [19:58:48] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [19:58:54] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/toolhub: apply [19:59:19] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/toolhub: apply [19:59:51] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply [20:00:08] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply [20:00:34] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/wikifeeds: apply [20:00:50] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [20:00:58] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [20:01:46] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [20:01:56] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/zotero: apply [20:02:13] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/zotero: apply [20:04:40] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12505442 (10phaultfinder) [20:07:41] (03CR) 10Ladsgroup: [C:03+1] "confirmed oob again" [puppet] - 10https://gerrit.wikimedia.org/r/1361545 (https://phabricator.wikimedia.org/T442758) (owner: 10Addshore) [20:12:36] =] [20:14:15] FIRING: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [20:17:48] 10SRE-swift-storage, 10MediaWiki-File-management, 06MediaWiki-Media-Platform-Team: Unknown error while undeleting a file on Wikimedia Commons - https://phabricator.wikimedia.org/T437114#12505484 (10Sreejithk2000) Someone renamed another file to this name after I reported it here, but if you see the history,... [20:19:15] RESOLVED: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [20:24:07] 10SRE-Access-Requests: Add FIDO-backed SSH keys for bearloga - https://phabricator.wikimedia.org/T443359 (10mpopov) 03NEW [20:25:14] (03PS1) 10Bearloga: admin: Add FIDO SSH keys for bearloga [puppet] - 10https://gerrit.wikimedia.org/r/1361550 (https://phabricator.wikimedia.org/T443359) [20:29:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12505609 (10phaultfinder) [20:33:53] !log rzl@deploy2003 helmfile [staging] START helmfile.d/services/miscweb: apply [20:34:56] !log rzl@deploy2003 helmfile [staging] DONE helmfile.d/services/miscweb: apply [20:36:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [20:41:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [20:43:03] !log cscott@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [20:43:55] !log cscott@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [20:47:14] 06SRE, 10LDAP-Access-Requests: Grant Access to Logstash for SomeRandomDeveloper - https://phabricator.wikimedia.org/T442855#12505727 (10KFrancis) Hi all, I also answered this on another ticket, but yes, the user was accidentally omitted from the spreadsheet. It has been corrected. Sorry for any confusion! [20:59:09] @Krinkle do you want to do your change first? I have a few [21:00:05] RoanKattouw, urbanecm, TheresNoTime, and cjming: Time to do the UTC late backport window deploy. Don't look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T2100). [21:00:05] jdlrobson and Krinkle: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [21:06:18] Krinkle: ? Or do you want me to go first? [21:06:59] o/ [21:07:02] Jdlrobson: go ahead [21:07:06] k [21:07:59] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy2003 using scap backport" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360497 (https://phabricator.wikimedia.org/T439273) (owner: 10Jdlrobson) [21:08:00] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy2003 using scap backport" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360498 (https://phabricator.wikimedia.org/T439273) (owner: 10Jdlrobson) [21:08:00] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy2003 using scap backport" [extensions/ReadingLists] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360551 (https://phabricator.wikimedia.org/T440916) (owner: 10Jdlrobson) [21:13:43] (03Merged) 10jenkins-bot: Skin: Update skin-responsive logic [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360497 (https://phabricator.wikimedia.org/T439273) (owner: 10Jdlrobson) [21:14:46] (03Merged) 10jenkins-bot: Skin: skin--responsive class no longer varies on user [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360498 (https://phabricator.wikimedia.org/T439273) (owner: 10Jdlrobson) [21:14:51] (03Merged) 10jenkins-bot: Fast forward wmf.1 branch to 0626e6 [extensions/ReadingLists] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1360551 (https://phabricator.wikimedia.org/T440916) (owner: 10Jdlrobson) [21:15:12] !log jdlrobson@deploy2003 Started scap sync-world: Backport for [[gerrit:1360497|Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498|Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551|Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) [21:15:31] T439273: Responsive preference class on BODY tag should be relocated to HTML tag - https://phabricator.wikimedia.org/T439273 [21:15:31] T440916: Reading Lists: A collection title can be saved with only whitespace - https://phabricator.wikimedia.org/T440916 [21:15:32] T441259: Visual refinements to collections picker - https://phabricator.wikimedia.org/T441259 [21:15:33] T441260: Item count is not localized for language - https://phabricator.wikimedia.org/T441260 [21:15:33] T441477: Instrument reading list engagement - https://phabricator.wikimedia.org/T441477 [21:15:34] T442906: ReadingLists: Load the anonymous bookmark CTA only when the user clicks - https://phabricator.wikimedia.org/T442906 [21:17:38] (03PS1) 10Krinkle: Handle \mathcal in mmlFilter instead of the font [extensions/Math] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361554 [21:18:00] (03PS1) 10Krinkle: Keep \mathrm upright inside \mathbf [extensions/Math] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361555 (https://phabricator.wikimedia.org/T442312) [21:18:23] (03PS1) 10Krinkle: Draw no outer line for a nested \hline [extensions/Math] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361556 [21:18:40] (03CR) 10Bking: [C:03+1] raid: Treat hot spare drives as healthy in the broadcom check [puppet] - 10https://gerrit.wikimedia.org/r/1361526 (https://phabricator.wikimedia.org/T441831) (owner: 10Btullis) [21:19:37] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442574#12505948 (10phaultfinder) [21:21:48] (03PS3) 10Jdlrobson: Enable Vector CTA on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359878 (https://phabricator.wikimedia.org/T441705) [21:21:56] (03PS4) 10Jdlrobson: Enable Vector CTA on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359878 (https://phabricator.wikimedia.org/T441705) [21:23:59] (03CR) 10CI reject: [V:04-1] Keep \mathrm upright inside \mathbf [extensions/Math] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361555 (https://phabricator.wikimedia.org/T442312) (owner: 10Krinkle) [21:25:21] (03CR) 10Krinkle: "recheck" [extensions/Math] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361555 (https://phabricator.wikimedia.org/T442312) (owner: 10Krinkle) [21:25:24] (03CR) 10Krinkle: "ref T401044" [extensions/Math] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361555 (https://phabricator.wikimedia.org/T442312) (owner: 10Krinkle) [21:33:10] 07Puppet, 10Beta-Cluster-Infrastructure: Cannot rollout Trixie hosts in deployment-prep without rm -rf /var/lib/puppet/ssl - https://phabricator.wikimedia.org/T443373 (10Southparkfan) 03NEW [21:33:51] 07Puppet, 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 06tools-infrastructure-team: Cannot rollout Trixie hosts in deployment-prep without rm -rf /var/lib/puppet/ssl - https://phabricator.wikimedia.org/T443373#12506006 (10Southparkfan) [21:36:37] !log jdlrobson@deploy2003 jdlrobson: Backport for [[gerrit:1360497|Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498|Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551|Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes [21:36:37] can now be verified there. [21:36:53] T439273: Responsive preference class on BODY tag should be relocated to HTML tag - https://phabricator.wikimedia.org/T439273 [21:36:54] T440916: Reading Lists: A collection title can be saved with only whitespace - https://phabricator.wikimedia.org/T440916 [21:36:55] T441259: Visual refinements to collections picker - https://phabricator.wikimedia.org/T441259 [21:36:55] T441260: Item count is not localized for language - https://phabricator.wikimedia.org/T441260 [21:36:56] T441477: Instrument reading list engagement - https://phabricator.wikimedia.org/T441477 [21:36:57] T442906: ReadingLists: Load the anonymous bookmark CTA only when the user clicks - https://phabricator.wikimedia.org/T442906 [21:38:06] !log jdlrobson@deploy2003 jdlrobson: Continuing with deployment [21:47:17] Krinkle: sorry scap was taking very long here... [21:47:33] the config change will hopefully be super quick but I had to do it separately to check the code was deployable. [21:50:24] !log jdlrobson@deploy2003 Finished scap sync-world: Backport for [[gerrit:1360497|Skin: Update skin-responsive logic (T439273)]] (1.48.0-wmf.1), [[gerrit:1360498|Skin: skin--responsive class no longer varies on user (T439273)]] (1.48.0-wmf.1), [[gerrit:1360551|Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906)]] (1.48.0-wmf.1) (duration: 35m 12s) [21:50:39] T439273: Responsive preference class on BODY tag should be relocated to HTML tag - https://phabricator.wikimedia.org/T439273 [21:50:40] T440916: Reading Lists: A collection title can be saved with only whitespace - https://phabricator.wikimedia.org/T440916 [21:50:41] T441259: Visual refinements to collections picker - https://phabricator.wikimedia.org/T441259 [21:50:41] T441260: Item count is not localized for language - https://phabricator.wikimedia.org/T441260 [21:50:42] T441477: Instrument reading list engagement - https://phabricator.wikimedia.org/T441477 [21:50:43] T442906: ReadingLists: Load the anonymous bookmark CTA only when the user clicks - https://phabricator.wikimedia.org/T442906 [21:51:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359878 (https://phabricator.wikimedia.org/T441705) (owner: 10Jdlrobson) [21:51:49] Jdlrobson: np [21:52:16] (03Merged) 10jenkins-bot: Enable Vector CTA on all wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1359878 (https://phabricator.wikimedia.org/T441705) (owner: 10Jdlrobson) [21:52:37] !log jdlrobson@deploy2003 Started scap sync-world: Backport for [[gerrit:1359878|Enable Vector CTA on all wikipedias (T441705)]] [21:52:43] T441705: Desktop CTA: Deploy the new config variable to launch the desktop CTA - https://phabricator.wikimedia.org/T441705 [21:56:54] !log jdlrobson@deploy2003 jdlrobson: Backport for [[gerrit:1359878|Enable Vector CTA on all wikipedias (T441705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:57:45] !log jdlrobson@deploy2003 jdlrobson: Continuing with deployment [21:59:36] ok Krinkle all yours when this is done [22:00:04] alexsanford, Reedy, sbassett, Maryum, and manfredi: gettimeofday() says it's time for Weekly Security deployment window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261102T2200) [22:02:40] 07Puppet, 10Beta-Cluster-Infrastructure, 10Cloud-VPS, 06tools-infrastructure-team: Cannot rollout Trixie hosts in deployment-prep without rm -rf /var/lib/puppet/ssl - https://phabricator.wikimedia.org/T443373#12506104 (10dancy) Thanks for the report @Southparkfan. I'll look into this. [22:03:37] I do have several security patches to get out [22:03:44] Krinkle are you already running scap? [22:04:12] maryum: jon is, I'm queued up after [22:04:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12506105 (10phaultfinder) [22:05:13] (03PS1) 10ArielGlenn: changeprop: node 20 to node 26 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361572 (https://phabricator.wikimedia.org/T442195) [22:05:35] maryum: should be done in ~30min [22:05:43] !log jdlrobson@deploy2003 Finished scap sync-world: Backport for [[gerrit:1359878|Enable Vector CTA on all wikipedias (T441705)]] (duration: 13m 05s) [22:05:49] T441705: Desktop CTA: Deploy the new config variable to launch the desktop CTA - https://phabricator.wikimedia.org/T441705 [22:06:32] Krinkle: thanks! [22:08:28] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy2003 using scap backport" [extensions/Math] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361554 (owner: 10Krinkle) [22:08:29] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy2003 using scap backport" [extensions/Math] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361555 (https://phabricator.wikimedia.org/T442312) (owner: 10Krinkle) [22:08:31] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy2003 using scap backport" [extensions/Math] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361556 (owner: 10Krinkle) [22:09:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442568#12506115 (10phaultfinder) [22:10:29] (03Merged) 10jenkins-bot: Handle \mathcal in mmlFilter instead of the font [extensions/Math] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361554 (owner: 10Krinkle) [22:10:39] (03Merged) 10jenkins-bot: Keep \mathrm upright inside \mathbf [extensions/Math] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361555 (https://phabricator.wikimedia.org/T442312) (owner: 10Krinkle) [22:10:52] (03Merged) 10jenkins-bot: Draw no outer line for a nested \hline [extensions/Math] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1361556 (owner: 10Krinkle) [22:11:11] !log krinkle@deploy2003 Started scap sync-world: Backport for [[gerrit:1361554|Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555|Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556|Draw no outer line for a nested \hline]] (1.48.0-wmf.1) [22:11:17] T442312: \mathrm inside \mathbf renders bold instead of upright - https://phabricator.wikimedia.org/T442312 [22:15:17] !log krinkle@deploy2003 krinkle: Backport for [[gerrit:1361554|Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555|Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556|Draw no outer line for a nested \hline]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [22:19:12] !log krinkle@deploy2003 krinkle: Continuing with deployment [22:24:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12506205 (10phaultfinder) [22:25:01] !log krinkle@deploy2003 Finished scap sync-world: Backport for [[gerrit:1361554|Handle \mathcal in mmlFilter instead of the font]] (1.48.0-wmf.1), [[gerrit:1361555|Keep \mathrm upright inside \mathbf (T442312)]] (1.48.0-wmf.1), [[gerrit:1361556|Draw no outer line for a nested \hline]] (1.48.0-wmf.1) (duration: 13m 50s) [22:25:07] T442312: \mathrm inside \mathbf renders bold instead of upright - https://phabricator.wikimedia.org/T442312 [22:25:20] maryum: all yours [22:26:33] (03CR) 10BPirkle: [C:03+1] "Tag confirmed on corresponding changeprop change." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361572 (https://phabricator.wikimedia.org/T442195) (owner: 10ArielGlenn) [22:29:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [22:35:02] !log cscott@deploy2003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [22:35:49] !log cscott@deploy2003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [22:39:52] Krinkle: thank you! [22:40:12] FIRING: [2x] ProbeDown: Service people1005:30443 has failed probes (http_security_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [22:42:53] 10SRE-tools, 06Infrastructure-Foundations, 10Packaging, 13Patch-For-Review: Upgrade prometheus-atlas-exporter - https://phabricator.wikimedia.org/T429672#12506289 (10Lizimedia) ` root@10801bc77cf6:/# curl -O https://apt-staging.wikimedia.org/wikimedia-staging/pool/main/p/prometheus-atlas-exporter/prometheu... [22:43:20] (03PS1) 10Ahmon Dancy: profile::puppet::agent: ignore system CAs in SSL re-anchor check [puppet] - 10https://gerrit.wikimedia.org/r/1361577 (https://phabricator.wikimedia.org/T443373) [22:44:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [22:49:43] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12506321 (10phaultfinder) [22:50:07] preparing to run scap [22:50:13] scap sync-world [23:24:39] !log Deployed security fix for T440502, T440330, T442830, T442861 [23:24:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:24:45] preparing to run a second scap [23:31:05] (03CR) 10BCornwall: "I feel that this effort might best manifest as a [MediaWiki-Docker configuration recipe](https://www.mediawiki.org/wiki/MediaWiki-Docker/C" [puppet] - 10https://gerrit.wikimedia.org/r/1361368 (owner: 10Vgutierrez) [23:32:47] (03PS1) 10Lizimedia: Add prometheus-atls-exporter for bookworm and staging [puppet] - 10https://gerrit.wikimedia.org/r/1361580 (https://phabricator.wikimedia.org/T429672) [23:33:32] (03PS2) 10Lizimedia: Add prometheus-atlas-exporter for bookworm and staging [puppet] - 10https://gerrit.wikimedia.org/r/1361580 (https://phabricator.wikimedia.org/T429672) [23:34:26] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:35:00] preparing to run a second scap [23:49:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b8-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T442577#12506516 (10phaultfinder)