[00:00:07] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [00:00:17] (03Merged) 10jenkins-bot: Enable chunked videos on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362217 (https://phabricator.wikimedia.org/T373546) (owner: 10Ladsgroup) [00:00:18] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [00:00:28] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/proton: apply [00:00:31] (03Merged) 10jenkins-bot: Avoid unnecessary/aborted requests in DASH mode [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1362212 (https://phabricator.wikimedia.org/T442988) (owner: 10Ladsgroup) [00:01:00] !log ladsgroup@deploy2003 Started scap sync-world: Backport for [[gerrit:1362212|Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217|Enable chunked videos on testwiki (T373546)]] [00:01:12] T442988: TMH DASH unnecessary/aborted requests - https://phabricator.wikimedia.org/T442988 [00:01:13] T373546: Migrate off HLS mov/mp4 experiment to chunked WebM MPEG-DASH - https://phabricator.wikimedia.org/T373546 [00:03:09] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.28 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [00:04:26] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:05:02] 10ops-codfw, 06SRE, 06DC-Ops: dbprov2004 - https://phabricator.wikimedia.org/T441236#12513777 (10Papaul) 05Open→03Resolved [00:05:36] !log ladsgroup@deploy2003 ladsgroup: Backport for [[gerrit:1362212|Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217|Enable chunked videos on testwiki (T373546)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [00:05:43] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1170 (T441633)', diff saved to https://phabricator.wikimedia.org/P97187 and previous config saved to /var/cache/conftool/dbconfig/20261104-000542-ladsgroup.json [00:06:01] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance [00:07:47] PROBLEM - Ensure traffic_server is running for instance backend on cp1100 is CRITICAL: PROCS CRITICAL: 3 processes with args /usr/bin/traffic_server -M --httpport 3128 https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [00:07:48] !log ladsgroup@deploy2003 ladsgroup: Continuing with deployment [00:08:36] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/push-notifications: apply [00:08:47] RECOVERY - Ensure traffic_server is running for instance backend on cp1100 is OK: PROCS OK: 1 process with args /usr/bin/traffic_server -M --httpport 3128 https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [00:09:03] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/push-notifications: apply [00:09:42] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/recommendation-api: apply [00:10:19] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply [00:10:56] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/sessionstore: apply [00:11:19] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance [00:11:20] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/sessionstore: apply [00:11:21] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97188 and previous config saved to /var/cache/conftool/dbconfig/20261104-001120-ladsgroup.json [00:12:05] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply [00:12:53] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply [00:13:07] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/termbox: apply [00:13:48] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/termbox: apply [00:13:53] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/toolhub: apply [00:14:02] !log ladsgroup@deploy2003 Finished scap sync-world: Backport for [[gerrit:1362212|Avoid unnecessary/aborted requests in DASH mode (T442988)]] (1.48.0-wmf.3), [[gerrit:1362217|Enable chunked videos on testwiki (T373546)]] (duration: 13m 02s) [00:14:09] T442988: TMH DASH unnecessary/aborted requests - https://phabricator.wikimedia.org/T442988 [00:14:09] T373546: Migrate off HLS mov/mp4 experiment to chunked WebM MPEG-DASH - https://phabricator.wikimedia.org/T373546 [00:14:35] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/toolhub: apply [00:14:47] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply [00:15:09] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply [00:15:39] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/wikifeeds: apply [00:15:53] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply [00:15:57] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [00:19:21] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97189 and previous config saved to /var/cache/conftool/dbconfig/20261104-001920-ladsgroup.json [00:26:07] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [00:27:38] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [00:27:47] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [00:28:08] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/zotero: apply [00:28:27] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/zotero: apply [00:29:03] done for now [00:29:23] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97190 and previous config saved to /var/cache/conftool/dbconfig/20261104-002922-ladsgroup.json [00:30:10] that's not true, one retry [00:30:52] !log rzl@deploy2003 helmfile [codfw] START helmfile.d/services/kartotherian: apply [00:32:00] !log rzl@deploy2003 helmfile [codfw] DONE helmfile.d/services/kartotherian: apply [00:32:13] that's more like it [00:38:58] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1362259 [00:38:58] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1362259 (owner: 10TrainBranchBot) [00:39:25] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97191 and previous config saved to /var/cache/conftool/dbconfig/20261104-003924-ladsgroup.json [00:46:18] (03CR) 10CI reject: [V:04-1] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1362259 (owner: 10TrainBranchBot) [00:49:27] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97192 and previous config saved to /var/cache/conftool/dbconfig/20261104-004926-ladsgroup.json [00:49:34] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance [00:49:36] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1190 (T441633)', diff saved to https://phabricator.wikimedia.org/P97193 and previous config saved to /var/cache/conftool/dbconfig/20261104-004935-ladsgroup.json [00:58:04] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1190 (T441633)', diff saved to https://phabricator.wikimedia.org/P97194 and previous config saved to /var/cache/conftool/dbconfig/20261104-005803-ladsgroup.json [01:08:06] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97195 and previous config saved to /var/cache/conftool/dbconfig/20261104-010805-ladsgroup.json [01:08:58] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1362287 [01:08:58] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1362287 (owner: 10TrainBranchBot) [01:15:39] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1362287 (owner: 10TrainBranchBot) [01:18:08] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97196 and previous config saved to /var/cache/conftool/dbconfig/20261104-011807-ladsgroup.json [01:28:10] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1190 (T441633)', diff saved to https://phabricator.wikimedia.org/P97197 and previous config saved to /var/cache/conftool/dbconfig/20261104-012809-ladsgroup.json [01:28:28] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance [01:28:30] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1191 (T441633)', diff saved to https://phabricator.wikimedia.org/P97198 and previous config saved to /var/cache/conftool/dbconfig/20261104-012829-ladsgroup.json [01:35:20] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1191 (T441633)', diff saved to https://phabricator.wikimedia.org/P97199 and previous config saved to /var/cache/conftool/dbconfig/20261104-013519-ladsgroup.json [01:45:22] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97200 and previous config saved to /var/cache/conftool/dbconfig/20261104-014521-ladsgroup.json [01:55:24] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97201 and previous config saved to /var/cache/conftool/dbconfig/20261104-015523-ladsgroup.json [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T0200) [02:00:50] !log mwpresync@deploy2003 Started scap build-images: Publishing wmf/next image [02:05:26] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1191 (T441633)', diff saved to https://phabricator.wikimedia.org/P97202 and previous config saved to /var/cache/conftool/dbconfig/20261104-020525-ladsgroup.json [02:05:44] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance [02:05:46] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1194 (T441633)', diff saved to https://phabricator.wikimedia.org/P97203 and previous config saved to /var/cache/conftool/dbconfig/20261104-020545-ladsgroup.json [02:08:23] !log mwpresync@deploy2003 Finished scap build-images: Publishing wmf/next image (duration: 07m 33s) [02:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:13:57] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1194 (T441633)', diff saved to https://phabricator.wikimedia.org/P97204 and previous config saved to /var/cache/conftool/dbconfig/20261104-021356-ladsgroup.json [02:16:42] FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:21:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:23:59] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97205 and previous config saved to /var/cache/conftool/dbconfig/20261104-022358-ladsgroup.json [02:34:01] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97206 and previous config saved to /var/cache/conftool/dbconfig/20261104-023401-ladsgroup.json [02:44:03] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1194 (T441633)', diff saved to https://phabricator.wikimedia.org/P97207 and previous config saved to /var/cache/conftool/dbconfig/20261104-024403-ladsgroup.json [02:44:22] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance [02:44:24] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1202 (T441633)', diff saved to https://phabricator.wikimedia.org/P97208 and previous config saved to /var/cache/conftool/dbconfig/20261104-024423-ladsgroup.json [02:51:24] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1202 (T441633)', diff saved to https://phabricator.wikimedia.org/P97209 and previous config saved to /var/cache/conftool/dbconfig/20261104-025123-ladsgroup.json [03:01:26] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97210 and previous config saved to /var/cache/conftool/dbconfig/20261104-030125-ladsgroup.json [03:05:27] PROBLEM - Check unit status of fundraising-data-import on deploy2003 is CRITICAL: CRITICAL: Status of the systemd unit fundraising-data-import https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [03:11:28] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97211 and previous config saved to /var/cache/conftool/dbconfig/20261104-031127-ladsgroup.json [03:21:30] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1202 (T441633)', diff saved to https://phabricator.wikimedia.org/P97212 and previous config saved to /var/cache/conftool/dbconfig/20261104-032129-ladsgroup.json [03:21:48] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance [03:21:50] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1227 (T441633)', diff saved to https://phabricator.wikimedia.org/P97213 and previous config saved to /var/cache/conftool/dbconfig/20261104-032149-ladsgroup.json [03:28:56] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1227 (T441633)', diff saved to https://phabricator.wikimedia.org/P97214 and previous config saved to /var/cache/conftool/dbconfig/20261104-032855-ladsgroup.json [03:38:58] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97215 and previous config saved to /var/cache/conftool/dbconfig/20261104-033857-ladsgroup.json [03:49:00] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97216 and previous config saved to /var/cache/conftool/dbconfig/20261104-034859-ladsgroup.json [03:59:02] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1227 (T441633)', diff saved to https://phabricator.wikimedia.org/P97217 and previous config saved to /var/cache/conftool/dbconfig/20261104-035901-ladsgroup.json [03:59:20] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance [03:59:22] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1231 (T441633)', diff saved to https://phabricator.wikimedia.org/P97218 and previous config saved to /var/cache/conftool/dbconfig/20261104-035922-ladsgroup.json [04:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:06:14] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1231 (T441633)', diff saved to https://phabricator.wikimedia.org/P97219 and previous config saved to /var/cache/conftool/dbconfig/20261104-040613-ladsgroup.json [04:16:16] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97220 and previous config saved to /var/cache/conftool/dbconfig/20261104-041615-ladsgroup.json [04:23:34] (03PS1) 10AKhatun: revision feature counts lac-lambda: Tune alerts and add 2 alerts [alerts] - 10https://gerrit.wikimedia.org/r/1362469 (https://phabricator.wikimedia.org/T442914) [04:26:18] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97221 and previous config saved to /var/cache/conftool/dbconfig/20261104-042617-ladsgroup.json [04:28:25] (03PS1) 10AKhatun: revision feature counts lac-lambda: reduce cpu and memory [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362476 (https://phabricator.wikimedia.org/T442914) [04:36:20] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1231 (T441633)', diff saved to https://phabricator.wikimedia.org/P97222 and previous config saved to /var/cache/conftool/dbconfig/20261104-043619-ladsgroup.json [04:36:38] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance [04:36:39] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1253 (T441633)', diff saved to https://phabricator.wikimedia.org/P97223 and previous config saved to /var/cache/conftool/dbconfig/20261104-043639-ladsgroup.json [04:43:46] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1253 (T441633)', diff saved to https://phabricator.wikimedia.org/P97224 and previous config saved to /var/cache/conftool/dbconfig/20261104-044345-ladsgroup.json [04:53:48] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97225 and previous config saved to /var/cache/conftool/dbconfig/20261104-045347-ladsgroup.json [05:03:50] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97226 and previous config saved to /var/cache/conftool/dbconfig/20261104-050349-ladsgroup.json [05:13:52] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1253 (T441633)', diff saved to https://phabricator.wikimedia.org/P97227 and previous config saved to /var/cache/conftool/dbconfig/20261104-051351-ladsgroup.json [05:14:11] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1273.eqiad.wmnet with reason: Maintenance [05:14:22] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance [05:14:24] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1273 (T441633)', diff saved to https://phabricator.wikimedia.org/P97228 and previous config saved to /var/cache/conftool/dbconfig/20261104-051423-ladsgroup.json [05:21:36] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1273 (T441633)', diff saved to https://phabricator.wikimedia.org/P97229 and previous config saved to /var/cache/conftool/dbconfig/20261104-052135-ladsgroup.json [05:24:23] 10ops-eqiad, 06DC-Ops: Unresponsive management for an-worker1181.mgmt:22 - https://phabricator.wikimedia.org/T443555 (10phaultfinder) 03NEW [05:31:38] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97230 and previous config saved to /var/cache/conftool/dbconfig/20261104-053137-ladsgroup.json [05:41:40] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97231 and previous config saved to /var/cache/conftool/dbconfig/20261104-054139-ladsgroup.json [05:51:42] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1273 (T441633)', diff saved to https://phabricator.wikimedia.org/P97232 and previous config saved to /var/cache/conftool/dbconfig/20261104-055141-ladsgroup.json [05:52:00] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1284.eqiad.wmnet with reason: Maintenance [05:52:01] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1284 (T441633)', diff saved to https://phabricator.wikimedia.org/P97233 and previous config saved to /var/cache/conftool/dbconfig/20261104-055201-ladsgroup.json [05:57:07] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1284 (T441633)', diff saved to https://phabricator.wikimedia.org/P97234 and previous config saved to /var/cache/conftool/dbconfig/20261104-055706-ladsgroup.json [06:07:09] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97235 and previous config saved to /var/cache/conftool/dbconfig/20261104-060708-ladsgroup.json [06:14:46] (03PS4) 10Winston Sung: Add language tags 'cmn-latn-*' to wmgExtraLanguageNames [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1354265 (https://phabricator.wikimedia.org/T440824) [06:15:00] (03PS5) 10Winston Sung: Add language tags 'cmn-latn-*' to wmgExtraLanguageNames for Wikidata termbox support [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1354265 (https://phabricator.wikimedia.org/T440824) [06:17:11] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97236 and previous config saved to /var/cache/conftool/dbconfig/20261104-061710-ladsgroup.json [06:27:12] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1284 (T441633)', diff saved to https://phabricator.wikimedia.org/P97237 and previous config saved to /var/cache/conftool/dbconfig/20261104-062712-ladsgroup.json [06:27:30] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1285.eqiad.wmnet with reason: Maintenance [06:30:53] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1288.eqiad.wmnet with reason: Maintenance [06:30:55] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db1288 (T441633)', diff saved to https://phabricator.wikimedia.org/P97238 and previous config saved to /var/cache/conftool/dbconfig/20261104-063055-ladsgroup.json [06:35:55] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1288 (T441633)', diff saved to https://phabricator.wikimedia.org/P97239 and previous config saved to /var/cache/conftool/dbconfig/20261104-063554-ladsgroup.json [06:41:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [06:45:57] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97240 and previous config saved to /var/cache/conftool/dbconfig/20261104-064556-ladsgroup.json [06:46:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [06:55:50] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Server didn't come back [06:55:59] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97241 and previous config saved to /var/cache/conftool/dbconfig/20261104-065558-ladsgroup.json [07:00:05] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T0700) [07:01:26] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12514263 (10Marostegui) @Papaul @VRiley-WMF can we get the host set to `Performance` so we can put it back to production. In any case I do think we need to open a case with Dell. Those two errors are clear and t... [07:05:34] 10ops-eqiad, 06SRE, 06DC-Ops, 06ServiceOps: Rerack mc1057, mc1061, mc1069 - https://phabricator.wikimedia.org/T443209#12514270 (10jijiki) @VRiley-WMF thank you! Could you please give me some alternatives I could look into? The limitations are not to have an mc-gp* host in the same rack, or more than 2 mc*... [07:06:01] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db1288 (T441633)', diff saved to https://phabricator.wikimedia.org/P97242 and previous config saved to /var/cache/conftool/dbconfig/20261104-070600-ladsgroup.json [07:06:20] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance [07:15:23] !log marostegui@cumin1004 START - Cookbook sre.mysql.pool pool db1274: after reboot [07:28:05] (03PS1) 10Marostegui: dbprov2005: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1362807 (https://phabricator.wikimedia.org/T441988) [07:29:20] (03PS6) 10Slyngshede: CAS 8.0.3 [software/cas-overlay-template] - 10https://gerrit.wikimedia.org/r/1346564 (https://phabricator.wikimedia.org/T439509) [07:29:30] (03CR) 10Marostegui: [C:03+2] dbprov2005: Disable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1362807 (https://phabricator.wikimedia.org/T441988) (owner: 10Marostegui) [07:30:03] (03PS1) 10Ayounsi: Nokia: handle access interface with no untagged_vlan [homer/public] - 10https://gerrit.wikimedia.org/r/1362812 [07:30:18] !log marostegui@cumin1004 START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS trixie [07:31:03] (03CR) 10CI reject: [V:04-1] Nokia: handle access interface with no untagged_vlan [homer/public] - 10https://gerrit.wikimedia.org/r/1362812 (owner: 10Ayounsi) [07:31:14] !log marostegui@cumin1004 START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (push config only) [07:31:19] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (push config only) [07:36:24] (03CR) 10Filippo Giunchedi: [C:03+2] team-wmcs: silence promql/series for CloudVirtDisabled [alerts] - 10https://gerrit.wikimedia.org/r/1361854 (owner: 10Filippo Giunchedi) [07:45:16] !log marostegui@cumin1004 START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (remove config only) [07:45:22] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (remove config only) [07:47:51] !log marostegui@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage [07:52:03] (03PS1) 10Muehlenhoff: Remove legacy key for astein [puppet] - 10https://gerrit.wikimedia.org/r/1362877 (https://phabricator.wikimedia.org/T442759) [07:53:34] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage [07:55:08] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy key for astein [puppet] - 10https://gerrit.wikimedia.org/r/1362877 (https://phabricator.wikimedia.org/T442759) (owner: 10Muehlenhoff) [08:00:05] Amir1, urbanecm, and awight: Your horoscope predicts another UTC morning backport window deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T0800). [08:00:05] No Gerrit patches in the queue for this window AFAICS. [08:00:14] o/ if nothing is happening in this window, I'd like to do a private deploy [08:00:27] !log marostegui@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: after reboot [08:01:22] going to start [08:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:06:48] !log jmm@cumin2003 START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org [08:08:30] (03CR) 10Muehlenhoff: [C:03+1] "The key has been verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1359804 (https://phabricator.wikimedia.org/T442665) (owner: 10Fabian Kaelin) [08:08:35] (03CR) 10Muehlenhoff: [C:03+2] admin: Add FIDO SSH key for fab [puppet] - 10https://gerrit.wikimedia.org/r/1359804 (https://phabricator.wikimedia.org/T442665) (owner: 10Fabian Kaelin) [08:30:26] (03PS4) 10Arnaudb: gerrit: fix the readonly plugin config [puppet] - 10https://gerrit.wikimedia.org/r/1356233 (https://phabricator.wikimedia.org/T440177) [08:31:53] (03CR) 10Muehlenhoff: [C:03+1] "Looks good!" [puppet] - 10https://gerrit.wikimedia.org/r/1361580 (https://phabricator.wikimedia.org/T429672) (owner: 10Lizimedia) [08:32:22] (03CR) 10Btullis: [C:03+1] admin: remove non FIDO SSH keys [puppet] - 10https://gerrit.wikimedia.org/r/1362898 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [08:34:40] (03CR) 10Arnaudb: "Thanks for the split and the message! Plugin commands are top-level, so I think `replication` is right, details inline." [puppet] - 10https://gerrit.wikimedia.org/r/1356233 (https://phabricator.wikimedia.org/T440177) (owner: 10Arnaudb) [08:34:42] (03PS1) 10Marostegui: installserver: Add trixie reuse recipes for dbprov [puppet] - 10https://gerrit.wikimedia.org/r/1362910 (https://phabricator.wikimedia.org/T441988) [08:35:00] (03PS2) 10Hashar: gerrit: allow `gerrit index` in readonly plugin [puppet] - 10https://gerrit.wikimedia.org/r/1358246 (https://phabricator.wikimedia.org/T440177) [08:35:33] (03CR) 10Arnaudb: "Thanks for splitting it!" [puppet] - 10https://gerrit.wikimedia.org/r/1358246 (https://phabricator.wikimedia.org/T440177) (owner: 10Hashar) [08:38:06] 10ops-esams, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: esams router and switch maint - https://phabricator.wikimedia.org/T440349#12514462 (10ayounsi) [08:40:59] (03CR) 10Marostegui: [C:03+2] installserver: Add trixie reuse recipes for dbprov [puppet] - 10https://gerrit.wikimedia.org/r/1362910 (https://phabricator.wikimedia.org/T441988) (owner: 10Marostegui) [08:42:38] (03CR) 10Gehel: [C:03+2] admin: remove non FIDO SSH keys [puppet] - 10https://gerrit.wikimedia.org/r/1362898 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [08:45:17] (03PS1) 10Gehel: admin: remove non FIDO SSH keys for passimacopoulos [puppet] - 10https://gerrit.wikimedia.org/r/1362920 (https://phabricator.wikimedia.org/T442756) [08:46:47] (03CR) 10Klausman: [C:03+1] admin: remove non FIDO SSH keys for passimacopoulos [puppet] - 10https://gerrit.wikimedia.org/r/1362920 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [08:47:18] (03PS8) 10Arnaudb: gerrit: add failover, DNS emergency and demotion cookbooks [cookbooks] - 10https://gerrit.wikimedia.org/r/1344697 (https://phabricator.wikimedia.org/T439025) [08:47:18] (03PS8) 10Arnaudb: gerrit: read the replica flag from the loaded unit [cookbooks] - 10https://gerrit.wikimedia.org/r/1344698 (https://phabricator.wikimedia.org/T439025) [08:47:18] (03PS8) 10Arnaudb: gerrit: reuse the shared helpers in sibling cookbooks [cookbooks] - 10https://gerrit.wikimedia.org/r/1344699 (https://phabricator.wikimedia.org/T439025) [08:50:34] !log marostegui@cumin1004 START - Cookbook sre.hosts.reimage for host dbprov2006.codfw.wmnet with OS trixie [08:51:31] !log marostegui@cumin1004 START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (push config only) [08:51:35] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (push config only) [08:53:19] (03CR) 10Gehel: [C:03+2] admin: remove non FIDO SSH keys for passimacopoulos [puppet] - 10https://gerrit.wikimedia.org/r/1362920 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [09:00:05] jnuche and jnuche: Time to snap out of that daydream and deploy MediaWiki train. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T0900). [09:00:12] hi, train is rolling out in a few minutes [09:02:32] (03PS1) 10TrainBranchBot: group1 to 1.48.0-wmf.3 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362941 (https://phabricator.wikimedia.org/T442221) [09:02:35] (03CR) 10TrainBranchBot: [C:03+2] "Initiated by jnuche@deploy2003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362941 (https://phabricator.wikimedia.org/T442221) (owner: 10TrainBranchBot) [09:03:29] (03Merged) 10jenkins-bot: group1 to 1.48.0-wmf.3 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362941 (https://phabricator.wikimedia.org/T442221) (owner: 10TrainBranchBot) [09:04:43] PROBLEM - Host titan1002 is DOWN: PING CRITICAL - Packet loss = 100% [09:05:01] !log marostegui@cumin1004 START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (remove config only) [09:05:07] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (remove config only) [09:05:25] RECOVERY - Host titan1002 is UP: PING OK - Packet loss = 0%, RTA = 0.15 ms [09:06:03] FIRING: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:07:12] !log arnaudb@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on gerrit2002.wikimedia.org with reason: T440177 offline reindex test [09:07:16] T440177: Add the the ability to warm caches to Gerrit switchover cookbook - https://phabricator.wikimedia.org/T440177 [09:07:29] !log marostegui@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage [09:08:19] !log cwilliams@cumin1004 START - Cookbook sre.mysql.pool pool es2036: Maintenance [09:08:31] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: es2036 crashed - https://phabricator.wikimedia.org/T443268#12514537 (10ops-monitoring-bot) Completed Setting section es6 as read-write by cwilliams@cumin1004: Maintenance [09:08:33] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: es2036 crashed - https://phabricator.wikimedia.org/T443268#12514539 (10ops-monitoring-bot) Starting pool of es2036 by cwilliams@cumin1004: Maintenance [09:11:03] RESOLVED: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:12:40] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage [09:13:52] !log jnuche@deploy2003 rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.3 refs T442221 [09:13:57] T442221: 1.48.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T442221 [09:14:00] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply [09:14:07] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply [09:21:41] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply [09:21:47] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply [09:26:16] (03CR) 10Hashar: "@dzahn@wikimedia.org that fixed it! Thank you 8-)" [puppet] - 10https://gerrit.wikimedia.org/r/1320916 (https://phabricator.wikimedia.org/T418521) (owner: 10Hashar) [09:27:41] (03CR) 10Gkyziridis: [C:03+2] ml-services: place jina-embeddings on 24G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361475 (https://phabricator.wikimedia.org/T443256) (owner: 10Gkyziridis) [09:29:24] (03Merged) 10jenkins-bot: ml-services: place jina-embeddings on 24G GPU partitions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361475 (https://phabricator.wikimedia.org/T443256) (owner: 10Gkyziridis) [09:31:04] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2006.codfw.wmnet with OS trixie [09:32:30] (03PS2) 10Arnaudb: gerrit: reindex the new primary offline before it starts [cookbooks] - 10https://gerrit.wikimedia.org/r/1357493 (https://phabricator.wikimedia.org/T440177) [09:32:30] (03CR) 10Arnaudb: "I ran the exact command of this PS on the spare gerrit2002, using the new Java 21: exit 0 in 1m45." [cookbooks] - 10https://gerrit.wikimedia.org/r/1357493 (https://phabricator.wikimedia.org/T440177) (owner: 10Arnaudb) [09:44:21] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply [09:45:17] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply [09:45:25] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply [09:45:27] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply [09:47:35] (03PS1) 10Marostegui: Revert "dbprov2005: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1362989 [09:48:49] !log marostegui@cumin1004 START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS trixie [09:49:45] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply [09:49:46] !log marostegui@cumin1004 START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (push config only) [09:49:46] (03PS7) 10Slyngshede: CAS 8.0.3 [software/cas-overlay-template] - 10https://gerrit.wikimedia.org/r/1346564 (https://phabricator.wikimedia.org/T439509) [09:49:50] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (push config only) [09:50:40] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply [09:51:01] (03CR) 10Awight: "Apparently the fido generation process overwrote my existing private key? So there is no harm in merging the patch here, I'm currently lo" [puppet] - 10https://gerrit.wikimedia.org/r/1361821 (https://phabricator.wikimedia.org/T442763) (owner: 10Awight) [09:52:26] (03PS12) 10Fabfur: profile:haproxy: kapow score setup [puppet] - 10https://gerrit.wikimedia.org/r/1328543 (https://phabricator.wikimedia.org/T435799) [09:53:29] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Maintenance [09:53:35] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: es2036 crashed - https://phabricator.wikimedia.org/T443268#12514638 (10ops-monitoring-bot) Completed pooling of es2036 by cwilliams@cumin1004: Maintenance [09:54:01] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply [09:54:07] !log brouberol@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply [09:57:49] (03PS3) 10Tiziano Fogli: metamonitoring: make public endpoint script aware of silenced DMS alerts [puppet] - 10https://gerrit.wikimedia.org/r/1361834 (https://phabricator.wikimedia.org/T443406) [09:58:08] (03PS1) 10Muehlenhoff: Add FIDO keys for andrew-wmde, catherinekelsey and seanleong-wmde [puppet] - 10https://gerrit.wikimedia.org/r/1363003 (https://phabricator.wikimedia.org/T442763) [09:58:39] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris [09:58:48] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris [09:59:00] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris [09:59:05] (03CR) 10Clément Goubert: [C:03+1] changeprop-jobqueue: Enable service mesh and mw-jobrunner listener [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325936 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [09:59:10] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris [09:59:19] (03CR) 10CI reject: [V:04-1] Add FIDO keys for andrew-wmde, catherinekelsey and seanleong-wmde [puppet] - 10https://gerrit.wikimedia.org/r/1363003 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [10:00:20] (03CR) 10Effie Mouzeli: [C:03+1] changeprop-jobqueue: Enable service mesh and mw-jobrunner listener [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325936 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [10:00:47] (03PS1) 10Ladsgroup: Better browser support for DASH [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363009 (https://phabricator.wikimedia.org/T442988) [10:01:16] (03CR) 10Blake: [C:03+2] changeprop-jobqueue: Enable service mesh and mw-jobrunner listener [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325936 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [10:02:07] jouncebot: nownadnext [10:02:16] jouncebot: nowandnext [10:02:16] For the next 0 hour(s) and 57 minute(s): MediaWiki train (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T0900) [10:02:16] In 0 hour(s) and 57 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1100) [10:02:48] (03PS4) 10Tiziano Fogli: metamonitoring: make public endpoint script aware of silenced DMS alerts [puppet] - 10https://gerrit.wikimedia.org/r/1361834 (https://phabricator.wikimedia.org/T443406) [10:02:59] (03Merged) 10jenkins-bot: changeprop-jobqueue: Enable service mesh and mw-jobrunner listener [deployment-charts] - 10https://gerrit.wikimedia.org/r/1325936 (https://phabricator.wikimedia.org/T434925) (owner: 10Scott French) [10:03:33] !log marostegui@cumin1004 START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (remove config only) [10:03:38] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (remove config only) [10:04:47] (03CR) 10Fabfur: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1328543 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [10:06:13] !log marostegui@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage [10:06:24] (03PS2) 10Muehlenhoff: Add FIDO keys for andrew-wmde, catherinekelsey and seanleong-wmde [puppet] - 10https://gerrit.wikimedia.org/r/1363003 (https://phabricator.wikimedia.org/T442763) [10:07:48] (03CR) 10Cathal Mooney: [C:03+2] lvs1020: add new vlan ints for rows a/b and move existing [puppet] - 10https://gerrit.wikimedia.org/r/1361867 (https://phabricator.wikimedia.org/T440859) (owner: 10Cathal Mooney) [10:09:39] !log blake@deploy2003 helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply [10:10:12] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO keys for andrew-wmde, catherinekelsey and seanleong-wmde [puppet] - 10https://gerrit.wikimedia.org/r/1363003 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [10:10:32] !log blake@deploy2003 helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply [10:10:51] !log blake@deploy2003 helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [10:11:58] !log blake@deploy2003 helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [10:12:20] !log cmooney@cumin1004 START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet [10:12:47] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12514701 (10ops-monitoring-bot) Host lvs1020.eqiad.wmnet rebooted by cmooney@cumin1004 with reason: apply vlan changes on lvs1020 [10:14:06] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage [10:17:30] !log blake@deploy2003 helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [10:17:41] (03CR) 10Muehlenhoff: "Not sure what you mean? Your current key should work as-is." [puppet] - 10https://gerrit.wikimedia.org/r/1361821 (https://phabricator.wikimedia.org/T442763) (owner: 10Awight) [10:18:29] !log blake@deploy2003 helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [10:19:13] (03PS1) 10Majavah: P:wmcs::novaproxy: Do not redirect from existing wmflabs.org proxies [puppet] - 10https://gerrit.wikimedia.org/r/1363024 (https://phabricator.wikimedia.org/T443531) [10:19:46] !log gkyziridis@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . [10:19:51] !log gkyziridis@deploy2003 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . [10:21:22] (03PS1) 10Jelto: etherpad: update image to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363025 (https://phabricator.wikimedia.org/T442177) [10:22:19] (03Abandoned) 10Btullis: hdfs: set an-worker1176 to reuse recipe [puppet] - 10https://gerrit.wikimedia.org/r/1163692 (https://phabricator.wikimedia.org/T390176) (owner: 10Stevemunene) [10:22:38] (03PS8) 10Majavah: P:wmcs::novaproxy: Test fetching data from requestctl [puppet] - 10https://gerrit.wikimedia.org/r/1362054 (https://phabricator.wikimedia.org/T409759) [10:23:44] (03CR) 10Majavah: P:wmcs::novaproxy: Test fetching data from requestctl (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1362054 (https://phabricator.wikimedia.org/T409759) (owner: 10Majavah) [10:24:48] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9595/co" [puppet] - 10https://gerrit.wikimedia.org/r/1362054 (https://phabricator.wikimedia.org/T409759) (owner: 10Majavah) [10:25:40] (03CR) 10Arnaudb: [C:03+1] "lgtm! image docker-registry.wikimedia.org/repos/sre/etherpad:2026-11-04-101552 exists and is pullable" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363025 (https://phabricator.wikimedia.org/T442177) (owner: 10Jelto) [10:27:19] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris [10:27:27] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris [10:28:00] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris [10:28:01] !log klausman@cumin1004 END (ERROR) - Cookbook sre.hadoop.check-user-leftovers (exit_code=97) Checking for leftover data for user akosiaris [10:28:04] !log klausman@cumin1004 START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris [10:28:13] !log klausman@cumin1004 END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris [10:31:05] (03CR) 10Muehlenhoff: [C:03+1] "Key has been verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1361821 (https://phabricator.wikimedia.org/T442763) (owner: 10Awight) [10:31:09] (03CR) 10Muehlenhoff: [C:03+2] Switch awight to a fido2 key [puppet] - 10https://gerrit.wikimedia.org/r/1361821 (https://phabricator.wikimedia.org/T442763) (owner: 10Awight) [10:31:37] (03PS5) 10Tiziano Fogli: metamonitoring: make public endpoint script aware of silenced DMS alerts [puppet] - 10https://gerrit.wikimedia.org/r/1361834 (https://phabricator.wikimedia.org/T443406) [10:32:36] !log marostegui@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS trixie [10:33:30] 06SRE, 10MediaWiki-General, 06ServiceOps, 07Epic: Introduce a global load shedding level - https://phabricator.wikimedia.org/T443236#12514777 (10jijiki) I like the idea a lot and I think it it is worth working out more fine grained targeted actions in order to keep serving our users while things happen. I... [10:34:21] !log cmooney@cumin1004 END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host lvs1020.eqiad.wmnet [10:34:36] (03CR) 10Filippo Giunchedi: "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/1363024 (https://phabricator.wikimedia.org/T443531) (owner: 10Majavah) [10:34:39] (03CR) 10Filippo Giunchedi: [C:03+1] P:wmcs::novaproxy: Do not redirect from existing wmflabs.org proxies [puppet] - 10https://gerrit.wikimedia.org/r/1363024 (https://phabricator.wikimedia.org/T443531) (owner: 10Majavah) [10:35:22] (03CR) 10Majavah: [C:03+2] P:wmcs::novaproxy: Do not redirect from existing wmflabs.org proxies [puppet] - 10https://gerrit.wikimedia.org/r/1363024 (https://phabricator.wikimedia.org/T443531) (owner: 10Majavah) [10:35:25] (03CR) 10David Caro: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1363024 (https://phabricator.wikimedia.org/T443531) (owner: 10Majavah) [10:39:52] (03PS13) 10Fabfur: profile:haproxy: kapow score setup [puppet] - 10https://gerrit.wikimedia.org/r/1328543 (https://phabricator.wikimedia.org/T435799) [10:41:47] jouncebot: nowandnext [10:41:48] For the next 0 hour(s) and 18 minute(s): MediaWiki train (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T0900) [10:41:48] In 0 hour(s) and 18 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1100) [10:41:59] jnuche: okay if I push a couple of changes? [10:43:57] (03CR) 10Vgutierrez: [C:03+1] profile:haproxy: kapow score setup [puppet] - 10https://gerrit.wikimedia.org/r/1328543 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [10:44:26] (03CR) 10Thibaut Le Page: [C:03+1] node22: add new pre-built image [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1362082 (https://phabricator.wikimedia.org/T443499) (owner: 10David Caro) [10:44:55] (03CR) 10David Caro: [C:03+2] node22: add new pre-built image [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1362082 (https://phabricator.wikimedia.org/T443499) (owner: 10David Caro) [10:45:37] (03Merged) 10jenkins-bot: node22: add new pre-built image [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1362082 (https://phabricator.wikimedia.org/T443499) (owner: 10David Caro) [10:45:49] (03CR) 10Fabfur: [C:03+2] profile:haproxy: kapow score setup [puppet] - 10https://gerrit.wikimedia.org/r/1328543 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [10:46:46] (03PS1) 10Daniel Kertesz: trafficserver: support migration to v10.x [puppet] - 10https://gerrit.wikimedia.org/r/1363051 (https://phabricator.wikimedia.org/T396611) [10:47:12] (03CR) 10Tiziano Fogli: metamonitoring: make public endpoint script aware of silenced DMS alerts (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1361834 (https://phabricator.wikimedia.org/T443406) (owner: 10Tiziano Fogli) [10:49:01] !log installing vim security updates [10:49:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:49:41] (03CR) 10Daniel Kertesz: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1363051 (https://phabricator.wikimedia.org/T396611) (owner: 10Daniel Kertesz) [10:50:45] FIRING: WidespreadPuppetFailure: Puppet has failed in magru - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [10:50:48] (03PS1) 10Fabfur: Revert "profile:haproxy: kapow score setup" [puppet] - 10https://gerrit.wikimedia.org/r/1363054 [10:51:50] (03PS1) 10Hashar: Update rubocop for ruby 3.3 support [puppet-lint/wmf_styleguide-check] - 10https://gerrit.wikimedia.org/r/1363055 (https://phabricator.wikimedia.org/T383337) [10:51:54] (03PS1) 10Hashar: Enable new cops in rubocop [puppet-lint/wmf_styleguide-check] - 10https://gerrit.wikimedia.org/r/1363056 (https://phabricator.wikimedia.org/T441847) [10:52:42] !log temporary disabling puppet on A:cp to fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328543 [10:52:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:54:29] (03CR) 10Ladsgroup: [C:03+2] Better browser support for DASH [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363009 (https://phabricator.wikimedia.org/T442988) (owner: 10Ladsgroup) [10:55:45] FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [10:55:50] (03Merged) 10jenkins-bot: Better browser support for DASH [extensions/TimedMediaHandler] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363009 (https://phabricator.wikimedia.org/T442988) (owner: 10Ladsgroup) [10:56:28] (03PS1) 10Clément Goubert: ProductionServices: Depool poolcounter1006 for reboot (#1/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363062 (https://phabricator.wikimedia.org/T441653) [10:56:29] (03PS1) 10Clément Goubert: ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363063 (https://phabricator.wikimedia.org/T441653) [10:56:29] (03PS1) 10Clément Goubert: ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363064 (https://phabricator.wikimedia.org/T441653) [10:56:30] (03PS1) 10Clément Goubert: ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363065 (https://phabricator.wikimedia.org/T441653) [10:56:31] (03PS1) 10Clément Goubert: ProductionServices: Repool poolcounter2006 after reboot (#5/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363066 (https://phabricator.wikimedia.org/T441653) [10:57:14] (03PS1) 10Fabfur: cache:haproxy: fix typo in puppetserver url [puppet] - 10https://gerrit.wikimedia.org/r/1363067 (https://phabricator.wikimedia.org/T435799) [10:58:26] (03CR) 10Blake: [C:03+1] ProductionServices: Depool poolcounter1006 for reboot (#1/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363062 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [10:58:38] 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade rubocop ruby linter on operations/puppet - https://phabricator.wikimedia.org/T441847#12514909 (10hashar) `operations/puppet-lint/wmf_styleguide-check` also has an outdated rubocop. We are dropping Bullseye from CI T383337 and thus ruby 2.7. I... [10:58:40] (03CR) 10Daniel Kertesz: [C:03+1] cache:haproxy: fix typo in puppetserver url [puppet] - 10https://gerrit.wikimedia.org/r/1363067 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [10:59:09] (03CR) 10Vgutierrez: [C:03+1] cache:haproxy: fix typo in puppetserver url [puppet] - 10https://gerrit.wikimedia.org/r/1363067 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [10:59:15] !log ladsgroup@deploy2003 Started scap sync-world: Backport for [[gerrit:1363009|Better browser support for DASH (T442988)]] (1.48.0-wmf.3) [10:59:20] T442988: TMH DASH unnecessary/aborted requests - https://phabricator.wikimedia.org/T442988 [10:59:33] (03CR) 10Blake: [C:03+1] ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363063 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [10:59:56] (03CR) 10Blake: [C:03+1] ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363064 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1100) [11:00:13] (03CR) 10Blake: [C:03+1] ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363065 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:00:25] (03CR) 10Blake: [C:03+1] ProductionServices: Repool poolcounter2006 after reboot (#5/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363066 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:00:44] (03CR) 10Fabfur: [C:03+2] cache:haproxy: fix typo in puppetserver url [puppet] - 10https://gerrit.wikimedia.org/r/1363067 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [11:00:45] FIRING: [5x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [11:00:55] Amir1: ping me when done, i'll be rebooting poolcounters during the infra window [11:01:35] Sure thing. I have a couple more non-urgent stuff afterwards, would you mind pinging me once I'm okay to use the window? :D [11:02:07] !log enabling puppet again on A:cp to apply fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363067 [11:02:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:04:00] !log ladsgroup@deploy2003 ladsgroup: Backport for [[gerrit:1363009|Better browser support for DASH (T442988)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:05:43] (03CR) 10Marostegui: [C:03+2] Revert "dbprov2005: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1362989 (owner: 10Marostegui) [11:06:12] !log ladsgroup@deploy2003 ladsgroup: Continuing with deployment [11:12:06] !log ladsgroup@deploy2003 Finished scap sync-world: Backport for [[gerrit:1363009|Better browser support for DASH (T442988)]] (1.48.0-wmf.3) (duration: 12m 51s) [11:12:11] T442988: TMH DASH unnecessary/aborted requests - https://phabricator.wikimedia.org/T442988 [11:12:20] claime: I'm done [11:12:28] Amir1: tyvm [11:12:41] FIRING: ConfdResourceFailed: confd resource _etc_haproxy_conf.d_tls.cfg.toml has errors - https://wikitech.wikimedia.org/wiki/Confd#Monitoring - https://grafana.wikimedia.org/d/OUJF1VI4k/confd - https://alerts.wikimedia.org/?q=alertname%3DConfdResourceFailed [11:14:47] !log Starting poolcounter reboot sequence [11:14:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:14:57] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cgoubert@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363062 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:15:03] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host an-mariadb1001.eqiad.wmnet [11:15:45] (03Merged) 10jenkins-bot: ProductionServices: Depool poolcounter1006 for reboot (#1/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363062 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:15:46] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [11:16:10] !log cgoubert@deploy2003 Started scap sync-world: Backport for [[gerrit:1363062|ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] [11:16:40] (03PS2) 10Samtar: IS: Enable watchstar popover everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356877 (https://phabricator.wikimedia.org/T417847) [11:17:01] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [11:17:24] PROBLEM - MariaDB Replica IO: analytics-meta-replica on an-mariadb1002 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl@10.64.138.8:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on 10.64.138.8 (111 Connection refused) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [11:17:46] PROBLEM - MariaDB Replica IO: analytics_meta on db1208 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl@10.64.138.8:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on 10.64.138.8 (111 Connection refused) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [11:17:54] (03PS1) 10Mszwarc: SECURITY: AbuseLogLookup: Don't count logs the user can't see [extensions/AbuseFilter] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1363090 (https://phabricator.wikimedia.org/T439885) [11:18:14] (03PS1) 10Mszwarc: SECURITY: AbuseLogLookup: Don't count logs the user can't see [extensions/AbuseFilter] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363091 (https://phabricator.wikimedia.org/T439885) [11:18:24] RECOVERY - MariaDB Replica IO: analytics-meta-replica on an-mariadb1002 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [11:18:29] !log cgoubert@deploy2003 cgoubert: Backport for [[gerrit:1363062|ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:18:46] RECOVERY - MariaDB Replica IO: analytics_meta on db1208 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [11:19:01] !log cgoubert@deploy2003 cgoubert: Continuing with deployment [11:20:03] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1262.eqiad.wmnet with reason: Reboot [11:20:20] (03CR) 10Clément Goubert: [C:03+2] ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363063 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:20:26] (03CR) 10Clément Goubert: [C:03+2] ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363064 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:20:33] (03CR) 10Clément Goubert: [C:03+2] ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363065 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:20:40] (03CR) 10Clément Goubert: [C:03+2] ProductionServices: Repool poolcounter2006 after reboot (#5/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363066 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:21:05] (03Merged) 10jenkins-bot: ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363063 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:21:08] (03Merged) 10jenkins-bot: ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363064 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:21:19] (03Merged) 10jenkins-bot: ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363065 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:21:31] (03Merged) 10jenkins-bot: ProductionServices: Repool poolcounter2006 after reboot (#5/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363066 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:21:40] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1001.eqiad.wmnet [11:24:49] !log cgoubert@deploy2003 Finished scap sync-world: Backport for [[gerrit:1363062|ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]] (duration: 08m 39s) [11:25:01] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet [11:25:45] RESOLVED: [5x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [11:26:27] (03CR) 10Hnowlan: [C:03+1] metamonitoring: make public endpoint script aware of silenced DMS alerts (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1361834 (https://phabricator.wikimedia.org/T443406) (owner: 10Tiziano Fogli) [11:27:25] (03CR) 10Volans: [C:03+1] "LGTM, thx" [puppet] - 10https://gerrit.wikimedia.org/r/1362054 (https://phabricator.wikimedia.org/T409759) (owner: 10Majavah) [11:28:39] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet [11:28:46] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Reboot [11:29:25] (03PS1) 10Fabfur: cache:haproxy: fix missing variable in template [puppet] - 10https://gerrit.wikimedia.org/r/1363100 (https://phabricator.wikimedia.org/T435799) [11:31:03] Amir1: sorry, somehow I didn't get the notification ping [11:31:12] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1363100 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [11:31:21] (03CR) 10Fabfur: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1363100 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [11:31:43] all good. no worries! [11:33:16] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12515024 (10Marostegui) I've changed db1262 to Performance: ` racadm>>get BIOS.SysProfileSettings.SysProfile [Key=BIOS.Setup.1-1#SysProfileSettings] SysProfile=PerfPerWattOptimizedOs racadm>>set BIOS.SysProfileS... [11:33:49] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12515029 (10Marostegui) @CWilliams-WMF good to close? [11:33:52] (03PS1) 10Clément Goubert: ProductionServices: Depool poolcounter1006 for reboot (#1/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363102 (https://phabricator.wikimedia.org/T441653) [11:33:52] (03PS1) 10Clément Goubert: ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363103 (https://phabricator.wikimedia.org/T441653) [11:33:53] (03PS1) 10Clément Goubert: ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363104 (https://phabricator.wikimedia.org/T441653) [11:33:53] (03PS1) 10Clément Goubert: ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363105 (https://phabricator.wikimedia.org/T441653) [11:33:54] (03PS1) 10Clément Goubert: ProductionServices: Repool poolcounter2006 after reboot (#5/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363106 (https://phabricator.wikimedia.org/T441653) [11:35:16] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance [11:35:18] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2175 (T441633)', diff saved to https://phabricator.wikimedia.org/P97256 and previous config saved to /var/cache/conftool/dbconfig/20261104-113517-ladsgroup.json [11:35:31] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cgoubert@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363102 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:35:32] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cgoubert@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363103 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:35:38] !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet [11:36:25] (03Merged) 10jenkins-bot: ProductionServices: Depool poolcounter1006 for reboot (#1/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363102 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:36:31] (03Merged) 10jenkins-bot: ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363103 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:36:42] (03CR) 10Hnowlan: [C:03+1] Clean up temp files when a request fails part way [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1357427 (https://phabricator.wikimedia.org/T431768) (owner: 10Ladsgroup-claude) [11:36:44] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1217 crashed - https://phabricator.wikimedia.org/T443208#12515036 (10CWilliams-WMF) 05Open→03Resolved @Marostegui yes, sure. [11:37:40] !log cgoubert@deploy2003 Started scap sync-world: Backport for [[gerrit:1363102|ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103|ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] [11:39:33] (03CR) 10Ladsgroup: [C:03+2] Clean up temp files when a request fails part way [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1357427 (https://phabricator.wikimedia.org/T431768) (owner: 10Ladsgroup-claude) [11:41:51] !log cgoubert@deploy2003 cgoubert: Backport for [[gerrit:1363102|ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103|ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [11:42:03] !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet [11:42:20] !log cgoubert@deploy2003 cgoubert: Continuing with deployment [11:42:51] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2175 (T441633)', diff saved to https://phabricator.wikimedia.org/P97257 and previous config saved to /var/cache/conftool/dbconfig/20261104-114251-ladsgroup.json [11:44:27] !log putting metamonitoring checks into "maintenance mode without sending a notification" to update the metamonitoring script T443406 [11:44:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:44:33] T443406: Make the metamonitoring script aware of Dead Man's Switch silences - https://phabricator.wikimedia.org/T443406 [11:45:49] (03CR) 10Tiziano Fogli: [C:03+2] metamonitoring: make public endpoint script aware of silenced DMS alerts [puppet] - 10https://gerrit.wikimedia.org/r/1361834 (https://phabricator.wikimedia.org/T443406) (owner: 10Tiziano Fogli) [11:48:04] (03Merged) 10jenkins-bot: Clean up temp files when a request fails part way [software/thumbor-plugins] - 10https://gerrit.wikimedia.org/r/1357427 (https://phabricator.wikimedia.org/T431768) (owner: 10Ladsgroup-claude) [11:48:05] !log cgoubert@deploy2003 Finished scap sync-world: Backport for [[gerrit:1363102|ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)]], [[gerrit:1363103|ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)]] (duration: 10m 24s) [11:49:14] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet [11:49:33] (03PS2) 10Fabfur: cache:haproxy: fix missing variable in template [puppet] - 10https://gerrit.wikimedia.org/r/1363100 (https://phabricator.wikimedia.org/T435799) [11:52:18] !log btullis@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw [11:52:22] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet [11:52:53] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97258 and previous config saved to /var/cache/conftool/dbconfig/20261104-115252-ladsgroup.json [11:52:58] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet [11:53:14] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet [11:53:39] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cgoubert@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363104 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:54:22] (03Merged) 10jenkins-bot: ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363104 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [11:54:45] !log cgoubert@deploy2003 Started scap sync-world: Backport for [[gerrit:1363104|ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] [11:55:01] !log blake@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw [11:55:05] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet [11:55:06] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet [11:56:55] (03PS3) 10Mvolz: citoid: Bump chrome version in user-agent in citoid [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358340 (https://phabricator.wikimedia.org/T442971) [11:57:00] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet [11:57:01] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet [11:57:08] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet [11:57:15] (03CR) 10Vgutierrez: [C:03+1] cache:haproxy: fix missing variable in template [puppet] - 10https://gerrit.wikimedia.org/r/1363100 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [11:57:20] (03CR) 10Mvolz: [C:03+2] citoid: Bump chrome version in user-agent in citoid [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358340 (https://phabricator.wikimedia.org/T442971) (owner: 10Mvolz) [11:57:44] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet [11:58:42] (03PS1) 10Tiziano Fogli: metamonitoring: adjust parameters passed to getHttpSession [puppet] - 10https://gerrit.wikimedia.org/r/1363133 (https://phabricator.wikimedia.org/T443406) [11:59:02] (03Merged) 10jenkins-bot: citoid: Bump chrome version in user-agent in citoid [deployment-charts] - 10https://gerrit.wikimedia.org/r/1358340 (https://phabricator.wikimedia.org/T442971) (owner: 10Mvolz) [11:59:16] !log cgoubert@deploy2003 cgoubert: Backport for [[gerrit:1363104|ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:00:05] mvolz: I, the Bot under the Fountain, call upon thee, The Deployer, to do Services – Citoid / Zotero deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1200). [12:00:06] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox: sync [12:00:12] (03CR) 10Tiziano Fogli: [C:03+2] metamonitoring: adjust parameters passed to getHttpSession [puppet] - 10https://gerrit.wikimedia.org/r/1363133 (https://phabricator.wikimedia.org/T443406) (owner: 10Tiziano Fogli) [12:00:30] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox: sync [12:00:34] !log root@deploy2003 helmfile [staging] START helmfile.d/services/shellbox-timeline: sync [12:00:54] (03CR) 10Rafka Aditia: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1350687 (https://phabricator.wikimedia.org/T440137) (owner: 10Amire80) [12:00:54] !log root@deploy2003 helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync [12:00:57] !log root@deploy2003 helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [12:01:10] !log root@deploy2003 helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [12:01:27] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet [12:01:29] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet [12:01:34] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet [12:01:35] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet [12:02:19] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [12:02:55] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97260 and previous config saved to /var/cache/conftool/dbconfig/20261104-120254-ladsgroup.json [12:03:02] (03CR) 10Fabfur: [C:03+2] cache:haproxy: fix missing variable in template [puppet] - 10https://gerrit.wikimedia.org/r/1363100 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [12:03:26] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet [12:03:28] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet [12:03:33] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet [12:03:42] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [12:03:58] (03CR) 10Hnowlan: "I'm happy with either option, do you have a strong preference? Happy to go with whatever works for you all best." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356269 (https://phabricator.wikimedia.org/T440988) (owner: 10Hnowlan) [12:04:09] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet [12:04:11] !log cgoubert@deploy2003 cgoubert: Continuing with deployment [12:04:26] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:06:32] (03CR) 10Hnowlan: [C:03+2] alerting: continue when alerting_hosts hits an error [software/spicerack] - 10https://gerrit.wikimedia.org/r/1355757 (https://phabricator.wikimedia.org/T395032) (owner: 10Hnowlan) [12:07:26] !log mvolz@deploy2003 Started scap deploy-service citoid: Update user-agent [12:07:26] !log mvolz@deploy2003 scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid [12:07:27] (03PS1) 10Brouberol: kafka/jumbo: update partition count for large codfw stream topics [puppet] - 10https://gerrit.wikimedia.org/r/1363140 (https://phabricator.wikimedia.org/T435284) [12:07:58] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet [12:08:00] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet [12:08:08] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet [12:08:40] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet [12:09:19] !log mvolz@deploy2003 scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid [12:09:34] (03CR) 10Btullis: [C:03+1] kafka/jumbo: update partition count for large codfw stream topics [puppet] - 10https://gerrit.wikimedia.org/r/1363140 (https://phabricator.wikimedia.org/T435284) (owner: 10Brouberol) [12:09:41] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet [12:09:42] (03Merged) 10jenkins-bot: alerting: continue when alerting_hosts hits an error [software/spicerack] - 10https://gerrit.wikimedia.org/r/1355757 (https://phabricator.wikimedia.org/T395032) (owner: 10Hnowlan) [12:09:43] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet [12:09:49] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet [12:10:00] (03CR) 10Brouberol: [C:03+2] kafka/jumbo: update partition count for large codfw stream topics [puppet] - 10https://gerrit.wikimedia.org/r/1363140 (https://phabricator.wikimedia.org/T435284) (owner: 10Brouberol) [12:10:05] !log cgoubert@deploy2003 Finished scap sync-world: Backport for [[gerrit:1363104|ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)]] (duration: 15m 19s) [12:10:24] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet [12:10:47] !log mvolz@deploy2003 scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid [12:10:49] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet [12:12:41] RESOLVED: ConfdResourceFailed: confd resource _etc_haproxy_conf.d_tls.cfg.toml has errors - https://wikitech.wikimedia.org/wiki/Confd#Monitoring - https://grafana.wikimedia.org/d/OUJF1VI4k/confd - https://alerts.wikimedia.org/?q=alertname%3DConfdResourceFailed [12:12:53] !log mvolz@deploy2003 Finished scap deploy-service citoid: Update user-agent (duration: 05m 26s) [12:12:57] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2175 (T441633)', diff saved to https://phabricator.wikimedia.org/P97261 and previous config saved to /var/cache/conftool/dbconfig/20261104-121256-ladsgroup.json [12:13:15] !log installing apt security updates [12:13:16] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance [12:13:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:13:18] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2189 (T441633)', diff saved to https://phabricator.wikimedia.org/P97262 and previous config saved to /var/cache/conftool/dbconfig/20261104-121318-ladsgroup.json [12:14:36] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet [12:15:04] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cgoubert@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363105 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [12:15:18] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet [12:15:20] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet [12:15:25] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet [12:15:55] (03Merged) 10jenkins-bot: ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363105 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [12:16:06] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet [12:16:08] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet [12:16:13] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2005.codfw.wmnet [12:16:15] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2005.codfw.wmnet [12:16:18] !log cgoubert@deploy2003 Started scap sync-world: Backport for [[gerrit:1363105|ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] [12:17:10] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [12:18:00] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [12:20:25] !log cgoubert@deploy2003 cgoubert: Backport for [[gerrit:1363105|ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:20:44] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet [12:21:03] !log cgoubert@deploy2003 cgoubert: Continuing with deployment [12:21:48] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2005.codfw.wmnet [12:21:50] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2005.codfw.wmnet [12:21:55] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet [12:22:31] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet [12:23:32] (03PS1) 10Fabfur: profile::cache,hiera: enable kapow score everywhere and log the var [puppet] - 10https://gerrit.wikimedia.org/r/1363149 (https://phabricator.wikimedia.org/T435799) [12:24:58] (03PS1) 10Slyngshede: P:tofurkey enable Tofurkey for remaining edge sites [puppet] - 10https://gerrit.wikimedia.org/r/1363151 (https://phabricator.wikimedia.org/T355446) [12:26:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [12:26:51] !log cgoubert@deploy2003 Finished scap sync-world: Backport for [[gerrit:1363105|ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)]] (duration: 10m 32s) [12:26:57] (03CR) 10Fabfur: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1363149 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [12:27:25] !log cgoubert@cumin2003 START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet [12:27:28] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet [12:27:30] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet [12:27:30] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw [12:27:54] !log blake@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad [12:27:58] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet [12:28:30] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet [12:28:32] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet [12:28:35] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet [12:28:35] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet [12:29:33] (03PS8) 10Slyngshede: CAS 8.0.3 [software/cas-overlay-template] - 10https://gerrit.wikimedia.org/r/1346564 (https://phabricator.wikimedia.org/T439509) [12:31:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [12:31:23] !log cgoubert@cumin2003 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet [12:32:09] (03CR) 10TrainBranchBot: [C:03+2] "Approved by cgoubert@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363106 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [12:33:01] (03Merged) 10jenkins-bot: ProductionServices: Repool poolcounter2006 after reboot (#5/5) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363106 (https://phabricator.wikimedia.org/T441653) (owner: 10Clément Goubert) [12:33:26] !log cgoubert@deploy2003 Started scap sync-world: Backport for [[gerrit:1363106|ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] [12:33:41] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet [12:34:25] (03PS1) 10Ladsgroup: Thumbor: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363156 (https://phabricator.wikimedia.org/T431768) [12:34:38] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet [12:34:39] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet [12:34:45] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet [12:37:52] !log cgoubert@deploy2003 cgoubert: Backport for [[gerrit:1363106|ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:38:38] !log cgoubert@deploy2003 cgoubert: Continuing with deployment [12:39:51] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet [12:40:11] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet [12:40:12] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet [12:40:21] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet [12:43:24] (03CR) 10Ladsgroup: [C:03+2] Thumbor: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363156 (https://phabricator.wikimedia.org/T431768) (owner: 10Ladsgroup) [12:44:09] jouncebot: next [12:44:09] In 1 hour(s) and 15 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1400) [12:44:24] !log cgoubert@deploy2003 Finished scap sync-world: Backport for [[gerrit:1363106|ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)]] (duration: 10m 58s) [12:45:25] (03PS3) 10Clément Goubert: eventschemas: allow any origin for the MediaWiki REST API schemas [puppet] - 10https://gerrit.wikimedia.org/r/1354903 (https://phabricator.wikimedia.org/T438209) (owner: 10Aghirelli) [12:45:26] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet [12:45:27] (03CR) 10Clément Goubert: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1354903 (https://phabricator.wikimedia.org/T438209) (owner: 10Aghirelli) [12:45:41] !log donne with poolcounter reboots [12:45:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:45:44] Amir1: ^ [12:45:46] (03Merged) 10jenkins-bot: Thumbor: Bump image [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363156 (https://phabricator.wikimedia.org/T431768) (owner: 10Ladsgroup) [12:46:01] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet [12:46:01] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet [12:46:07] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet [12:46:16] !log ladsgroup@deploy2003 helmfile [staging] START helmfile.d/services/thumbor: apply [12:46:35] !log ladsgroup@deploy2003 helmfile [staging] DONE helmfile.d/services/thumbor: apply [12:46:53] awesome. I push more changes soon [12:47:11] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet [12:47:19] !log ladsgroup@deploy2003 helmfile [eqiad] START helmfile.d/services/thumbor: apply [12:48:43] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 04 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360437 (https://phabricator.wikimedia.org/T438016) (owner: 10Neriah) [12:48:58] (03CR) 10Effie Mouzeli: [C:03+1] Configure cookie-based enrollment in PHP 8.5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362144 (https://phabricator.wikimedia.org/T441689) (owner: 10Kamila Součková) [12:50:31] !log ladsgroup@deploy2003 helmfile [eqiad] DONE helmfile.d/services/thumbor: apply [12:50:57] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet [12:50:59] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet [12:51:05] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet [12:51:38] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet [12:52:49] (03PS1) 10Dpogorzelski: keda: add KEDA 2.21.0 images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1363166 (https://phabricator.wikimedia.org/T436650) [12:53:02] (03CR) 10Jelto: [C:03+2] etherpad: update image to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363025 (https://phabricator.wikimedia.org/T442177) (owner: 10Jelto) [12:54:02] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet [12:54:03] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet [12:54:09] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet [12:54:17] (03PS2) 10Dpogorzelski: keda: add KEDA 2.21.0 images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1363166 (https://phabricator.wikimedia.org/T436650) [12:54:47] (03Merged) 10jenkins-bot: etherpad: update image to newest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363025 (https://phabricator.wikimedia.org/T442177) (owner: 10Jelto) [12:54:57] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet [12:55:32] (03CR) 10Clément Goubert: "Looks sound in principle, all examples of `map` i can find quote the regex, and I can't find documentation one way or another, so I would " [puppet] - 10https://gerrit.wikimedia.org/r/1354903 (https://phabricator.wikimedia.org/T438209) (owner: 10Aghirelli) [12:56:48] !log ladsgroup@deploy2003 helmfile [codfw] START helmfile.d/services/thumbor: apply [12:57:10] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet [12:57:12] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet [12:57:18] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet [12:57:22] !log ayounsi@cumin1004 START - Cookbook sre.dns.netbox [12:57:51] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet [12:58:48] !log ladsgroup@deploy2003 helmfile [codfw] DONE helmfile.d/services/thumbor: apply [13:01:15] !log ayounsi@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004" [13:01:18] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004" [13:01:19] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [13:01:24] !log blake@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet [13:01:25] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet [13:01:25] !log blake@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad [13:03:23] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet [13:03:25] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet [13:03:30] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet [13:04:03] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet [13:07:23] I'm going to do a backport. Does anyone mind? [13:08:05] no objection from train conductor [13:08:25] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mszwarc@deploy2003 using scap backport" [extensions/AbuseFilter] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1363090 (https://phabricator.wikimedia.org/T439885) (owner: 10Mszwarc) [13:08:25] (03CR) 10TrainBranchBot: [C:03+2] "Approved by mszwarc@deploy2003 using scap backport" [extensions/AbuseFilter] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363091 (https://phabricator.wikimedia.org/T439885) (owner: 10Mszwarc) [13:09:23] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet [13:09:25] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet [13:09:31] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet [13:09:44] (03Merged) 10jenkins-bot: SECURITY: AbuseLogLookup: Don't count logs the user can't see [extensions/AbuseFilter] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1363090 (https://phabricator.wikimedia.org/T439885) (owner: 10Mszwarc) [13:09:52] (03Merged) 10jenkins-bot: SECURITY: AbuseLogLookup: Don't count logs the user can't see [extensions/AbuseFilter] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363091 (https://phabricator.wikimedia.org/T439885) (owner: 10Mszwarc) [13:10:04] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet [13:10:15] (03PS3) 10Hnowlan: alerting: improve removing alertmanager silences [software/spicerack] - 10https://gerrit.wikimedia.org/r/1355701 (https://phabricator.wikimedia.org/T395032) [13:10:21] !log mszwarc@deploy2003 Started scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see (T439885) ([[gerrit:1363090|1.48.0-wmf.1]], [[gerrit:1363091|1.48.0-wmf.3]]) [13:13:02] (03CR) 10CI reject: [V:04-1] alerting: improve removing alertmanager silences [software/spicerack] - 10https://gerrit.wikimedia.org/r/1355701 (https://phabricator.wikimedia.org/T395032) (owner: 10Hnowlan) [13:13:42] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2189 (T441633)', diff saved to https://phabricator.wikimedia.org/P97263 and previous config saved to /var/cache/conftool/dbconfig/20261104-131341-ladsgroup.json [13:14:22] !log mszwarc@deploy2003 mszwarc: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see (T439885) ([[gerrit:1363090|1.48.0-wmf.1]], [[gerrit:1363091|1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [13:15:24] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet [13:15:26] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet [13:15:32] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet [13:15:41] !log mszwarc@deploy2003 mszwarc: Continuing with deployment [13:16:06] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet [13:17:15] !log installing expat security updates [13:17:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:21:10] (03PS1) 10Hashar: Revert "Gemfile: pin openssl below 4 instead of net-ssh below 8" [puppet] - 10https://gerrit.wikimedia.org/r/1363193 [13:21:26] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet [13:21:27] !log mszwarc@deploy2003 Finished scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see (T439885) ([[gerrit:1363090|1.48.0-wmf.1]], [[gerrit:1363091|1.48.0-wmf.3]]) (duration: 11m 06s) [13:21:27] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet [13:21:33] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet [13:21:42] (03CR) 10Effie Mouzeli: [C:03+1] mw-web: Scale next release to 10% of main [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362162 (https://phabricator.wikimedia.org/T441689) (owner: 10Kamila Součková) [13:22:09] Finished deploying [13:23:44] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97264 and previous config saved to /var/cache/conftool/dbconfig/20261104-132343-ladsgroup.json [13:26:37] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet [13:29:14] (03CR) 10Cathal Mooney: [C:03+1] "LGTM!" [homer/public] - 10https://gerrit.wikimedia.org/r/1361729 (https://phabricator.wikimedia.org/T443003) (owner: 10Ayounsi) [13:31:21] (03CR) 10Btullis: "Just wondering why you chose production-images for this, rather than GitLab-CI and blubber/kokkuri?" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1363166 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [13:31:57] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet [13:31:59] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet [13:32:05] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet [13:32:37] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet [13:33:37] (03CR) 10Jelto: [C:03+2] Revert "Gemfile: pin openssl below 4 instead of net-ssh below 8" [puppet] - 10https://gerrit.wikimedia.org/r/1363193 (owner: 10Hashar) [13:33:46] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97265 and previous config saved to /var/cache/conftool/dbconfig/20261104-133345-ladsgroup.json [13:39:53] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet [13:39:55] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet [13:40:00] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet [13:41:29] !log jelto@deploy2003 helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply [13:43:48] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2189 (T441633)', diff saved to https://phabricator.wikimedia.org/P97266 and previous config saved to /var/cache/conftool/dbconfig/20261104-134347-ladsgroup.json [13:44:03] (03PS1) 10Muehlenhoff: Remove old key of seanleong-wmde [puppet] - 10https://gerrit.wikimedia.org/r/1363213 (https://phabricator.wikimedia.org/T442763) [13:44:08] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance [13:45:03] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet [13:49:02] (03PS1) 10Tiziano Fogli: metamonitoring: add info-level logging for silences [puppet] - 10https://gerrit.wikimedia.org/r/1363218 (https://phabricator.wikimedia.org/T443406) [13:49:50] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance [13:49:52] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97267 and previous config saved to /var/cache/conftool/dbconfig/20261104-134951-ladsgroup.json [13:50:50] (03CR) 10Tiziano Fogli: [C:03+2] metamonitoring: add info-level logging for silences [puppet] - 10https://gerrit.wikimedia.org/r/1363218 (https://phabricator.wikimedia.org/T443406) (owner: 10Tiziano Fogli) [13:52:11] (03CR) 10Majavah: [V:03+1 C:03+2] P:wmcs::novaproxy: Test fetching data from requestctl [puppet] - 10https://gerrit.wikimedia.org/r/1362054 (https://phabricator.wikimedia.org/T409759) (owner: 10Majavah) [13:52:22] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet [13:52:24] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet [13:52:30] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet [13:52:42] (03PS1) 10Muehlenhoff: Record LDAP access for cfoster [puppet] - 10https://gerrit.wikimedia.org/r/1363221 [13:53:02] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet [13:53:48] (03CR) 10CI reject: [V:04-1] Record LDAP access for cfoster [puppet] - 10https://gerrit.wikimedia.org/r/1363221 (owner: 10Muehlenhoff) [13:55:20] (03CR) 10Muehlenhoff: [C:03+2] Remove old key of seanleong-wmde [puppet] - 10https://gerrit.wikimedia.org/r/1363213 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [13:57:02] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97268 and previous config saved to /var/cache/conftool/dbconfig/20261104-135701-ladsgroup.json [13:58:25] (03CR) 10Hashar: "I have manually triggered a postmerge action to refresh the doc after I have updated the CI image from Bullseye to Trixie (T383337)." [software/ecs] - 10https://gerrit.wikimedia.org/r/1280737 (https://phabricator.wikimedia.org/T423986) (owner: 10Cwhite) [13:59:24] (03CR) 10Kamila Součková: [C:03+2] mw-web: Scale next release to 10% of main [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362162 (https://phabricator.wikimedia.org/T441689) (owner: 10Kamila Součková) [14:00:05] urbanecm and TheresNoTime: gettimeofday() says it's time for UTC afternoon backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1400) [14:00:05] Nvdtn19, stephanebisson, Raine, and vs1: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [14:00:14] o/ [14:00:14] o/ [14:00:16] a lot of stuff to deploy! [14:00:23] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet [14:00:24] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet [14:00:25] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw [14:00:40] I can deploy the patches by non-deployers and then hand it over to stephanebisson and others to self-serve? [14:00:51] (I'm also happy to deploy other stuff, too) [14:01:02] That works for me [14:01:14] (03CR) 10Hnowlan: "recheck" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1355701 (https://phabricator.wikimedia.org/T395032) (owner: 10Hnowlan) [14:01:24] vs1: re https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1360437, have you checked `logout` is actually a valid value? [14:01:36] (03Merged) 10jenkins-bot: mw-web: Scale next release to 10% of main [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362162 (https://phabricator.wikimedia.org/T441689) (owner: 10Kamila Součková) [14:01:38] Nvdtn19: hey, around? [14:01:41] !log jelto@deploy2003 helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply [14:01:42] I'll need longer than usual to test my stuff so maybe clear the simple patches before. [14:01:47] (03PS5) 10Atsuko: airflow3: bypassing slow view [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362154 (https://phabricator.wikimedia.org/T443252) [14:01:48] stephanebisson: ack [14:01:51] o/ [14:01:59] Nvdtn19: hey, let's start with you [14:02:05] (03PS2) 10Nvdtn19: Re-enable NewUserMessage for viwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362080 (https://phabricator.wikimedia.org/T442485) [14:02:08] (03CR) 10Urbanecm: [C:03+2] Re-enable NewUserMessage for viwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362080 (https://phabricator.wikimedia.org/T442485) (owner: 10Nvdtn19) [14:03:00] (03Merged) 10jenkins-bot: Re-enable NewUserMessage for viwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362080 (https://phabricator.wikimedia.org/T442485) (owner: 10Nvdtn19) [14:03:45] !log urbanecm@deploy2003 Started scap sync-world: Backport for [[gerrit:1362080|Re-enable NewUserMessage for viwikivoyage (T442485)]] [14:03:50] T442485: Enabling NewUserMessage extension for vi.wikivoyage - https://phabricator.wikimedia.org/T442485 [14:03:55] Go ahead with nvdtn19's patch first, I'm rechecking [14:04:35] (03Abandoned) 10Muehlenhoff: Record LDAP access for cfoster [puppet] - 10https://gerrit.wikimedia.org/r/1363221 (owner: 10Muehlenhoff) [14:05:07] !log kamila@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-web: apply [14:05:53] !log kamila@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-web: apply [14:06:07] !log kamila@deploy2003 helmfile [eqiad] DONE helmfile.d/services/mw-web: apply [14:06:40] sorry, flailing... my patch can go along with someone else's [14:07:04] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97269 and previous config saved to /var/cache/conftool/dbconfig/20261104-140703-ladsgroup.json [14:07:07] Raine: in that case i can process it as well, as long as you promise it won't break anything :D [14:07:13] it's a no-op, so nothing to test there [14:07:22] (03CR) 10Urbanecm: [C:03+2] Configure cookie-based enrollment in PHP 8.5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362144 (https://phabricator.wikimedia.org/T441689) (owner: 10Kamila Součková) [14:07:37] urbanecm: I withdraw. [14:07:40] yes please, and thanks urbanecm <3 [14:07:51] vs1: what did you find? [14:08:00] no problem Raine [14:08:14] (03Merged) 10jenkins-bot: Configure cookie-based enrollment in PHP 8.5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362144 (https://phabricator.wikimedia.org/T441689) (owner: 10Kamila Součková) [14:08:21] !log urbanecm@deploy2003 urbanecm, nvdtn19: Backport for [[gerrit:1362080|Re-enable NewUserMessage for viwikivoyage (T442485)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:08:44] Nvdtn19: please test [14:09:04] I'll test now [14:09:17] urbanecm: you'll see a replicas bump to 80 in the -next release because I didn't get enough sleep to keep track of what time it is, it's safe to deploy [14:09:28] ack [14:09:38] good to know [14:09:43] (sorry, wanted to get it in before the window and failed) [14:09:53] urbanecm: Did a look on the wmf-config codebase and can't find any use of `logout` value. Will have to recheck mw docs. [14:10:01] no worries, as long as i know why scap complains, i'm good :D [14:10:49] I keep getting disconnected, I don't know why. [14:12:07] it seems libera web doesn't like you :-( [14:12:15] (03PS1) 10Brouberol: airflow: update airlow images [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363231 [14:12:52] vs1: fwiw, it indeed does seem logout is a valid right... so it might work [14:13:26] i'm not sure it would be intended to use like this though [14:13:39] so i'll leave a comment on the task and see where it goes [14:14:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:14:16] (03CR) 10Btullis: [C:03+1] airflow: update airlow images [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363231 (owner: 10Brouberol) [14:14:17] urbanecm: Any documentation? [14:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [14:14:43] (03CR) 10Brouberol: [C:03+2] airflow: update airlow images [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363231 (owner: 10Brouberol) [14:14:59] (03CR) 10Urbanecm: [C:04-1] "logout seems to be a special right. it is not available for users editing via botpasswords or OAuth, so this would restrict the namespace " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360437 (https://phabricator.wikimedia.org/T438016) (owner: 10Neriah) [14:15:00] I cant see it on doc.wikimedia or mediawikiwiki [14:15:21] !log urbanecm@deploy2003 urbanecm, nvdtn19: Continuing with deployment [14:15:35] urbanecm vs1: Patch seems not working on viwikivoyage yet. (For contexts, NewUserMessages is server-side) [14:15:50] Nvdtn19: what do you mean by "not working"? [14:15:51] !log urbanecm@deploy2003 Started scap sync-world: Backport for [[gerrit:1362080|Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144|Configure cookie-based enrollment in PHP 8.5 (T441689)]] [14:15:57] T442485: Enabling NewUserMessage extension for vi.wikivoyage - https://phabricator.wikimedia.org/T442485 [14:15:58] T441689: [PHP 8.5] Migrate production mw-web service to PHP 8.5 (stage 1: cookie-based) - https://phabricator.wikimedia.org/T441689 [14:16:08] (03PS1) 10Kamila Součková: trafficserver: enable mw-next-rounting [puppet] - 10https://gerrit.wikimedia.org/r/1363234 (https://phabricator.wikimedia.org/T441689) [14:16:27] urbanecm: Yesterday I and Nvdtn19 was here. [14:16:31] (03PS1) 10Muehlenhoff: Remove legacy key of catherinekelsey [puppet] - 10https://gerrit.wikimedia.org/r/1363235 (https://phabricator.wikimedia.org/T442763) [14:16:51] NUM need to be synced to production so that it can be tested. [14:17:06] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97270 and previous config saved to /var/cache/conftool/dbconfig/20261104-141705-ladsgroup.json [14:17:09] yeah, makes sense, there are a lot of jobs [14:17:11] i'll sync it [14:18:09] !log urbanecm@deploy2003 kamila, nvdtn19, urbanecm: Backport for [[gerrit:1362080|Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144|Configure cookie-based enrollment in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:18:20] (i just added Rai.ne's patch too, ignore) [14:18:53] (03PS1) 10Hnowlan: statsd: remove check [alerts] - 10https://gerrit.wikimedia.org/r/1363237 (https://phabricator.wikimedia.org/T443230) [14:19:01] (03PS1) 10Majavah: P:conftool::requestctl_client: Fetch group members from passwd [puppet] - 10https://gerrit.wikimedia.org/r/1363238 (https://phabricator.wikimedia.org/T443608) [14:19:12] (03PS2) 10Majavah: P:conftool::requestctl_client: Fetch group members from passwd [puppet] - 10https://gerrit.wikimedia.org/r/1363238 (https://phabricator.wikimedia.org/T443608) [14:19:20] !log urbanecm@deploy2003 kamila, nvdtn19, urbanecm: Continuing with deployment [14:20:02] (03CR) 10CI reject: [V:04-1] P:conftool::requestctl_client: Fetch group members from passwd [puppet] - 10https://gerrit.wikimedia.org/r/1363238 (https://phabricator.wikimedia.org/T443608) (owner: 10Majavah) [14:22:47] vs1: i posted a comment on the task, too [14:23:50] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy key of catherinekelsey [puppet] - 10https://gerrit.wikimedia.org/r/1363235 (https://phabricator.wikimedia.org/T442763) (owner: 10Muehlenhoff) [14:24:47] (03CR) 10Urbanecm: [C:03+2] Rename project namespace on Madurese Wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1350687 (https://phabricator.wikimedia.org/T440137) (owner: 10Amire80) [14:25:17] (03PS1) 10Majavah: Add fake WMCS WAF hiddenparma root token [labs/private] - 10https://gerrit.wikimedia.org/r/1363244 [14:25:35] (03Merged) 10jenkins-bot: Rename project namespace on Madurese Wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1350687 (https://phabricator.wikimedia.org/T440137) (owner: 10Amire80) [14:25:42] (03CR) 10Majavah: [V:03+2 C:03+2] Add fake WMCS WAF hiddenparma root token [labs/private] - 10https://gerrit.wikimedia.org/r/1363244 (owner: 10Majavah) [14:27:08] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97272 and previous config saved to /var/cache/conftool/dbconfig/20261104-142707-ladsgroup.json [14:27:28] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance [14:27:29] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2225 (T441633)', diff saved to https://phabricator.wikimedia.org/P97273 and previous config saved to /var/cache/conftool/dbconfig/20261104-142729-ladsgroup.json [14:27:59] (03CR) 10Tiziano Fogli: [C:03+1] swagger: Make alerting message more useful [alerts] - 10https://gerrit.wikimedia.org/r/1361473 (https://phabricator.wikimedia.org/T442710) (owner: 10Hnowlan) [14:28:33] (03PS1) 10Jforrester: wikifunctions: Upgrade evaluators from 2026-10-28-122530 to 2026-11-04-142036 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363247 (https://phabricator.wikimedia.org/T442213) [14:28:38] (03PS1) 10Jforrester: wikifunctions: Upgrade orchestrator from 2026-10-28-013937 to 2026-11-04-133739 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363248 (https://phabricator.wikimedia.org/T404869) [14:29:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [14:29:04] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 90.16% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [14:29:05] (03PS3) 10Majavah: P:conftool::requestctl_client: Fetch group members from passwd [puppet] - 10https://gerrit.wikimedia.org/r/1363238 (https://phabricator.wikimedia.org/T443608) [14:29:05] (03PS1) 10Majavah: P:wmcs::waf::hiddenparma: Enable requestctl CLI profile [puppet] - 10https://gerrit.wikimedia.org/r/1363249 (https://phabricator.wikimedia.org/T443608) [14:29:58] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9596/co" [puppet] - 10https://gerrit.wikimedia.org/r/1363238 (https://phabricator.wikimedia.org/T443608) (owner: 10Majavah) [14:30:29] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9598/co" [puppet] - 10https://gerrit.wikimedia.org/r/1363249 (https://phabricator.wikimedia.org/T443608) (owner: 10Majavah) [14:31:03] urbanecm are you still deploying? [14:31:22] (03PS2) 10Majavah: P:wmcs::waf::hiddenparma: Enable requestctl CLI profile [puppet] - 10https://gerrit.wikimedia.org/r/1363249 (https://phabricator.wikimedia.org/T443608) [14:31:23] stephanebisson: yes [14:31:49] Raine: hmm... scap said "release next failed"... [14:32:14] but the difference is the 80 vs 2 replicas you mentioned [14:32:19] so i guess i should retry? [14:32:21] urbanecm: did it time out? [14:32:26] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9599/co" [puppet] - 10https://gerrit.wikimedia.org/r/1363249 (https://phabricator.wikimedia.org/T443608) (owner: 10Majavah) [14:32:29] !log pt1979@cumin1004 START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: router upgrade, T416563] [14:32:34] T416563: eqsin: upgrade routers (2026) - https://phabricator.wikimedia.org/T416563 [14:32:36] this is the error i'm seeing https://www.irccloud.com/pastebin/fXb1GHIw/ [14:32:47] !log pt1979@cumin1004 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: router upgrade, T416563] [14:32:47] "UPGRADE FAILED: release next failed, and has been rolled back due to atomic being set: context deadline exceeded" [14:32:48] (03CR) 10Hnowlan: [C:03+2] swagger: Make alerting message more useful [alerts] - 10https://gerrit.wikimedia.org/r/1361473 (https://phabricator.wikimedia.org/T442710) (owner: 10Hnowlan) [14:33:03] right, so timeout... I was hoping that wouldn't be a problem anymore... [14:33:04] sorry [14:33:06] !log cwilliams@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet [14:33:16] lemme quickly bump those... [14:33:21] (03Abandoned) 10Dpogorzelski: keda: add KEDA 2.21.0 images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1363166 (https://phabricator.wikimedia.org/T436650) (owner: 10Dpogorzelski) [14:33:24] okay, waiting [14:34:29] (03CR) 10Daniel Kertesz: [C:03+1] profile::cache,hiera: enable kapow score everywhere and log the var [puppet] - 10https://gerrit.wikimedia.org/r/1363149 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [14:34:35] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2225 (T441633)', diff saved to https://phabricator.wikimedia.org/P97274 and previous config saved to /var/cache/conftool/dbconfig/20261104-143434-ladsgroup.json [14:36:09] cwilliams@cumin1004 upgrade-firmware (PID 640634) is awaiting input [14:36:25] (03Merged) 10jenkins-bot: swagger: Make alerting message more useful [alerts] - 10https://gerrit.wikimedia.org/r/1361473 (https://phabricator.wikimedia.org/T442710) (owner: 10Hnowlan) [14:37:14] (03PS6) 10Atsuko: airflow3: bypassing slow view [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362154 (https://phabricator.wikimedia.org/T443252) [14:37:50] (03PS7) 10Atsuko: airflow3: bypassing slow view [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362154 (https://phabricator.wikimedia.org/T443252) [14:38:35] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [14:39:23] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [14:39:35] (03PS4) 10Majavah: P:conftool::requestctl_client: Fetch group members from passwd [puppet] - 10https://gerrit.wikimedia.org/r/1363238 (https://phabricator.wikimedia.org/T443608) [14:39:35] (03PS3) 10Majavah: P:wmcs::waf::hiddenparma: Enable requestctl CLI profile [puppet] - 10https://gerrit.wikimedia.org/r/1363249 (https://phabricator.wikimedia.org/T443608) [14:39:35] (03PS1) 10Majavah: P:conftool::requestctl_client: Allow setting REQUESTCTL_BASE_URL [puppet] - 10https://gerrit.wikimedia.org/r/1363254 (https://phabricator.wikimedia.org/T443608) [14:39:43] (03PS1) 10Kamila Součková: mw-web: temporarily increase timeout and maxUnavailable for -next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363255 (https://phabricator.wikimedia.org/T439955) [14:40:33] (03CR) 10Kamila Součková: [V:03+2 C:03+2] "skipping CI as deployers are waiting for this" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363255 (https://phabricator.wikimedia.org/T439955) (owner: 10Kamila Součková) [14:42:16] !log pt1979@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt with reason: Junos upgrade [14:42:27] (03Merged) 10jenkins-bot: mw-web: temporarily increase timeout and maxUnavailable for -next [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363255 (https://phabricator.wikimedia.org/T439955) (owner: 10Kamila Součková) [14:42:37] (03PS2) 10Majavah: P:conftool::requestctl_client: Allow setting REQUESTCTL_BASE_URL [puppet] - 10https://gerrit.wikimedia.org/r/1363254 (https://phabricator.wikimedia.org/T443608) [14:42:37] (03PS4) 10Majavah: P:wmcs::waf::hiddenparma: Enable requestctl CLI profile [puppet] - 10https://gerrit.wikimedia.org/r/1363249 (https://phabricator.wikimedia.org/T443608) [14:43:04] !log kamila@deploy2003 helmfile [eqiad] START helmfile.d/services/mw-web: apply [14:43:11] urbanecm: my apologies, I shouldn't have done the replica bump so close to the deploy window :-/ I lost track of time [14:43:15] anyway, merged now [14:43:21] Raine: so i should retry? [14:43:22] yes [14:43:26] doing [14:43:38] (03PS8) 10Atsuko: airflow3: bypassing slow view [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362154 (https://phabricator.wikimedia.org/T443252) [14:44:37] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97275 and previous config saved to /var/cache/conftool/dbconfig/20261104-144436-ladsgroup.json [14:44:51] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9600/co" [puppet] - 10https://gerrit.wikimedia.org/r/1363249 (https://phabricator.wikimedia.org/T443608) (owner: 10Majavah) [14:45:07] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9601/co" [puppet] - 10https://gerrit.wikimedia.org/r/1363254 (https://phabricator.wikimedia.org/T443608) (owner: 10Majavah) [14:45:23] !log urbanecm@deploy2003 Finished scap sync-world: Backport for [[gerrit:1362080|Re-enable NewUserMessage for viwikivoyage (T442485)]], [[gerrit:1362144|Configure cookie-based enrollment in PHP 8.5 (T441689)]] (duration: 29m 32s) [14:45:28] seems it worked, much better [14:45:29] T442485: Enabling NewUserMessage extension for vi.wikivoyage - https://phabricator.wikimedia.org/T442485 [14:45:29] T441689: [PHP 8.5] Migrate production mw-web service to PHP 8.5 (stage 1: cookie-based) - https://phabricator.wikimedia.org/T441689 [14:45:29] urbanecm: Thanks. My patch is working as expected (https://vi.wikivoyage.org/wiki/Th%E1%BA%A3o_lu%E1%BA%ADn_Th%C3%A0nh_vi%C3%AAn:UK9sd32t4). You can continue [14:45:30] thanks Raine [14:45:47] sending out the other patch i merged, then i'll hand it to stephanebisson [14:46:00] !log urbanecm@deploy2003 Started scap sync-world: Backport for [[gerrit:1350687|Rename project namespace on Madurese Wiktionary (T440137)]] [14:46:05] T440137: Rename project namespace of mad.wiktionary from "Wikikamus" to "Wikioca'" - https://phabricator.wikimedia.org/T440137 [14:46:44] (03PS1) 10Kosta Harlan: AbuseReview: Add undo and rollback links to all queues [extensions/WikimediaAntiAbuse] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363260 (https://phabricator.wikimedia.org/T443591) [14:47:13] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 04 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [extensions/WikimediaAntiAbuse] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363260 (https://phabricator.wikimedia.org/T443591) (owner: 10Kosta Harlan) [14:47:22] urbanecm: np, and sorry for the flail [14:47:28] I've added one more patch to the window, but can move it to later if needed [14:47:29] it happens to all of us [14:47:39] kostajh: there very likely won't be time for it, stephanebisson is still pending [14:47:52] ok [14:48:33] !log urbanecm@deploy2003 urbanecm, amire80: Backport for [[gerrit:1350687|Rename project namespace on Madurese Wiktionary (T440137)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [14:49:07] !log cwilliams@cumin1004 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1176.eqiad.wmnet [14:49:32] (03CR) 10Atsuko: "Actual patch: https://phabricator.wikimedia.org/P97276" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362154 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [14:49:54] !log urbanecm@deploy2003 urbanecm, amire80: Continuing with deployment [14:52:07] (03CR) 10Clément Goubert: [C:03+1] revision feature counts lac-lambda: reduce cpu and memory [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362476 (https://phabricator.wikimedia.org/T442914) (owner: 10AKhatun) [14:52:43] (03PS1) 10Ejegg: Update CentralNotice for new mixin [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363263 [14:53:40] (03CR) 10Ottomata: [C:03+1] "Didn't review closely but thank you!" [alerts] - 10https://gerrit.wikimedia.org/r/1362469 (https://phabricator.wikimedia.org/T442914) (owner: 10AKhatun) [14:54:00] (03PS1) 10Ejegg: Update CentralNotice for new mixin [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1363265 [14:54:03] (03CR) 10Ottomata: [C:03+1] revision feature counts lac-lambda: reduce cpu and memory [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362476 (https://phabricator.wikimedia.org/T442914) (owner: 10AKhatun) [14:54:39] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97277 and previous config saved to /var/cache/conftool/dbconfig/20261104-145438-ladsgroup.json [14:54:47] (03PS4) 10Aghirelli: eventschemas: allow CORS requests from any origin [puppet] - 10https://gerrit.wikimedia.org/r/1354903 (https://phabricator.wikimedia.org/T438209) [14:55:26] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 04 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1363265 (owner: 10Ejegg) [14:55:33] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: es2036 crashed - https://phabricator.wikimedia.org/T443268#12516213 (10CWilliams-WMF) @Marostegui do you want this left open for any reason, or good to close? [14:55:39] (03CR) 10CI reject: [V:04-1] eventschemas: allow CORS requests from any origin [puppet] - 10https://gerrit.wikimedia.org/r/1354903 (https://phabricator.wikimedia.org/T438209) (owner: 10Aghirelli) [14:55:50] jouncebot next [14:55:50] In 0 hour(s) and 4 minute(s): Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1500) [14:55:52] (03PS9) 10Atsuko: airflow3: bypassing slow view [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362154 (https://phabricator.wikimedia.org/T443252) [14:57:11] (03PS1) 10Daniel Kertesz: trafficserver: remove support for running multiple instances [puppet] - 10https://gerrit.wikimedia.org/r/1363267 (https://phabricator.wikimedia.org/T443596) [14:57:14] !log urbanecm@deploy2003 Finished scap sync-world: Backport for [[gerrit:1350687|Rename project namespace on Madurese Wiktionary (T440137)]] (duration: 11m 14s) [14:57:19] T440137: Rename project namespace of mad.wiktionary from "Wikikamus" to "Wikioca'" - https://phabricator.wikimedia.org/T440137 [14:57:19] stephanebisson: over to you [14:57:24] apologies it took longer than expected :-/ [14:57:50] Argh. [14:58:03] urbanecm thanks [14:58:08] stephanebisson: Just MW-land? We can deploy in parallel. [14:58:26] James_F mw-config [14:58:32] Ack, go for it. [14:58:35] yew, i think we can [14:58:41] *yes [14:58:52] Starting... [14:59:05] !log urbanecm@deploy2003 mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary # T440137 [14:59:19] !log urbanecm@deploy2003 mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --fix # T440137 [14:59:21] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbisson@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357358 (https://phabricator.wikimedia.org/T442164) (owner: 10Sbisson) [14:59:22] (03CR) 10TrainBranchBot: [C:03+2] "Approved by sbisson@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362118 (https://phabricator.wikimedia.org/T442035) (owner: 10Sbisson) [14:59:32] !log urbanecm@deploy2003 mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary # T440137 [14:59:34] (03CR) 10Jforrester: [C:03+2] wikifunctions: Upgrade evaluators from 2026-10-28-122530 to 2026-11-04-142036 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363247 (https://phabricator.wikimedia.org/T442213) (owner: 10Jforrester) [15:00:05] !log urbanecm@deploy2003 mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --add-prefix=BROKEN --fix # T440137 [15:00:05] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1500) [15:00:17] (03Merged) 10jenkins-bot: ArticleGuidance: Enable the reference risk model in production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1357358 (https://phabricator.wikimedia.org/T442164) (owner: 10Sbisson) [15:00:20] !log urbanecm@deploy2003 mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary # T440137 [15:00:20] (03Merged) 10jenkins-bot: Load ArticleGuidance on all Wikipedias [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362118 (https://phabricator.wikimedia.org/T442035) (owner: 10Sbisson) [15:00:45] !log sbisson@deploy2003 Started scap sync-world: Backport for [[gerrit:1357358|ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118|Load ArticleGuidance on all Wikipedias (T442035)]] [15:00:51] T442164: Sources step: Reject risky sources identified by the reference-risk model - https://phabricator.wikimedia.org/T442164 [15:00:51] T442035: Article Guidance Community Configuration dark launch - https://phabricator.wikimedia.org/T442035 [15:00:58] (03Merged) 10jenkins-bot: wikifunctions: Upgrade evaluators from 2026-10-28-122530 to 2026-11-04-142036 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363247 (https://phabricator.wikimedia.org/T442213) (owner: 10Jforrester) [15:01:11] (03PS5) 10Aghirelli: eventschemas: allow CORS requests from any origin [puppet] - 10https://gerrit.wikimedia.org/r/1354903 (https://phabricator.wikimedia.org/T438209) [15:01:13] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 04 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363263 (owner: 10Ejegg) [15:01:20] (03CR) 10Aghirelli: eventschemas: allow CORS requests from any origin (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1354903 (https://phabricator.wikimedia.org/T438209) (owner: 10Aghirelli) [15:01:27] !log jforrester@deploy2003 Started scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first (1363247) [15:01:27] !log jforrester@deploy2003 scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions [15:01:32] (03PS1) 10Kamila Součková: Enroll 1% of client sessions in PHP 8.5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363270 (https://phabricator.wikimedia.org/T441689) [15:01:35] (03CR) 10Vgutierrez: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1363267 (https://phabricator.wikimedia.org/T443596) (owner: 10Daniel Kertesz) [15:02:02] (03CR) 10CI reject: [V:04-1] eventschemas: allow CORS requests from any origin [puppet] - 10https://gerrit.wikimedia.org/r/1354903 (https://phabricator.wikimedia.org/T438209) (owner: 10Aghirelli) [15:02:33] !log jforrester@deploy2003 scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions [15:02:50] (03CR) 10Vgutierrez: trafficserver: remove support for running multiple instances (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363267 (https://phabricator.wikimedia.org/T443596) (owner: 10Daniel Kertesz) [15:03:22] !log jforrester@deploy2003 scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions [15:03:23] (03CR) 10Vgutierrez: trafficserver: remove support for running multiple instances (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363267 (https://phabricator.wikimedia.org/T443596) (owner: 10Daniel Kertesz) [15:03:40] !log cwilliams@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 8 hosts with reason: Testing [15:03:55] !log jforrester@deploy2003 Finished scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first (1363247) (duration: 02m 28s) [15:04:01] (03CR) 10Jforrester: [C:03+2] wikifunctions: Upgrade orchestrator from 2026-10-28-013937 to 2026-11-04-133739 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363248 (https://phabricator.wikimedia.org/T404869) (owner: 10Jforrester) [15:04:41] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2225 (T441633)', diff saved to https://phabricator.wikimedia.org/P97278 and previous config saved to /var/cache/conftool/dbconfig/20261104-150440-ladsgroup.json [15:05:00] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance [15:05:02] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2226 (T441633)', diff saved to https://phabricator.wikimedia.org/P97279 and previous config saved to /var/cache/conftool/dbconfig/20261104-150502-ladsgroup.json [15:05:05] !log start maintenance on cr2-eqsin [15:05:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:05:30] (03Merged) 10jenkins-bot: wikifunctions: Upgrade orchestrator from 2026-10-28-013937 to 2026-11-04-133739 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363248 (https://phabricator.wikimedia.org/T404869) (owner: 10Jforrester) [15:05:48] !log sbisson@deploy2003 sbisson: Backport for [[gerrit:1357358|ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118|Load ArticleGuidance on all Wikipedias (T442035)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:05:54] T442164: Sources step: Reject risky sources identified by the reference-risk model - https://phabricator.wikimedia.org/T442164 [15:05:55] T442035: Article Guidance Community Configuration dark launch - https://phabricator.wikimedia.org/T442035 [15:06:00] (03CR) 10Daniel Kertesz: trafficserver: remove support for running multiple instances (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363267 (https://phabricator.wikimedia.org/T443596) (owner: 10Daniel Kertesz) [15:06:29] !log jforrester@deploy2003 Started scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second (1363248) [15:06:30] !log jforrester@deploy2003 scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions [15:06:54] !log jforrester@deploy2003 scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions [15:07:20] !log sbisson@deploy2003 sbisson: Continuing with deployment [15:07:39] !log jforrester@deploy2003 scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions [15:08:01] !log cwilliams@cumin1004 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet [15:08:16] !log jforrester@deploy2003 Finished scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second (1363248) (duration: 01m 46s) [15:09:54] (03CR) 10Clément Goubert: eventschemas: allow CORS requests from any origin (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1354903 (https://phabricator.wikimedia.org/T438209) (owner: 10Aghirelli) [15:10:31] (03CR) 10MSantos: [C:03+2] Add my public key to mediawiki.org/keys [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362130 (https://phabricator.wikimedia.org/T441081) (owner: 10ArielGlenn) [15:10:45] !log cwilliams@cumin1004 START - Cookbook sre.hosts.reboot-single for host db1176.eqiad.wmnet [15:11:26] (03CR) 10MSantos: [C:03+2] "As the PM for the release working group I'm approving this, but if it's stepping on someone else's toes, I apologize in advance." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362130 (https://phabricator.wikimedia.org/T441081) (owner: 10ArielGlenn) [15:11:32] (03PS1) 10Gehel: admin: adding William Brown as contact for novemlinguae [puppet] - 10https://gerrit.wikimedia.org/r/1363277 (https://phabricator.wikimedia.org/T442756) [15:11:33] (03Merged) 10jenkins-bot: Add my public key to mediawiki.org/keys [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1362130 (https://phabricator.wikimedia.org/T441081) (owner: 10ArielGlenn) [15:11:51] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2226 (T441633)', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20261104-151151-ladsgroup.json [15:11:58] !log btullis@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad [15:12:02] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet [15:12:13] (03PS2) 10Gehel: admin: adding William Brown as contact for novemlinguae [puppet] - 10https://gerrit.wikimedia.org/r/1363277 (https://phabricator.wikimedia.org/T442756) [15:12:18] (03PS10) 10Atsuko: airflow3: bypassing slow view [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362154 (https://phabricator.wikimedia.org/T443252) [15:12:21] (03CR) 10CI reject: [V:04-1] admin: adding William Brown as contact for novemlinguae [puppet] - 10https://gerrit.wikimedia.org/r/1363277 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [15:12:37] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet [15:13:05] (03CR) 10CI reject: [V:04-1] admin: adding William Brown as contact for novemlinguae [puppet] - 10https://gerrit.wikimedia.org/r/1363277 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [15:13:14] !log sbisson@deploy2003 Finished scap sync-world: Backport for [[gerrit:1357358|ArticleGuidance: Enable the reference risk model in production (T442164)]], [[gerrit:1362118|Load ArticleGuidance on all Wikipedias (T442035)]] (duration: 12m 29s) [15:13:20] T442164: Sources step: Reject risky sources identified by the reference-risk model - https://phabricator.wikimedia.org/T442164 [15:13:20] T442035: Article Guidance Community Configuration dark launch - https://phabricator.wikimedia.org/T442035 [15:13:42] (03CR) 10Dreamy Jazz: admin: adding William Brown as contact for novemlinguae (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363277 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [15:14:09] I'm done [15:16:03] (03CR) 10BPirkle: rest-gateway: front wdqs v2 endpoints in shadow mode (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333815 (https://phabricator.wikimedia.org/T436778) (owner: 10Gmodena) [15:16:10] (03CR) 10Brouberol: [C:03+1] "Hacky, but what çan we do? LG" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362154 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [15:16:42] FIRING: JobUnavailable: Reduced availability for job mysql-test in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:17:12] (03PS3) 10Gehel: admin: adding William Brown as contact for novemlinguae [puppet] - 10https://gerrit.wikimedia.org/r/1363277 (https://phabricator.wikimedia.org/T442756) [15:17:39] (03CR) 10Gehel: admin: adding William Brown as contact for novemlinguae (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363277 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [15:17:49] (03CR) 10Dreamy Jazz: admin: adding William Brown as contact for novemlinguae (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363277 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [15:18:04] (03CR) 10CI reject: [V:04-1] admin: adding William Brown as contact for novemlinguae [puppet] - 10https://gerrit.wikimedia.org/r/1363277 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [15:18:18] (03CR) 10Dreamy Jazz: [C:03+1] "Tests don't seem to like how it's being added? However, I'm fine with being contact for novemlinguae" [puppet] - 10https://gerrit.wikimedia.org/r/1363277 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [15:18:36] urbanecm: can I sync my patch now? [15:18:37] (03CR) 10Klausman: [C:03+2] services_proxy: raise tts-section-generator timeouts to 30 minutes [puppet] - 10https://gerrit.wikimedia.org/r/1362111 (https://phabricator.wikimedia.org/T441983) (owner: 10Kevin Bazira) [15:18:44] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1176.eqiad.wmnet [15:18:46] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1176.eqiad.wmnet [15:19:04] (03PS4) 10Gehel: admin: adding William Brown as contact for novemlinguae [puppet] - 10https://gerrit.wikimedia.org/r/1363277 (https://phabricator.wikimedia.org/T442756) [15:19:37] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet [15:19:38] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet [15:19:44] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet [15:19:58] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy2003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363260 (https://phabricator.wikimedia.org/T443591) (owner: 10Kosta Harlan) [15:20:16] (03CR) 10Gehel: [C:03+2] admin: adding William Brown as contact for novemlinguae [puppet] - 10https://gerrit.wikimedia.org/r/1363277 (https://phabricator.wikimedia.org/T442756) (owner: 10Gehel) [15:20:19] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet [15:20:58] (03Merged) 10jenkins-bot: AbuseReview: Add undo and rollback links to all queues [extensions/WikimediaAntiAbuse] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363260 (https://phabricator.wikimedia.org/T443591) (owner: 10Kosta Harlan) [15:21:53] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97280 and previous config saved to /var/cache/conftool/dbconfig/20261104-152152-ladsgroup.json [15:22:12] !log kharlan@deploy2003 Started scap sync-world: Backport for [[gerrit:1363260|AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) [15:22:29] PROBLEM - OSPF status on cr3-eqsin is CRITICAL: OSPFv2: 2/4 UP : OSPFv3: 2/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [15:22:31] PROBLEM - OSPF status on cr1-codfw is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [15:22:31] PROBLEM - OSPF status on cr1-eqiad is CRITICAL: OSPFv2: 6/7 UP : OSPFv3: 6/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [15:22:45] (03PS1) 10Jelto: sre.k8s.upgrade-apiservers-inplace: hold kubernetes-node and kubernetes-master [cookbooks] - 10https://gerrit.wikimedia.org/r/1363285 (https://phabricator.wikimedia.org/T437480) [15:23:10] (03PS2) 10Gehel: data: add novemlinguae's yubikey [puppet] - 10https://gerrit.wikimedia.org/r/1360846 (https://phabricator.wikimedia.org/T442756) (owner: 10Novem Linguae) [15:23:10] FIRING: BFDdown: BFD session down between cr1-codfw and 103.102.166.149 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:23:39] FIRING: CoreBGPDown: Core BGP session down between cr1-codfw and cr2-eqsin (103.102.166.149) - group Confed_eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqsin&var-bgp_neighbor=cr2-eqsin - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:23:51] FIRING: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-603-eqsin:ethernet-1/55 (Core: cr2-eqsin:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [15:24:30] !log kharlan@deploy2003 kharlan: Backport for [[gerrit:1363260|AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:24:35] T443591: Add undo and rollback links for all review queues in Special:AbuseReview - https://phabricator.wikimedia.org/T443591 [15:24:48] (03PS1) 10Atsuko: airflow: typo in airflow image name [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363287 [15:25:08] !log kharlan@deploy2003 kharlan: Continuing with deployment [15:25:41] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet [15:25:42] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet [15:25:48] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet [15:26:15] (03PS6) 10Aghirelli: eventschemas: allow CORS requests from any origin [puppet] - 10https://gerrit.wikimedia.org/r/1354903 (https://phabricator.wikimedia.org/T438209) [15:26:16] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply [15:26:21] (03CR) 10Aghirelli: eventschemas: allow CORS requests from any origin (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1354903 (https://phabricator.wikimedia.org/T438209) (owner: 10Aghirelli) [15:26:23] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet [15:26:37] (03CR) 10Atsuko: [C:03+2] airflow: typo in airflow image name [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363287 (owner: 10Atsuko) [15:26:42] RESOLVED: JobUnavailable: Reduced availability for job mysql-test in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:27:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [15:27:56] (03Merged) 10jenkins-bot: airflow: typo in airflow image name [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363287 (owner: 10Atsuko) [15:28:00] FIRING: [2x] PingLossPercent: Blackbox probe packet loss 11.88% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [15:28:10] FIRING: [3x] BFDdown: BFD session down between cr1-codfw and 103.102.166.149 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:28:49] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply [15:29:02] (03CR) 10Dreamy Jazz: [C:03+1] "I have confirmed that novemlinguae posted this SSH key by asking them on their Discord account (that I know to be theirs)" [puppet] - 10https://gerrit.wikimedia.org/r/1360846 (https://phabricator.wikimedia.org/T442756) (owner: 10Novem Linguae) [15:29:48] (03PS11) 10Atsuko: airflow3: bypassing slow view [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362154 (https://phabricator.wikimedia.org/T443252) [15:30:04] Deploy window Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1500) [15:30:05] Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1530) [15:30:29] RECOVERY - OSPF status on cr3-eqsin is OK: OSPFv2: 4/4 UP : OSPFv3: 4/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [15:30:31] RECOVERY - OSPF status on cr1-codfw is OK: OSPFv2: 8/8 UP : OSPFv3: 8/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [15:30:31] RECOVERY - OSPF status on cr1-eqiad is OK: OSPFv2: 7/7 UP : OSPFv3: 7/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [15:31:27] !log kharlan@deploy2003 Finished scap sync-world: Backport for [[gerrit:1363260|AbuseReview: Add undo and rollback links to all queues (T443591)]] (1.48.0-wmf.3) (duration: 09m 14s) [15:31:31] T443591: Add undo and rollback links for all review queues in Special:AbuseReview - https://phabricator.wikimedia.org/T443591 [15:31:55] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97281 and previous config saved to /var/cache/conftool/dbconfig/20261104-153154-ladsgroup.json [15:32:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [15:33:10] RESOLVED: [3x] BFDdown: BFD session down between cr1-codfw and 103.102.166.149 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [15:33:20] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet [15:33:21] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet [15:33:27] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1004.eqiad.wmnet [15:33:27] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1004.eqiad.wmnet [15:33:39] RESOLVED: CoreBGPDown: Core BGP session down between cr1-codfw and cr2-eqsin (103.102.166.149) - group Confed_eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cr1-codfw:9804&var-bgp_group=Confed_eqsin&var-bgp_neighbor=cr2-eqsin - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [15:33:51] RESOLVED: [2x] SwitchCoreInterfaceDown: Switch core interface down - asw1-603-eqsin:ethernet-1/55 (Core: cr2-eqsin:et-0/0/1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [15:34:43] (03PS2) 10Daniel Kertesz: trafficserver: remove support for running multiple instances [puppet] - 10https://gerrit.wikimedia.org/r/1363267 (https://phabricator.wikimedia.org/T443596) [15:37:18] (03CR) 10Ssingh: "Question: since we do plan on rolling this out everywhere, isn't this better suited for hieradata/role/common/cache/upload.yaml and hierad" [puppet] - 10https://gerrit.wikimedia.org/r/1363151 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [15:37:19] (03CR) 10Daniel Kertesz: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1363267 (https://phabricator.wikimedia.org/T443596) (owner: 10Daniel Kertesz) [15:37:58] jouncebot: nowandnext [15:37:58] For the next 0 hour(s) and 22 minute(s): Wikifunctions Services UTC Afternoon (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1500) [15:37:58] For the next 0 hour(s) and 22 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1530) [15:37:58] In 2 hour(s) and 22 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1800) [15:38:08] We’re done with our window. [15:38:10] (WF) [15:38:31] thanks James_F :-) I'm not ready yet, just eyeballing things [15:39:04] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [15:39:09] (03CR) 10Daniel Kertesz: trafficserver: remove support for running multiple instances (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1363267 (https://phabricator.wikimedia.org/T443596) (owner: 10Daniel Kertesz) [15:39:15] (03CR) 10Clément Goubert: [C:03+1] eventschemas: allow CORS requests from any origin [puppet] - 10https://gerrit.wikimedia.org/r/1354903 (https://phabricator.wikimedia.org/T438209) (owner: 10Aghirelli) [15:39:28] (03CR) 10CDanis: "Let's discuss this at the summit perhaps?" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1356527 (https://phabricator.wikimedia.org/T438896) (owner: 10CDanis) [15:40:16] !log atsuko@deploy2003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [15:40:22] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1004.eqiad.wmnet [15:40:23] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1004.eqiad.wmnet [15:40:24] (03CR) 10Atsuko: [C:03+2] "Taking previously applied +1 back" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362154 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [15:40:28] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet [15:41:03] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet [15:41:57] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2226 (T441633)', diff saved to https://phabricator.wikimedia.org/P97282 and previous config saved to /var/cache/conftool/dbconfig/20261104-154156-ladsgroup.json [15:42:17] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance [15:42:18] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2238 (T441633)', diff saved to https://phabricator.wikimedia.org/P97283 and previous config saved to /var/cache/conftool/dbconfig/20261104-154218-ladsgroup.json [15:42:22] (03Merged) 10jenkins-bot: airflow3: bypassing slow view [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362154 (https://phabricator.wikimedia.org/T443252) (owner: 10Atsuko) [15:44:25] (03CR) 10Clément Goubert: [C:03+1] "Functional no-op until Ib4f24c035e73c489cfbfe9a21164e38a9729d2d1 is merged" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363270 (https://phabricator.wikimedia.org/T441689) (owner: 10Kamila Součková) [15:45:30] (03CR) 10Clément Goubert: [C:03+1] trafficserver: enable mw-next-rounting [puppet] - 10https://gerrit.wikimedia.org/r/1363234 (https://phabricator.wikimedia.org/T441689) (owner: 10Kamila Součková) [15:46:23] (03CR) 10Kamila Součková: [C:03+2] trafficserver: enable mw-next-rounting [puppet] - 10https://gerrit.wikimedia.org/r/1363234 (https://phabricator.wikimedia.org/T441689) (owner: 10Kamila Součková) [15:46:49] !log ayounsi@cumin1004 START - Cookbook sre.dns.netbox [15:48:04] (03CR) 10Clément Goubert: basic-app: Introduce basic chart for basic apps (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362085 (https://phabricator.wikimedia.org/T436785) (owner: 10Clément Goubert) [15:48:24] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet [15:48:25] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet [15:48:28] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet [15:48:33] (03PS1) 10Clément Goubert: modules.mesh: Experiment with mesh.all [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362123 [15:49:01] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet [15:49:42] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2238 (T441633)', diff saved to https://phabricator.wikimedia.org/P97284 and previous config saved to /var/cache/conftool/dbconfig/20261104-154941-ladsgroup.json [15:50:28] (03PS2) 10Clément Goubert: modules.mesh: Introduce mesh.all [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362123 [15:51:18] !log ayounsi@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004" [15:51:21] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004" [15:51:21] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [15:51:27] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: dbproxy2007 down - https://phabricator.wikimedia.org/T442357#12516789 (10Jhancock.wm) papaul did firmware updates with this and it's now booting to os [15:54:44] (03CR) 10Muehlenhoff: [C:03+1] "The key has been verified out of band" [puppet] - 10https://gerrit.wikimedia.org/r/1362168 (https://phabricator.wikimedia.org/T443508) (owner: 10SBassett) [15:54:47] okay let's go 🍿 [15:54:57] (03CR) 10Muehlenhoff: [C:03+2] admin: Add FIDO SSH key for sbassett [puppet] - 10https://gerrit.wikimedia.org/r/1362168 (https://phabricator.wikimedia.org/T443508) (owner: 10SBassett) [15:55:02] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kamila@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363270 (https://phabricator.wikimedia.org/T441689) (owner: 10Kamila Součková) [15:55:40] Good luck! [15:56:07] (03Merged) 10jenkins-bot: Enroll 1% of client sessions in PHP 8.5 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363270 (https://phabricator.wikimedia.org/T441689) (owner: 10Kamila Součková) [15:56:20] ty dancy <3 [15:56:29] !log kamila@deploy2003 Started scap sync-world: Backport for [[gerrit:1363270|Enroll 1% of client sessions in PHP 8.5 (T441689)]] [15:56:34] T441689: [PHP 8.5] Migrate production mw-web service to PHP 8.5 (stage 1: cookie-based) - https://phabricator.wikimedia.org/T441689 [15:58:46] !log kamila@deploy2003 kamila: Backport for [[gerrit:1363270|Enroll 1% of client sessions in PHP 8.5 (T441689)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [15:59:07] !log pt1979@cumin1004 START - Cookbook sre.hosts.remove-downtime for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt [15:59:09] !log pt1979@cumin1004 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt [15:59:44] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97285 and previous config saved to /var/cache/conftool/dbconfig/20261104-155943-ladsgroup.json [16:00:25] !log kamila@deploy2003 kamila: Continuing with deployment [16:02:41] (03CR) 10Dpogorzelski: [C:03+1] kserve: allow deploy users to delete Knative revisions [deployment-charts] - 10https://gerrit.wikimedia.org/r/1360421 (https://phabricator.wikimedia.org/T442872) (owner: 10Bartosz Wójtowicz) [16:03:00] RESOLVED: [2x] PingLossPercent: Blackbox probe packet loss 4.583% from codfw to eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=eqsin - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [16:03:20] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet [16:03:21] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet [16:03:30] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet [16:03:48] !log pt1979@cumin1004 START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, T416563] [16:03:53] T416563: eqsin: upgrade routers (2026) - https://phabricator.wikimedia.org/T416563 [16:03:55] !log pt1979@cumin1004 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, T416563] [16:04:26] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:06:31] !log kamila@deploy2003 Finished scap sync-world: Backport for [[gerrit:1363270|Enroll 1% of client sessions in PHP 8.5 (T441689)]] (duration: 10m 02s) [16:06:36] T441689: [PHP 8.5] Migrate production mw-web service to PHP 8.5 (stage 1: cookie-based) - https://phabricator.wikimedia.org/T441689 [16:08:06] (03CR) 10Ottomata: [C:03+1] eventschemas: allow CORS requests from any origin [puppet] - 10https://gerrit.wikimedia.org/r/1354903 (https://phabricator.wikimedia.org/T438209) (owner: 10Aghirelli) [16:08:36] (03PS1) 10Muehlenhoff: Add FIDO keys for a-pizzata and greyolson [puppet] - 10https://gerrit.wikimedia.org/r/1363302 (https://phabricator.wikimedia.org/T442758) [16:09:46] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97286 and previous config saved to /var/cache/conftool/dbconfig/20261104-160946-ladsgroup.json [16:11:42] FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:11:42] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO keys for a-pizzata and greyolson [puppet] - 10https://gerrit.wikimedia.org/r/1363302 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [16:11:53] (03CR) 10Gmodena: [C:03+2] rest-gateway: front wdqs v2 endpoints in shadow mode [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333815 (https://phabricator.wikimedia.org/T436778) (owner: 10Gmodena) [16:12:37] (03PS2) 10Clément Goubert: bento-mw-accesslog-metrics: set up kubeconfig file [puppet] - 10https://gerrit.wikimedia.org/r/1354784 (https://phabricator.wikimedia.org/T371214) (owner: 10Tiziano Fogli) [16:12:37] (03PS1) 10Clément Goubert: parsedmard, dmarc-metrics-exporter: add kubeconfig [puppet] - 10https://gerrit.wikimedia.org/r/1363303 [16:12:37] (03PS1) 10Clément Goubert: ncmonitor: add kubeconfig [puppet] - 10https://gerrit.wikimedia.org/r/1363304 [16:12:38] (03PS1) 10Clément Goubert: pampinus: add kubeconfig [puppet] - 10https://gerrit.wikimedia.org/r/1363305 [16:14:00] (03Merged) 10jenkins-bot: rest-gateway: front wdqs v2 endpoints in shadow mode [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333815 (https://phabricator.wikimedia.org/T436778) (owner: 10Gmodena) [16:16:42] RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:17:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [16:17:22] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [16:18:48] !log gmodena@deploy2003 helmfile [staging] START helmfile.d/services/rest-gateway: apply [16:19:00] !log gmodena@deploy2003 helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [16:19:48] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2238 (T441633)', diff saved to https://phabricator.wikimedia.org/P97287 and previous config saved to /var/cache/conftool/dbconfig/20261104-161948-ladsgroup.json [16:20:13] (03PS1) 10Muehlenhoff: Add FIDO key for phuedx [puppet] - 10https://gerrit.wikimedia.org/r/1363314 (https://phabricator.wikimedia.org/T442758) [16:21:52] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs1011.eqiad.wmnet, repooling both afterwards [16:21:56] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [16:23:31] (03PS2) 10Clément Goubert: dmarc-tools: add kubeconfig [puppet] - 10https://gerrit.wikimedia.org/r/1363303 [16:23:39] (03CR) 10Gehel: [C:03+2] data: add novemlinguae's yubikey [puppet] - 10https://gerrit.wikimedia.org/r/1360846 (https://phabricator.wikimedia.org/T442756) (owner: 10Novem Linguae) [16:23:54] (03CR) 10Muehlenhoff: [C:03+2] Add FIDO key for phuedx [puppet] - 10https://gerrit.wikimedia.org/r/1363314 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [16:27:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of failed login attempts (unknown device and IP) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [16:27:44] (03CR) 10Tiziano Fogli: [C:03+1] statsd: remove check [alerts] - 10https://gerrit.wikimedia.org/r/1363237 (https://phabricator.wikimedia.org/T443230) (owner: 10Hnowlan) [16:28:07] (03CR) 10Hnowlan: [C:03+2] statsd: remove check [alerts] - 10https://gerrit.wikimedia.org/r/1363237 (https://phabricator.wikimedia.org/T443230) (owner: 10Hnowlan) [16:29:49] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs1011.eqiad.wmnet, repooling both afterwards [16:31:47] !log gmodena@deploy2003 helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [16:32:05] !log gmodena@deploy2003 helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [16:33:33] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet [16:36:23] !log gmodena@deploy2003 helmfile [codfw] START helmfile.d/services/rest-gateway: apply [16:36:38] !log gmodena@deploy2003 helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [16:41:45] (03PS1) 10Clément Goubert: Add new service discovery records for multiple services [dns] - 10https://gerrit.wikimedia.org/r/1363323 [16:43:01] 06SRE, 06DC-Ops, 06tools-infrastructure-team, 06cloud-services-team (Hardware), 06Data-Platform-SRE (2026-10-30 - 2026-11-20): How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12517370 (10Gehel) [16:44:58] (03PS2) 10Clément Goubert: Add new discovery records for multiple services [dns] - 10https://gerrit.wikimedia.org/r/1363323 [16:45:03] jouncebot: nowandnext [16:45:03] No deployments scheduled for the next 1 hour(s) and 14 minute(s) [16:45:03] In 1 hour(s) and 14 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1800) [16:45:06] (03PS4) 10Clément Goubert: bento-mw-accesslog-metrics: add namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1354785 (https://phabricator.wikimedia.org/T371214) (owner: 10Tiziano Fogli) [16:45:07] AWESOME [16:45:10] (03PS1) 10Clément Goubert: pampinus: add namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363315 [16:45:14] (03PS1) 10Clément Goubert: ncmonitor: add namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363316 [16:45:14] BABY GLOBE TIME [16:45:18] (03PS1) 10Clément Goubert: dmarc-tools: add namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363317 [16:46:00] (03CR) 10BCornwall: [C:03+1] ncmonitor: add namespace [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363316 (owner: 10Clément Goubert) [16:46:58] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet [16:46:59] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet [16:47:05] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet [16:47:41] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet [16:47:42] 06SRE, 06DC-Ops, 06tools-infrastructure-team, 06cloud-services-team (Hardware), and 2 others: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12517437 (10Gehel) p:05Triage→03High [16:47:48] (03PS1) 10OSleger: add osleger as a new member of mw release working group to releasers-mediawiki [puppet] - 10https://gerrit.wikimedia.org/r/1363327 [16:48:05] Amir1: lol! [16:48:42] (03CR) 10CI reject: [V:04-1] add osleger as a new member of mw release working group to releasers-mediawiki [puppet] - 10https://gerrit.wikimedia.org/r/1363327 (owner: 10OSleger) [16:49:51] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ladsgroup@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356470 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup) [16:50:37] (03Merged) 10jenkins-bot: Redesign 503 pages [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356470 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup) [16:50:58] !log ladsgroup@deploy2003 Started scap sync-world: Backport for [[gerrit:1356470|Redesign 503 pages (T129433)]] [16:51:03] T129433: Improve design for wiki-facing error pages - https://phabricator.wikimedia.org/T129433 [16:52:14] 10SRE-Access-Requests: Requesting access to releasers-mediawiki for osleger - https://phabricator.wikimedia.org/T443655 (10OSleger-WMF) 03NEW [16:52:19] (03PS1) 10Urbanecm: GrowthExperiments: Remove settings that nothing reads [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363328 (https://phabricator.wikimedia.org/T443648) [16:53:16] !log ladsgroup@deploy2003 ladsgroup: Backport for [[gerrit:1356470|Redesign 503 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [16:53:19] (03PS2) 10OSleger: add osleger as part of mw release working group to releasers-mediawiki [puppet] - 10https://gerrit.wikimedia.org/r/1363327 (https://phabricator.wikimedia.org/T443655) [16:54:13] (03PS3) 10Dzahn: admin: add osleger to releasers-mediawiki [puppet] - 10https://gerrit.wikimedia.org/r/1363327 (https://phabricator.wikimedia.org/T443655) (owner: 10OSleger) [16:55:31] (03CR) 10Dzahn: "just fixing commit message / CI voting" [puppet] - 10https://gerrit.wikimedia.org/r/1363327 (https://phabricator.wikimedia.org/T443655) (owner: 10OSleger) [16:56:13] 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to releasers-mediawiki for osleger - https://phabricator.wikimedia.org/T443655#12517566 (10Dzahn) a:05Dzahn→03None [16:56:18] !log ladsgroup@deploy2003 ladsgroup: Continuing with deployment [16:56:32] 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to releasers-mediawiki for osleger - https://phabricator.wikimedia.org/T443655#12517569 (10OSleger-WMF) I please request for approval from @SLopes-WMF (as my manager) and @MSantos (as PM of MediaWiki Release Working Group). [16:58:14] (03CR) 10CI reject: [V:04-1] statsd: remove check [alerts] - 10https://gerrit.wikimedia.org/r/1363237 (https://phabricator.wikimedia.org/T443230) (owner: 10Hnowlan) [16:58:23] 10SRE-Access-Requests, 06Wikimedia Enterprise: Request for additional rights for SanchezA-WMF to manage analytics - https://phabricator.wikimedia.org/T443636#12517610 (10Dzahn) [16:58:40] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet [16:58:41] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet [16:58:47] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet [16:59:19] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet [16:59:33] (03PS1) 10Muehlenhoff: Remove legacy key for phuedx [puppet] - 10https://gerrit.wikimedia.org/r/1363332 (https://phabricator.wikimedia.org/T442758) [17:00:11] 10SRE-Access-Requests, 06Wikimedia Enterprise: Request for additional rights for SanchezA-WMF to manage analytics - https://phabricator.wikimedia.org/T443636#12517626 (10Dzahn) Hi! analytics-privatedata-users has 3 different levels. So by itself it's unfortunately not specific enough which type of access is r... [17:01:12] 10SRE-Access-Requests, 06Wikimedia Enterprise: Request for additional rights for SanchezA-WMF to manage analytics - https://phabricator.wikimedia.org/T443636#12517643 (10Dzahn) Since it sounds like you are coming from an onboarding document, it would be ideal if you could also update that with the additional p... [17:02:17] !log ladsgroup@deploy2003 Finished scap sync-world: Backport for [[gerrit:1356470|Redesign 503 pages (T129433)]] (duration: 11m 19s) [17:02:22] T129433: Improve design for wiki-facing error pages - https://phabricator.wikimedia.org/T129433 [17:02:27] !log btullis@cumin1004 START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster [17:06:38] (03PS3) 10Clément Goubert: Add new discovery records for multiple services [dns] - 10https://gerrit.wikimedia.org/r/1363323 [17:06:55] (03CR) 10Muehlenhoff: [C:03+2] Remove legacy key for phuedx [puppet] - 10https://gerrit.wikimedia.org/r/1363332 (https://phabricator.wikimedia.org/T442758) (owner: 10Muehlenhoff) [17:07:30] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet [17:07:31] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet [17:07:36] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet [17:08:47] (03CR) 10Hnowlan: statsd: remove check [alerts] - 10https://gerrit.wikimedia.org/r/1363237 (https://phabricator.wikimedia.org/T443230) (owner: 10Hnowlan) [17:08:55] (03CR) 10Hnowlan: "recheck" [alerts] - 10https://gerrit.wikimedia.org/r/1363237 (https://phabricator.wikimedia.org/T443230) (owner: 10Hnowlan) [17:12:24] 10SRE-Access-Requests: Add FIDO-backed SSH keys for Rmaung - https://phabricator.wikimedia.org/T443666 (10Rmaung) 03NEW [17:13:08] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b3-eqiad,ssw1-a[1,8]-eqiad with reason: desperate troubleshooting measure [17:13:21] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: eqiad row A/B switch refresh: update LVS configuration - https://phabricator.wikimedia.org/T440859#12517784 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=df199593-4830-4db0-8b30-ba2e71011704) set by cmooney@cumi... [17:14:19] (03PS1) 10Dzahn: scap: switch mediawiki_runtime_image to php8.5 [puppet] - 10https://gerrit.wikimedia.org/r/1363343 [17:14:36] (03PS2) 10Dzahn: scap: switch mediawiki_runtime_image to php8.5 [puppet] - 10https://gerrit.wikimedia.org/r/1363343 [17:14:44] (03CR) 10CI reject: [V:04-1] scap: switch mediawiki_runtime_image to php8.5 [puppet] - 10https://gerrit.wikimedia.org/r/1363343 (owner: 10Dzahn) [17:14:55] (03CR) 10Dzahn: "--> replace with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363343 ?" [puppet] - 10https://gerrit.wikimedia.org/r/1362119 (owner: 10Kamila Součková) [17:15:53] (03CR) 10Kamila Součková: [C:03+1] scap: switch mediawiki_runtime_image to php8.5 [puppet] - 10https://gerrit.wikimedia.org/r/1363343 (owner: 10Dzahn) [17:16:56] (03PS2) 10Ladsgroup: errorpage: Redesign the layout, fonts and colours [puppet] - 10https://gerrit.wikimedia.org/r/1363344 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [17:16:56] (03CR) 10Ladsgroup: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/1363344 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [17:17:03] (03CR) 10Dzahn: "https://gerrit.wikimedia.org/r/c/operations/puppet/+/1359987 adds both, eqiad and codfw, docker-registry services to allowed_services as s" [puppet] - 10https://gerrit.wikimedia.org/r/1359404 (https://phabricator.wikimedia.org/T439046) (owner: 10Majavah) [17:17:09] (03CR) 10Dzahn: [C:03+2] gitlab_runner: add both docker registries to allowed_services [puppet] - 10https://gerrit.wikimedia.org/r/1359987 (https://phabricator.wikimedia.org/T439046) (owner: 10Dzahn) [17:17:28] (03PS3) 10Ladsgroup: errorpage: Redesign the layout, fonts and colours [puppet] - 10https://gerrit.wikimedia.org/r/1363344 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [17:18:06] (03PS1) 10Ottomata: deployment_server: Add editcheck-headless to deploy_service catalog [puppet] - 10https://gerrit.wikimedia.org/r/1363348 (https://phabricator.wikimedia.org/T430712) [17:18:55] (03CR) 10Kamila Součková: "Awesome, thank you!" [puppet] - 10https://gerrit.wikimedia.org/r/1362119 (owner: 10Kamila Součková) [17:19:02] (03Abandoned) 10Kamila Součková: Revert "scap: use php8.3 multiversion for scap php operations" [puppet] - 10https://gerrit.wikimedia.org/r/1362119 (owner: 10Kamila Součková) [17:19:17] (03CR) 10Dzahn: [C:03+1] "let's ship it. seems like we are all ok with this compromise" [puppet] - 10https://gerrit.wikimedia.org/r/1308081 (https://phabricator.wikimedia.org/T431398) (owner: 10Arnaudb) [17:23:18] (03PS2) 10Ladsgroup: php7-fatal-error: Apply the error page redesign [puppet] - 10https://gerrit.wikimedia.org/r/1363346 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [17:23:18] (03CR) 10Ladsgroup: "Do we even use these?" [puppet] - 10https://gerrit.wikimedia.org/r/1363346 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [17:24:21] (03CR) 10Hnowlan: [C:03+2] statsd: remove check [alerts] - 10https://gerrit.wikimedia.org/r/1363237 (https://phabricator.wikimedia.org/T443230) (owner: 10Hnowlan) [17:26:16] !log jhancock@cumin2003 START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [17:26:35] !log apt: included haproxykafka 0.3.18 into trixie-wikimedia (T443632) [17:26:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:26:39] T443632: Support x-kapow-score header in HaproxyKafka - https://phabricator.wikimedia.org/T443632 [17:27:01] (03Merged) 10jenkins-bot: statsd: remove check [alerts] - 10https://gerrit.wikimedia.org/r/1363237 (https://phabricator.wikimedia.org/T443230) (owner: 10Hnowlan) [17:28:23] (03PS1) 10Hnowlan: test__log: change test to account for race condition [software/spicerack] - 10https://gerrit.wikimedia.org/r/1363355 [17:29:07] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: es2036 crashed - https://phabricator.wikimedia.org/T443268#12517946 (10Marostegui) I think we can close. @Jhancock.wm please let us know if there's anything coming from Dell even if the ticket is closed. Thanks! [17:29:39] 10ops-codfw, 06SRE, 06DBA, 06DC-Ops: es2036 crashed - https://phabricator.wikimedia.org/T443268#12517965 (10Marostegui) 05Open→03Resolved [17:30:51] (03CR) 10CI reject: [V:04-1] test__log: change test to account for race condition [software/spicerack] - 10https://gerrit.wikimedia.org/r/1363355 (owner: 10Hnowlan) [17:31:45] !log jhancock@cumin2003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [17:36:55] (03CR) 10Dzahn: [C:03+2] trafficserver: add maps for codesearch [puppet] - 10https://gerrit.wikimedia.org/r/1342804 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [17:37:39] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet [17:38:21] 06SRE, 06DC-Ops, 06tools-infrastructure-team, 06cloud-services-team (Hardware), and 2 others: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12518079 (10BTullis) Thanks for the ping and apologies for being slow to respond. I'm looking into this now, too. I'm noti... [17:39:33] (03CR) 10ArielGlenn: [C:03+2] changeprop: node 20 to node 26 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361572 (https://phabricator.wikimedia.org/T442195) (owner: 10ArielGlenn) [17:41:01] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363366 [17:41:02] (03Merged) 10jenkins-bot: changeprop: node 20 to node 26 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1361572 (https://phabricator.wikimedia.org/T442195) (owner: 10ArielGlenn) [17:41:55] (03CR) 10Slyngshede: "Our document says: Rolling out Tofurkey to all edge sites" [puppet] - 10https://gerrit.wikimedia.org/r/1363151 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [17:43:01] (03PS3) 10Ladsgroup: errorpage: Add Wikimedia branding to the error pages [puppet] - 10https://gerrit.wikimedia.org/r/1363345 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [17:43:14] (03PS2) 10Ladsgroup: errorpage: Show baby globe on wikipedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1363347 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [17:43:46] (03CR) 10RLazarus: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9602/co" [puppet] - 10https://gerrit.wikimedia.org/r/1363344 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [17:44:03] FIRING: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:44:37] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet [17:44:38] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet [17:44:43] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet [17:45:16] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet [17:45:35] !log ariel@deploy2003 helmfile [staging] START helmfile.d/services/changeprop: apply [17:46:25] !log ariel@deploy2003 helmfile [staging] DONE helmfile.d/services/changeprop: apply [17:47:23] !log ariel@deploy2003 helmfile [eqiad] START helmfile.d/services/changeprop: apply [17:48:04] !log ariel@deploy2003 helmfile [eqiad] DONE helmfile.d/services/changeprop: apply [17:48:28] !log ariel@deploy2003 helmfile [codfw] START helmfile.d/services/changeprop: apply [17:48:43] 06SRE, 10FY26-27 ST6.4 - Logged-in performance, 06serviceops-deprecated, 07Wikimedia-Performance-recommendation: Evaluate using igbinary for MW php-apcu at WMF - https://phabricator.wikimedia.org/T225074#12518268 (10Krinkle) 05Open→03Stalled [17:49:03] RESOLVED: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:49:07] !log ariel@deploy2003 helmfile [codfw] DONE helmfile.d/services/changeprop: apply [17:51:28] !log ariel@deploy2003 helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply [17:51:42] !log ariel@deploy2003 helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply [17:51:55] !log ariel@deploy2003 helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [17:52:00] !log ariel@deploy2003 helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [17:52:08] !log ariel@deploy2003 helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply [17:52:11] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet [17:52:12] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet [17:52:17] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet [17:53:12] !log ariel@deploy2003 helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply [17:53:21] (03CR) 10RLazarus: [C:03+1] errorpage: Redesign the layout, fonts and colours (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363344 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [17:53:22] !log ariel@deploy2003 helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply [17:53:48] I did not in fact apply to codfw, I bailed, and did eqiad first [17:54:26] !log ariel@deploy2003 helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply [17:55:19] (03PS4) 10Ladsgroup: errorpage: Redesign the layout, fonts and colours [puppet] - 10https://gerrit.wikimedia.org/r/1363344 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [17:55:22] you really think logmsgbot would do that? just come into #wikimedia-operations and tell lies? [17:55:31] (03CR) 10Ladsgroup: errorpage: Redesign the layout, fonts and colours (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363344 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [17:56:00] cough [17:56:08] well I did start it and then say 'uh nope thanks' [17:56:59] (03CR) 10Dzahn: "thinking: why not send cloud traffic through CDN as well?" [puppet] - 10https://gerrit.wikimedia.org/r/1357232 (https://phabricator.wikimedia.org/T425441) (owner: 10Arnaudb) [17:57:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by krinkle@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311062 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [17:57:42] (03Abandoned) 10Neriah: ukwiki: limit "user" namespace to only registered users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1360437 (https://phabricator.wikimedia.org/T438016) (owner: 10Neriah) [17:57:58] (03Merged) 10jenkins-bot: Remove Mathoid SVG option from wgMathValidModes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311062 (https://phabricator.wikimedia.org/T271001) (owner: 10Krinkle) [17:58:22] !log krinkle@deploy2003 Started scap sync-world: Backport for [[gerrit:1311062|Remove Mathoid SVG option from wgMathValidModes (T271001)]] [17:58:27] T271001: Transition to client-side MathJax SVG rendering as default - https://phabricator.wikimedia.org/T271001 [18:00:05] Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1800) [18:00:52] dancy: should I merge this? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363343 [18:01:18] !log krinkle@deploy2003 krinkle: Backport for [[gerrit:1311062|Remove Mathoid SVG option from wgMathValidModes (T271001)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [18:02:14] If you want it to take effect, yes. :-) It LGTM [18:02:21] (03CR) 10Ahmon Dancy: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1363343 (owner: 10Dzahn) [18:02:30] (03CR) 10Dzahn: [C:03+2] scap: switch mediawiki_runtime_image to php8.5 [puppet] - 10https://gerrit.wikimedia.org/r/1363343 (owner: 10Dzahn) [18:04:01] (03CR) 10Dzahn: [C:03+1] "Antoine suggested it, and we agreed, shipping it." [puppet] - 10https://gerrit.wikimedia.org/r/1308081 (https://phabricator.wikimedia.org/T431398) (owner: 10Arnaudb) [18:04:16] (03CR) 10Dzahn: [C:03+2] gerrit: cap index query results at 5000 [puppet] - 10https://gerrit.wikimedia.org/r/1308081 (https://phabricator.wikimedia.org/T431398) (owner: 10Arnaudb) [18:06:06] !log krinkle@deploy2003 krinkle: Continuing with deployment [18:09:32] (03CR) 10Ssingh: "Yeah good point and it leads to the general confusion around our use of the word "public edge". But yeah, this should be everywhere so I t" [puppet] - 10https://gerrit.wikimedia.org/r/1363151 (https://phabricator.wikimedia.org/T355446) (owner: 10Slyngshede) [18:11:06] (03Abandoned) 10Hnowlan: service: add discovery active/active config for thumbor [puppet] - 10https://gerrit.wikimedia.org/r/1164458 (https://phabricator.wikimedia.org/T397618) (owner: 10Hnowlan) [18:11:11] jouncebot: nowandnext [18:11:11] For the next 0 hour(s) and 48 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1800) [18:11:11] In 0 hour(s) and 48 minute(s): MediaWiki train - Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1900) [18:11:18] !log gerrit2003/gerrit2002 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 (T431398) [18:11:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:11:37] (03Abandoned) 10Hnowlan: wmnet: add discovery records for thumbor [dns] - 10https://gerrit.wikimedia.org/r/1164457 (https://phabricator.wikimedia.org/T397618) (owner: 10Hnowlan) [18:11:57] (03CR) 10RLazarus: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9603/co" [puppet] - 10https://gerrit.wikimedia.org/r/1363347 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [18:12:06] !log krinkle@deploy2003 Finished scap sync-world: Backport for [[gerrit:1311062|Remove Mathoid SVG option from wgMathValidModes (T271001)]] (duration: 13m 43s) [18:12:11] T271001: Transition to client-side MathJax SVG rendering as default - https://phabricator.wikimedia.org/T271001 [18:12:22] Amir1: looking to deploy? [18:12:41] yeah, I have a thing or two. Ping me once done? [18:12:59] I'm done [18:13:04] Awesome. Thanks! [18:13:52] (03PS5) 10Ladsgroup: errorpage: Redesign the layout, fonts and colours [puppet] - 10https://gerrit.wikimedia.org/r/1363344 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [18:13:58] (03CR) 10Ladsgroup: [V:03+2 C:03+2] errorpage: Redesign the layout, fonts and colours [puppet] - 10https://gerrit.wikimedia.org/r/1363344 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [18:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [18:14:35] (03PS4) 10Ladsgroup: errorpage: Add Wikimedia branding to the error pages [puppet] - 10https://gerrit.wikimedia.org/r/1363345 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [18:14:43] (03CR) 10Ladsgroup: [V:03+2 C:03+2] errorpage: Add Wikimedia branding to the error pages [puppet] - 10https://gerrit.wikimedia.org/r/1363345 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [18:15:15] (03PS3) 10Ladsgroup: php7-fatal-error: Apply the error page redesign [puppet] - 10https://gerrit.wikimedia.org/r/1363346 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [18:15:23] (03CR) 10Ladsgroup: [V:03+2 C:03+2] php7-fatal-error: Apply the error page redesign [puppet] - 10https://gerrit.wikimedia.org/r/1363346 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [18:16:00] (03PS3) 10Ladsgroup: errorpage: Show baby globe on wikipedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1363347 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [18:16:08] (03CR) 10Ladsgroup: [V:03+2 C:03+2] errorpage: Show baby globe on wikipedia.org [puppet] - 10https://gerrit.wikimedia.org/r/1363347 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [18:16:58] (03CR) 10Hnowlan: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9604/co" [puppet] - 10https://gerrit.wikimedia.org/r/1359819 (https://phabricator.wikimedia.org/T435340) (owner: 10Hnowlan) [18:16:59] wait, this is happening!? [18:17:03] ! [18:17:23] Baby Globe-do-doo-do-do-do-do-dooh. [18:18:42] sukhe: you don't take my threats seriously :P [18:18:55] :P [18:18:56] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1012.eqiad.wmnet, repooling both afterwards [18:19:00] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [18:19:18] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2007.codfw.wmnet, repooling both afterwards [18:20:23] (03CR) 10Ahmon Dancy: deployment_server: Add editcheck-headless to deploy_service catalog (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363348 (https://phabricator.wikimedia.org/T430712) (owner: 10Ottomata) [18:21:20] now someone just needs to cause an outage to test these out [18:21:30] 🏃 [18:21:34] … let’s not jinx it. [18:21:35] hold my milk [18:21:59] 18:17:39 -- │ You have been marked as being away [18:22:12] Not planning on changing that thanks [18:22:21] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet [18:23:03] jouncebot: nowandnext [18:23:03] For the next 0 hour(s) and 36 minute(s): MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1800) [18:23:03] In 0 hour(s) and 36 minute(s): MediaWiki train - Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1900) [18:23:07] taavi: while I have your attention, wanna check using svg instead of png for cloud vps/toolforge? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363390 [18:23:15] mutante: I'm deploying baby globe [18:23:45] Amir1: want to restart gerrit to deploy a config change that needs it. should not log users out but would disconnect. waiting. [18:24:16] (03PS1) 10Ladsgroup: errorpage: Use SVG logos and remove logo_srcset [puppet] - 10https://gerrit.wikimedia.org/r/1363390 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [18:24:54] I should finish the 404 page too, at least we can show that without needing to cause an outage [18:26:51] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1012.eqiad.wmnet, repooling both afterwards [18:26:56] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [18:27:00] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2007.codfw.wmnet, repooling both afterwards [18:28:12] Amir1: have you checked that the dollar sign you're adding into the CSS is fine when we feed that into the envoy config? [18:28:36] (should the CSS selector for only changing some logos be a check in the ERB template instead?) [18:28:38] we did pcc, didn't break anything but let me triple check [18:29:05] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 92.29% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [18:29:12] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet [18:29:13] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet [18:29:19] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet [18:29:30] see the 'The output must not contain:' comment at the top of the page, a percent sign is not listed but worth checking in advance ihmo [18:29:36] yeah I don't think $ is significant to SubstitutionFormatter [18:29:46] percent sign I checked [18:29:55] s/percent/dollar/ [18:30:01] wait the javascript in that page already has a dollar sign, so it's probably fine [18:30:23] did you say you had a PCC? [18:30:39] docs are here, there's nothing dollary https://www.envoyproxy.io/docs/envoy/latest/configuration/advanced/substitution_formatter [18:31:00] (PCC will test the ERB but won't actually run the config through envoy) [18:31:23] ﷼ would break it though [18:33:14] okay, puppet agent ran smoothly on deploy2003 [18:34:05] (03PS1) 10Dzahn: cache: add codesearch front- and backend to alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/1363418 (https://phabricator.wikimedia.org/T268199) [18:34:32] !log ladsgroup@deploy2003 Started scap sync-world: Pushing 503 redesigns (T129433) [18:34:37] T129433: Improve design for wiki-facing error pages - https://phabricator.wikimedia.org/T129433 [18:35:13] !log ladsgroup@deploy2003 Stopping before sync operations [18:35:30] Amir1: hmh, the new designs link wikimediastatus.net from the toolforge/cloud vps pages, which we don't probably want [18:36:12] also there's https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_use#4.7_Use_of_Wikimedia_logos which means the toolforge/cloud vps pages (or the ones on beta!) shouldn't have the foundation logo at the bottom [18:37:09] I'll see what I can do to fix it [18:37:25] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1013.eqiad.wmnet, repooling both afterwards [18:37:34] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [18:37:38] it can take parameters [18:37:49] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling both afterwards [18:39:01] !log ladsgroup@deploy2003 Started scap sync-world: Pushing 503 redesigns (T129433) [18:39:04] (03CR) 10Majavah: [C:03+1] "changes to logo handling seem fine" [puppet] - 10https://gerrit.wikimedia.org/r/1363390 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [18:40:00] (03CR) 10Ladsgroup: [V:03+2 C:03+2] errorpage: Use SVG logos and remove logo_srcset [puppet] - 10https://gerrit.wikimedia.org/r/1363390 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [18:40:33] !log ladsgroup@deploy2003 ladsgroup: Pushing 503 redesigns (T129433) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [18:40:37] T129433: Improve design for wiki-facing error pages - https://phabricator.wikimedia.org/T129433 [18:40:38] (03PS2) 10Ottomata: deployment_server: Add editcheck-headless to deploy_service catalog [puppet] - 10https://gerrit.wikimedia.org/r/1363348 (https://phabricator.wikimedia.org/T430712) [18:40:42] (03CR) 10Ottomata: deployment_server: Add editcheck-headless to deploy_service catalog (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1363348 (https://phabricator.wikimedia.org/T430712) (owner: 10Ottomata) [18:42:39] !log ladsgroup@deploy2003 ladsgroup: Continuing with deployment [18:44:01] (03CR) 10Mforns: "Hi all! Is it too late to change the name of this field? I think naming it KAPOW score might be confusing." [puppet] - 10https://gerrit.wikimedia.org/r/1363149 (https://phabricator.wikimedia.org/T435799) (owner: 10Fabfur) [18:45:27] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1013.eqiad.wmnet, repooling both afterwards [18:45:32] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [18:45:32] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling both afterwards [18:45:54] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1014.eqiad.wmnet, repooling both afterwards [18:46:09] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2011.codfw.wmnet, repooling both afterwards [18:47:18] (03PS2) 10Ladsgroup: errorpage: Hide Wikimedia branding outside production [puppet] - 10https://gerrit.wikimedia.org/r/1363420 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [18:47:29] (03CR) 10Ssingh: [C:03+1] cache: add codesearch front- and backend to alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/1363418 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [18:48:20] (03Abandoned) 10Dzahn: gerrit: recover a killed replica reindex before the replica check [puppet] - 10https://gerrit.wikimedia.org/r/1352297 (https://phabricator.wikimedia.org/T440177) (owner: 10Dzahn) [18:48:31] (03CR) 10Ahmon Dancy: [C:03+1] deployment_server: Add editcheck-headless to deploy_service catalog [puppet] - 10https://gerrit.wikimedia.org/r/1363348 (https://phabricator.wikimedia.org/T430712) (owner: 10Ottomata) [18:48:36] !log ladsgroup@deploy2003 Finished scap sync-world: Pushing 503 redesigns (T129433) (duration: 09m 40s) [18:48:41] T129433: Improve design for wiki-facing error pages - https://phabricator.wikimedia.org/T129433 [18:48:45] (03PS1) 10Southparkfan: profile::pki::client: Properly update system trust store [puppet] - 10https://gerrit.wikimedia.org/r/1363421 (https://phabricator.wikimedia.org/T436562) [18:49:08] (03CR) 10Dzahn: [C:03+2] cache: add codesearch front- and backend to alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/1363418 (https://phabricator.wikimedia.org/T268199) (owner: 10Dzahn) [18:49:34] (03CR) 10Southparkfan: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1363421 (https://phabricator.wikimedia.org/T436562) (owner: 10Southparkfan) [18:50:09] (03CR) 10CI reject: [V:04-1] profile::pki::client: Properly update system trust store [puppet] - 10https://gerrit.wikimedia.org/r/1363421 (https://phabricator.wikimedia.org/T436562) (owner: 10Southparkfan) [18:52:24] (03PS2) 10Southparkfan: profile::pki::client: Properly update system trust store [puppet] - 10https://gerrit.wikimedia.org/r/1363421 (https://phabricator.wikimedia.org/T436562) [18:53:42] (03CR) 10Southparkfan: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1363421 (https://phabricator.wikimedia.org/T436562) (owner: 10Southparkfan) [18:53:47] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1014.eqiad.wmnet, repooling both afterwards [18:53:51] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2011.codfw.wmnet, repooling both afterwards [18:53:52] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [18:59:21] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet [18:59:53] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2012.codfw.wmnet, repooling both afterwards [18:59:57] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [19:00:05] jnuche and dduvall: I, the Bot under the Fountain, call upon thee, The Deployer, to do MediaWiki train - Utc-7 Version deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1900). [19:00:06] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling both afterwards [19:01:29] PROBLEM - Host an-worker1154 is DOWN: PING CRITICAL - Packet loss = 100% [19:05:42] (03Abandoned) 10Ladsgroup: [WIP] Redesign 404 pages [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356471 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup) [19:06:13] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet [19:06:14] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet [19:06:20] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet [19:07:29] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2012.codfw.wmnet, repooling both afterwards [19:07:34] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [19:08:01] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling both afterwards [19:08:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:08:43] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1017.eqiad.wmnet, repooling both afterwards [19:08:58] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2013.codfw.wmnet, repooling both afterwards [19:13:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [19:15:18] (03PS1) 10Ladsgroup: Redesign 404 pages [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363423 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [19:16:18] (03CR) 10Ladsgroup: [C:03+2] "In mw-experimental, it all worked except the static image which I think is caused by some issues in mw-experimental setup. Even other imag" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363423 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [19:16:26] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1017.eqiad.wmnet, repooling both afterwards [19:16:27] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2013.codfw.wmnet, repooling both afterwards [19:16:31] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [19:17:02] (03Merged) 10jenkins-bot: Redesign 404 pages [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363423 (https://phabricator.wikimedia.org/T129433) (owner: 10Ladsgroup-claude) [19:18:14] !log ladsgroup@deploy2003 Started scap sync-world: Backport for [[gerrit:1363423|Redesign 404 pages (T129433)]] [19:18:20] T129433: Improve design for wiki-facing error pages - https://phabricator.wikimedia.org/T129433 [19:20:34] !log ladsgroup@deploy2003 ladsgroup, ladsgroup-claude: Backport for [[gerrit:1363423|Redesign 404 pages (T129433)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [19:21:39] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling both afterwards [19:21:44] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [19:21:59] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1018.eqiad.wmnet, repooling both afterwards [19:22:04] !log ladsgroup@deploy2003 ladsgroup, ladsgroup-claude: Continuing with deployment [19:22:29] PROBLEM - Confd vcl based reload on cp7001 is CRITICAL: reload-vcl failed to run since 0h, 2 minutes. https://wikitech.wikimedia.org/wiki/Varnish [19:22:55] hmm ok [19:25:08] related to what though? wdqs or earlier change? [19:25:27] mutante: probably related to a rule we just pushed (see -sec) [19:25:39] it happens because of a race condition in two consecutive reloads [19:26:29] ACK! thanks [19:28:01] !log ladsgroup@deploy2003 Finished scap sync-world: Backport for [[gerrit:1363423|Redesign 404 pages (T129433)]] (duration: 09m 46s) [19:28:06] T129433: Improve design for wiki-facing error pages - https://phabricator.wikimedia.org/T129433 [19:29:07] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling both afterwards [19:29:10] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [19:29:46] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1018.eqiad.wmnet, repooling both afterwards [19:31:17] (03CR) 10BCornwall: [C:03+1] ncmonitor: add kubeconfig [puppet] - 10https://gerrit.wikimedia.org/r/1363304 (owner: 10Clément Goubert) [19:32:16] jouncebot: now [19:32:16] For the next 1 hour(s) and 27 minute(s): MediaWiki train - Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T1900) [19:32:25] still happening? [19:33:10] mutante: The train shipped in the earlier slot. [19:33:33] James_F: thanks! gerrit restart coming in then [19:34:47] !log gerrit1003 - - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 (T431398) [19:34:49] mutante: Failed to log message to wiki. Somebody should check the error logs. [19:35:09] !log gerrit1003 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 (T431398) [19:35:11] mutante: Failed to log message to wiki. Somebody should check the error logs. [19:35:59] btullis@cumin1004 reboot-workers (PID 660710) is awaiting input [19:36:22] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet [19:36:24] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [19:37:17] !log !log is not !logging [19:37:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:38:02] !log gerrit1003 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 (T431398) [19:38:04] mutante: Failed to log message to wiki. Somebody should check the error logs. [19:38:09] wth [19:40:41] (03CR) 10Dzahn: [C:03+2] "deployed, restarted gerrit service and verified. Since the CDN already doesn't let this through, via ssh API." [puppet] - 10https://gerrit.wikimedia.org/r/1308081 (https://phabricator.wikimedia.org/T431398) (owner: 10Arnaudb) [19:42:12] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance [19:42:13] ladsgroup@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [19:42:13] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2168 (T441633)', diff saved to https://phabricator.wikimedia.org/P97289 and previous config saved to /var/cache/conftool/dbconfig/20261104-194213-ladsgroup.json [19:42:15] ladsgroup@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [19:42:30] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet [19:42:31] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet [19:42:31] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [19:42:32] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [19:42:36] 06SRE, 10SRE-Access-Requests: Add FIDO-backed SSH keys for bearloga - https://phabricator.wikimedia.org/T443359#12518842 (10mpopov) 05Open→03Resolved a:03mpopov Verified that the new keys work [19:42:36] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet [19:42:38] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [19:45:38] 06SRE, 06Infrastructure-Foundations, 10netops: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687 (10cmooney) 03NEW p:05Triage→03High [19:46:15] 06SRE, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh configuration - https://phabricator.wikimedia.org/T439535#12518898 (10cmooney) [19:46:16] 06SRE, 06Infrastructure-Foundations, 10netops: SR-Linux v26.7.3 drops ARP packets in MAC-VRF - https://phabricator.wikimedia.org/T443687#12518897 (10cmooney) [19:48:11] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1019.eqiad.wmnet, repooling both afterwards [19:48:14] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [19:48:15] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [19:48:25] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2015.codfw.wmnet, repooling both afterwards [19:48:28] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [19:49:51] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2168 (T441633)', diff saved to https://phabricator.wikimedia.org/P97290 and previous config saved to /var/cache/conftool/dbconfig/20261104-194950-ladsgroup.json [19:49:53] ladsgroup@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [19:55:47] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2015.codfw.wmnet, repooling both afterwards [19:55:50] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [19:55:51] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [19:55:54] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1019.eqiad.wmnet, repooling both afterwards [19:55:57] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [19:59:53] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P97291 and previous config saved to /var/cache/conftool/dbconfig/20261104-195952-ladsgroup.json [19:59:54] ladsgroup@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:00:27] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2021.codfw.wmnet, repooling both afterwards [20:00:30] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [20:01:23] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1020.eqiad.wmnet, repooling both afterwards [20:01:26] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [20:01:26] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [20:04:05] RESOLVED: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 90.87% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [20:04:27] FIRING: SystemdUnitFailed: wmf_auto_restart_ipmiseld.service on ml-lab1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:07:13] (03PS3) 10Jforrester: Simplify Math config, reduce mention of Mathoid, prepare for removal [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1311063 (https://phabricator.wikimedia.org/T432511) (owner: 10Krinkle) [20:07:51] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2021.codfw.wmnet, repooling both afterwards [20:07:54] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [20:07:55] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [20:08:24] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2022.codfw.wmnet, repooling both afterwards [20:08:27] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [20:09:09] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1020.eqiad.wmnet, repooling both afterwards [20:09:12] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [20:09:55] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P97292 and previous config saved to /var/cache/conftool/dbconfig/20261104-200954-ladsgroup.json [20:09:56] ladsgroup@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:12:05] (03PS1) 10Jdlrobson: UnsaveDialog: Don't render in place in bottom sheet mode [extensions/ReadingLists] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363460 (https://phabricator.wikimedia.org/T443481) [20:12:39] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet [20:12:40] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:12:45] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1021.eqiad.wmnet, repooling both afterwards [20:12:47] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [20:15:50] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2022.codfw.wmnet, repooling both afterwards [20:15:53] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [20:15:54] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [20:17:22] FIRING: CertAlmostExpired: gNMI TLS certificate for lsw1-d6-eqiad.mgmt.eqiad.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [20:19:26] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet [20:19:27] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet [20:19:28] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:19:29] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:19:33] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet [20:19:34] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:19:57] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2168 (T441633)', diff saved to https://phabricator.wikimedia.org/P97293 and previous config saved to /var/cache/conftool/dbconfig/20261104-201956-ladsgroup.json [20:19:58] ladsgroup@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:20:14] (03PS1) 10Ebernhardson: cirrus: Provide short cluster names [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363463 (https://phabricator.wikimedia.org/T440376) [20:20:15] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1021.eqiad.wmnet, repooling both afterwards [20:20:16] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance [20:20:18] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [20:20:18] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2169 (T441633)', diff saved to https://phabricator.wikimedia.org/P97294 and previous config saved to /var/cache/conftool/dbconfig/20261104-202017-ladsgroup.json [20:20:19] ladsgroup@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:20:20] ladsgroup@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:20:52] (03CR) 10CI reject: [V:04-1] cirrus: Provide short cluster names [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363463 (https://phabricator.wikimedia.org/T440376) (owner: 10Ebernhardson) [20:22:39] (03PS2) 10Ebernhardson: cirrus: Provide short cluster names [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363463 (https://phabricator.wikimedia.org/T440376) [20:23:17] (03CR) 10CI reject: [V:04-1] cirrus: Provide short cluster names [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363463 (https://phabricator.wikimedia.org/T440376) (owner: 10Ebernhardson) [20:24:18] 06SRE, 10LDAP-Access-Requests: Grant Access to [analytics resources: Superset, Jupyter, Growthbook, Turnilo, Test Kitchen UI] for cfoster - https://phabricator.wikimedia.org/T443699 (10cfoster) 03NEW [20:24:36] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet [20:24:37] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:24:56] 06SRE, 10LDAP-Access-Requests: Grant Access to [analytics resources: Superset, Jupyter, Growthbook, Turnilo, Test Kitchen UI] for cfoster - https://phabricator.wikimedia.org/T443699#12519106 (10cfoster) [20:25:18] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1022.eqiad.wmnet, repooling both afterwards [20:25:21] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [20:25:21] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [20:26:00] (03PS3) 10Ebernhardson: cirrus: Provide short cluster names [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363463 (https://phabricator.wikimedia.org/T440376) [20:27:30] (03CR) 10CI reject: [V:04-1] cirrus: Provide short cluster names [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363463 (https://phabricator.wikimedia.org/T440376) (owner: 10Ebernhardson) [20:27:44] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2169 (T441633)', diff saved to https://phabricator.wikimedia.org/P97295 and previous config saved to /var/cache/conftool/dbconfig/20261104-202744-ladsgroup.json [20:27:46] ladsgroup@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:28:34] 10ops-eqiad, 06SRE, 06DC-Ops: Unresponsive management for an-worker1181.mgmt:22 - https://phabricator.wikimedia.org/T443555#12519124 (10VRiley-WMF) 05Open→03Resolved a:03VRiley-WMF Reseated cable [20:30:39] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet [20:30:40] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet [20:30:40] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:30:42] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:30:45] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet [20:30:47] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:33:06] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1022.eqiad.wmnet, repooling both afterwards [20:33:09] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [20:33:09] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [20:37:46] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P97296 and previous config saved to /var/cache/conftool/dbconfig/20261104-203745-ladsgroup.json [20:37:48] ladsgroup@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:39:58] (03PS4) 10Ebernhardson: cirrus: Provide short cluster names [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363463 (https://phabricator.wikimedia.org/T440376) [20:41:25] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1033.eqiad.wmnet, repooling both afterwards [20:41:28] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [20:41:28] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [20:42:11] (03CR) 10VolkerE: [C:03+1] UnsaveDialog: Don't render in place in bottom sheet mode [extensions/ReadingLists] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363460 (https://phabricator.wikimedia.org/T443481) (owner: 10Jdlrobson) [20:44:41] (03PS1) 10Ryan Kemper: airflow-ml: upgrade to Airflow 3.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363466 (https://phabricator.wikimedia.org/T435481) [20:46:24] (03PS2) 10Ryan Kemper: airflow-ml: upgrade to Airflow 3.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363466 (https://phabricator.wikimedia.org/T435481) [20:47:48] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P97297 and previous config saved to /var/cache/conftool/dbconfig/20261104-204747-ladsgroup.json [20:47:50] ladsgroup@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:48:57] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1033.eqiad.wmnet, repooling both afterwards [20:49:00] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [20:49:00] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [20:53:04] !log bking@cumin2003 START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1034.eqiad.wmnet, repooling both afterwards [20:53:07] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [20:53:26] (03PS5) 10Herron: rsyslog: use pmnormalize for json parsing [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362315 (https://phabricator.wikimedia.org/T438375) [20:53:26] (03CR) 10Herron: "This prepares the rsyslog config for ratelimit. The current mmjsonparse approach runs a step after ratelimit would take effect, so this m" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1362315 (https://phabricator.wikimedia.org/T438375) (owner: 10Herron) [20:54:07] (03PS1) 10Tsevener: Allow ReadingLists path in Apple App Site Association File [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363467 (https://phabricator.wikimedia.org/T443456) [20:55:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [20:55:40] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Wednesday, November 04 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363467 (https://phabricator.wikimedia.org/T443456) (owner: 10Tsevener) [20:57:50] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2169 (T441633)', diff saved to https://phabricator.wikimedia.org/P97298 and previous config saved to /var/cache/conftool/dbconfig/20261104-205750-ladsgroup.json [20:57:52] ladsgroup@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:58:09] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2182.codfw.wmnet with reason: Maintenance [20:58:11] ladsgroup@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [20:58:12] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2182 (T441633)', diff saved to https://phabricator.wikimedia.org/P97299 and previous config saved to /var/cache/conftool/dbconfig/20261104-205811-ladsgroup.json [20:58:13] ladsgroup@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [21:00:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:00:04] RoanKattouw, urbanecm, TheresNoTime, and cjming: Time to snap out of that daydream and deploy UTC late backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T2100). [21:00:05] ejegg and toni_: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [21:00:39] !log bking@cumin2003 END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1034.eqiad.wmnet, repooling both afterwards [21:00:41] bking@cumin2003: Failed to log message to wiki. Somebody should check the error logs. [21:00:42] T441682: WDQS: investigate fleet-wide categories lag - https://phabricator.wikimedia.org/T441682 [21:00:48] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1012.eqiad.wmnet [21:00:50] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [21:03:04] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:05:29] o/ [21:05:43] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2182 (T441633)', diff saved to https://phabricator.wikimedia.org/P97300 and previous config saved to /var/cache/conftool/dbconfig/20261104-210542-ladsgroup.json [21:05:44] ladsgroup@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [21:06:11] hi folks [21:06:28] toni_: do you want to do your deploy first? [21:07:12] mine has some i18n changes so will be a long sync [21:08:04] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [21:08:05] hi, sure [21:09:08] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet [21:09:09] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [21:09:10] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet [21:09:11] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [21:09:16] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet [21:09:17] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [21:09:41] starting mine now [21:10:14] (03CR) 10TrainBranchBot: [C:03+2] "Approved by tsev@deploy2003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363467 (https://phabricator.wikimedia.org/T443456) (owner: 10Tsevener) [21:10:59] (03Merged) 10jenkins-bot: Allow ReadingLists path in Apple App Site Association File [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363467 (https://phabricator.wikimedia.org/T443456) (owner: 10Tsevener) [21:11:17] !log btullis@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1013.eqiad.wmnet [21:11:18] btullis@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [21:11:22] !log tsev@deploy2003 Started scap sync-world: Backport for [[gerrit:1363467|Allow ReadingLists path in Apple App Site Association File (T443456)]] [21:11:25] tsev@deploy2003: Failed to log message to wiki. Somebody should check the error logs. [21:11:26] T443456: Discover: Allow Special:ReadingLists import links in the AASA file so they open in the iOS app - https://phabricator.wikimedia.org/T443456 [21:13:40] !log tsev@deploy2003 tsev: Backport for [[gerrit:1363467|Allow ReadingLists path in Apple App Site Association File (T443456)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:15:44] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97301 and previous config saved to /var/cache/conftool/dbconfig/20261104-211544-ladsgroup.json [21:16:39] changes look good, continuing deployment [21:16:43] !log tsev@deploy2003 tsev: Continuing with deployment [21:22:49] !log tsev@deploy2003 Finished scap sync-world: Backport for [[gerrit:1363467|Allow ReadingLists path in Apple App Site Association File (T443456)]] (duration: 11m 26s) [21:22:51] T443456: Discover: Allow Special:ReadingLists import links in the AASA file so they open in the iOS app - https://phabricator.wikimedia.org/T443456 [21:25:44] ejegg: done with mine [21:25:46] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97302 and previous config saved to /var/cache/conftool/dbconfig/20261104-212545-ladsgroup.json [21:26:12] thanks toni_ [21:28:10] 06SRE, 10LDAP-Access-Requests: Grant Access to [analytics resources: Superset, Jupyter, Growthbook, Turnilo, Test Kitchen UI] for cfoster - https://phabricator.wikimedia.org/T443699#12519566 (10cfoster) [21:28:14] (03CR) 10Ejegg: [C:03+2] Update CentralNotice for new mixin [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1363265 (owner: 10Ejegg) [21:28:17] (03CR) 10Ejegg: [C:03+2] Update CentralNotice for new mixin [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363263 (owner: 10Ejegg) [21:29:53] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet [21:29:55] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet [21:30:00] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet [21:31:43] (03Merged) 10jenkins-bot: Update CentralNotice for new mixin [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1363265 (owner: 10Ejegg) [21:32:07] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ejegg@deploy2003 using scap backport" [core] (wmf/1.48.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1363265 (owner: 10Ejegg) [21:32:08] (03CR) 10TrainBranchBot: [C:03+2] "Approved by ejegg@deploy2003 using scap backport" [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363263 (owner: 10Ejegg) [21:35:02] (03Merged) 10jenkins-bot: Update CentralNotice for new mixin [core] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363263 (owner: 10Ejegg) [21:35:23] PROBLEM - MariaDB Replica Lag: s4 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 608.13 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [21:35:27] !log ejegg@deploy2003 Started scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265|1.48.0-wmf.1]], [[gerrit:1363263|1.48.0-wmf.3]]) [21:35:48] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2182 (T441633)', diff saved to https://phabricator.wikimedia.org/P97303 and previous config saved to /var/cache/conftool/dbconfig/20261104-213547-ladsgroup.json [21:36:08] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance [21:40:48] (03PS1) 10Eric Gardner: ShareCard: Show animated GIFs as a still image [extensions/WikimediaCustomizations] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363474 (https://phabricator.wikimedia.org/T442968) [21:41:15] !log tsev@deploy2003 mwscript-k8s job started: purgeList.php # T443456 [21:41:18] T443456: Discover: Allow Special:ReadingLists import links in the AASA file so they open in the iOS app - https://phabricator.wikimedia.org/T443456 [21:42:06] (03PS1) 10Jforrester: wikifunctions: Drop the evaluators' "request complete" log line [deployment-charts] - 10https://gerrit.wikimedia.org/r/1363475 [21:43:48] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance [21:43:49] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2208 (T441633)', diff saved to https://phabricator.wikimedia.org/P97304 and previous config saved to /var/cache/conftool/dbconfig/20261104-214349-ladsgroup.json [21:48:14] !log tsev@deploy2003 mwscript-k8s job started: purgeList.php # T443456 [21:48:17] T443456: Discover: Allow Special:ReadingLists import links in the AASA file so they open in the iOS app - https://phabricator.wikimedia.org/T443456 [21:50:57] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2208 (T441633)', diff saved to https://phabricator.wikimedia.org/P97305 and previous config saved to /var/cache/conftool/dbconfig/20261104-215057-ladsgroup.json [21:52:04] FIRING: MediaWikiAccountCreationFailures: Elevated MediaWiki account creation failures: 90.04% - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=23 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiAccountCreationFailures [21:53:28] (03PS1) 10SBassett: admin: Add and update FIDO SSH key for sbassett [puppet] - 10https://gerrit.wikimedia.org/r/1363478 (https://phabricator.wikimedia.org/T443508) [21:53:34] 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Add FIDO-backed SSH key for sbassett - https://phabricator.wikimedia.org/T443508#12519691 (10sbassett) New patch coming that adds a backup FIDO and updates the first FIDO (since I apparently did not jot down the password correctly) [21:57:12] !log ejegg@deploy2003 ejegg: Backport for Update CentralNotice for new mixin ([[gerrit:1363265|1.48.0-wmf.1]], [[gerrit:1363263|1.48.0-wmf.3]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [21:57:19] verifying [22:00:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:00:04] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet [22:00:05] Deploy window Wikifunctions Services UTC Late (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T2200) [22:00:59] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97306 and previous config saved to /var/cache/conftool/dbconfig/20261104-220059-ladsgroup.json [22:01:16] (03PS1) 10BryanDavis: wmcs: use bd808 hardware-backed ssh key [puppet] - 10https://gerrit.wikimedia.org/r/1363482 (https://phabricator.wikimedia.org/T443002) [22:02:11] ok, UI is there on meta and subscribing module exists on wp.o [22:02:18] !log ejegg@deploy2003 ejegg: Continuing with deployment [22:05:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:05:44] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet [22:05:45] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet [22:05:50] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet [22:06:27] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet [22:11:01] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97307 and previous config saved to /var/cache/conftool/dbconfig/20261104-221101-ladsgroup.json [22:12:37] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet [22:12:37] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet [22:12:43] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet [22:13:19] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet [22:14:23] FIRING: HelmReleaseBadStatus: Helm release etherpad/main on k8s-aux@eqiad in state failed - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-aux&var-namespace=etherpad - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [22:17:00] !log ejegg@deploy2003 Finished scap sync-world: Backport for Update CentralNotice for new mixin ([[gerrit:1363265|1.48.0-wmf.1]], [[gerrit:1363263|1.48.0-wmf.3]]) (duration: 41m 33s) [22:17:27] RECOVERY - Confd vcl based reload on cp7001 is OK: reload-vcl successfully ran 0h, 0 minutes ago. https://wikitech.wikimedia.org/wiki/Varnish [22:19:01] ok, seeing the new CN feature on prod. looks good so far! [22:19:38] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet [22:19:39] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet [22:19:44] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet [22:20:20] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet [22:21:03] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2208 (T441633)', diff saved to https://phabricator.wikimedia.org/P97308 and previous config saved to /var/cache/conftool/dbconfig/20261104-222102-ladsgroup.json [22:21:23] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance [22:21:25] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2218 (T441633)', diff saved to https://phabricator.wikimedia.org/P97309 and previous config saved to /var/cache/conftool/dbconfig/20261104-222124-ladsgroup.json [22:26:30] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet [22:26:30] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet [22:26:36] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet [22:28:16] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2218 (T441633)', diff saved to https://phabricator.wikimedia.org/P97310 and previous config saved to /var/cache/conftool/dbconfig/20261104-222815-ladsgroup.json [22:28:38] !log btullis@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1018.eqiad.wmnet [22:36:23] RECOVERY - MariaDB Replica Lag: s4 on db2239 is OK: OK slave_sql_lag Replication lag: 0.29 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [22:38:18] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97311 and previous config saved to /var/cache/conftool/dbconfig/20261104-223817-ladsgroup.json [22:41:49] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet [22:41:50] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet [22:41:55] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet [22:43:04] (03PS2) 10Dillon: PersonalDashboard: Clean up config and set QA values for test and beta wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356990 (https://phabricator.wikimedia.org/T441729) [22:43:57] !log btullis@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1019.eqiad.wmnet [22:48:20] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97312 and previous config saved to /var/cache/conftool/dbconfig/20261104-224819-ladsgroup.json [22:50:17] (03CR) 10Dillon: "Thanks for the review! I moved the new values to T441729 and replied inline where I kept the existing tags. PS2 also makes the QA values t" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1356990 (https://phabricator.wikimedia.org/T441729) (owner: 10Dillon) [22:51:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:56:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [22:58:22] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2218 (T441633)', diff saved to https://phabricator.wikimedia.org/P97314 and previous config saved to /var/cache/conftool/dbconfig/20261104-225821-ladsgroup.json [22:58:41] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance [22:58:43] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2220 (T441633)', diff saved to https://phabricator.wikimedia.org/P97315 and previous config saved to /var/cache/conftool/dbconfig/20261104-225843-ladsgroup.json [23:00:00] o// starting a deploy shortly [23:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20261104T2300) [23:01:37] (03CR) 10TrainBranchBot: [C:03+2] "Approved by jdlrobson@deploy2003 using scap backport" [extensions/ReadingLists] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363460 (https://phabricator.wikimedia.org/T443481) (owner: 10Jdlrobson) [23:05:27] (03PS1) 10JHathaway: Remove legacy ssh keys for bearloga [puppet] - 10https://gerrit.wikimedia.org/r/1363492 (https://phabricator.wikimedia.org/T442758) [23:05:54] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2220 (T441633)', diff saved to https://phabricator.wikimedia.org/P97316 and previous config saved to /var/cache/conftool/dbconfig/20261104-230554-ladsgroup.json [23:06:10] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet [23:06:11] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet [23:06:17] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet [23:07:09] (03CR) 10JHathaway: [C:03+2] Remove legacy ssh keys for bearloga [puppet] - 10https://gerrit.wikimedia.org/r/1363492 (https://phabricator.wikimedia.org/T442758) (owner: 10JHathaway) [23:07:22] (03Merged) 10jenkins-bot: UnsaveDialog: Don't render in place in bottom sheet mode [extensions/ReadingLists] (wmf/1.48.0-wmf.3) - 10https://gerrit.wikimedia.org/r/1363460 (https://phabricator.wikimedia.org/T443481) (owner: 10Jdlrobson) [23:07:54] !log jdlrobson@deploy2003 Started scap sync-world: Backport for [[gerrit:1363460|UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) [23:07:57] T443481: Unsave confirmation dialog is partially hidden on small screens (and/or with a page in numerous collections) - https://phabricator.wikimedia.org/T443481 [23:08:19] !log btullis@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet [23:13:15] PROBLEM - MariaDB Replica Lag: s3 on db2239 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 646.13 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [23:15:20] !log jdlrobson@deploy2003 jdlrobson: Backport for [[gerrit:1363460|UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [23:15:23] T443481: Unsave confirmation dialog is partially hidden on small screens (and/or with a page in numerous collections) - https://phabricator.wikimedia.org/T443481 [23:15:56] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97317 and previous config saved to /var/cache/conftool/dbconfig/20261104-231555-ladsgroup.json [23:16:17] !log jdlrobson@deploy2003 jdlrobson: Continuing with deployment [23:19:20] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12520049 (10VRiley-WMF) Collecting logs to give to Dell [23:24:29] !log jdlrobson@deploy2003 Finished scap sync-world: Backport for [[gerrit:1363460|UnsaveDialog: Don't render in place in bottom sheet mode (T443481)]] (1.48.0-wmf.3) (duration: 16m 34s) [23:24:31] T443481: Unsave confirmation dialog is partially hidden on small screens (and/or with a page in numerous collections) - https://phabricator.wikimedia.org/T443481 [23:25:52] done [23:25:58] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97318 and previous config saved to /var/cache/conftool/dbconfig/20261104-232557-ladsgroup.json [23:27:31] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet [23:27:32] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet [23:27:37] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet [23:27:43] (03PS1) 10SBassett: Enable DBSC report-only mode on all Wikimedia wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363494 (https://phabricator.wikimedia.org/T393212) [23:28:19] (03CR) 10SBassett: "Previous: Ie8e2daa40" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363494 (https://phabricator.wikimedia.org/T393212) (owner: 10SBassett) [23:28:57] (03CR) 10SBassett: [C:04-2] "Hold for config deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363494 (https://phabricator.wikimedia.org/T393212) (owner: 10SBassett) [23:33:55] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, November 10 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal-" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1363494 (https://phabricator.wikimedia.org/T393212) (owner: 10SBassett) [23:36:00] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2220 (T441633)', diff saved to https://phabricator.wikimedia.org/P97319 and previous config saved to /var/cache/conftool/dbconfig/20261104-233559-ladsgroup.json [23:36:19] !log ladsgroup@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance [23:36:21] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Depooling db2221 (T441633)', diff saved to https://phabricator.wikimedia.org/P97320 and previous config saved to /var/cache/conftool/dbconfig/20261104-233620-ladsgroup.json [23:43:31] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2221 (T441633)', diff saved to https://phabricator.wikimedia.org/P97321 and previous config saved to /var/cache/conftool/dbconfig/20261104-234330-ladsgroup.json [23:45:03] FIRING: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [23:46:53] 10ops-eqiad, 06SRE, 06DBA, 06DC-Ops: db1262 crashed - https://phabricator.wikimedia.org/T442800#12520112 (10VRiley-WMF) Created a service request being 233830087 [23:50:03] RESOLVED: MediaWikiLoginFailures: Elevated MediaWiki centrallogin failures (centralauth_error_badtoken) - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLoginFailures [23:53:33] !log ladsgroup@cumin1004 dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97322 and previous config saved to /var/cache/conftool/dbconfig/20261104-235332-ladsgroup.json [23:57:15] RECOVERY - MariaDB Replica Lag: s3 on db2239 is OK: OK slave_sql_lag Replication lag: 0.28 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [23:57:40] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet