[00:01:38] <wikibugs>	 (03PS1) 10Urbanecm: Babel: Remove config that is now in community configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115518 (https://phabricator.wikimedia.org/T385239)
[00:04:28] <wikibugs>	 (03PS1) 10Urbanecm: Babel: Do not use a wmg variable for BabelDefaultLevel [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115520 (https://phabricator.wikimedia.org/T119117)
[00:04:40] <logmsgbot>	 !log pt1979@cumin1002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin1002"
[00:05:16] <logmsgbot>	 !log pt1979@cumin1002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin1002"
[00:05:16] <logmsgbot>	 !log pt1979@cumin1002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1252.eqiad.wmnet with OS bookworm
[00:05:21] <wikibugs>	 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: Q2:rack/setup/install db125[0-4] - https://phabricator.wikimedia.org/T380083#10510761 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1002 for host db1252.eqiad.wmnet with OS bookworm completed: - db1252 (**PASS**)   - Rem...
[00:05:47] <wikibugs>	 (03CR) 10Urbanecm: [C:04-2] "let's wait a bit, in case we need to rollback the deployment" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115518 (https://phabricator.wikimedia.org/T385239) (owner: 10Urbanecm)
[00:06:15] <logmsgbot>	 !log pt1979@cumin1002 START - Cookbook sre.hosts.reimage for host db1253.eqiad.wmnet with OS bookworm
[00:06:22] <wikibugs>	 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: Q2:rack/setup/install db125[0-4] - https://phabricator.wikimedia.org/T380083#10510772 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1002 for host db1253.eqiad.wmnet with OS bookworm
[00:07:05] <logmsgbot>	 !log pt1979@cumin1002 START - Cookbook sre.hosts.provision for host db1254.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
[00:09:21] <wikibugs>	 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: Q2:rack/setup/install db125[0-4] - https://phabricator.wikimedia.org/T380083#10510776 (10Papaul)
[00:16:40] <jinxer-wm>	 FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1146:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1146 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown
[00:21:53] <logmsgbot>	 !log pt1979@cumin1002 START - Cookbook sre.hosts.downtime for 2:00:00 on db1253.eqiad.wmnet with reason: host reimage
[00:22:58] <wikibugs>	 (03PS2) 10Dwisehaupt: Add aws validation key for lp.email ssl cert generation [dns] - 10https://gerrit.wikimedia.org/r/1115494 (https://phabricator.wikimedia.org/T384931)
[00:23:43] <wikibugs>	 (03CR) 10Dwisehaupt: Add aws validation key for lp.email ssl cert generation (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/1115494 (https://phabricator.wikimedia.org/T384931) (owner: 10Dwisehaupt)
[00:25:22] <logmsgbot>	 !log pt1979@cumin1002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1253.eqiad.wmnet with reason: host reimage
[00:30:51] <logmsgbot>	 !log pt1979@cumin1002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db1254.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
[00:32:53] <wikibugs>	 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: Q2:rack/setup/install db125[0-4] - https://phabricator.wikimedia.org/T380083#10510851 (10Papaul)
[00:33:54] <logmsgbot>	 !log pt1979@cumin1002 START - Cookbook sre.hosts.reimage for host db1254.eqiad.wmnet with OS bookworm
[00:34:03] <wikibugs>	 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: Q2:rack/setup/install db125[0-4] - https://phabricator.wikimedia.org/T380083#10510852 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1002 for host db1254.eqiad.wmnet with OS bookworm
[00:38:08] <wikibugs>	 (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1115540
[00:38:08] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1115540 (owner: 10TrainBranchBot)
[00:42:41] <logmsgbot>	 !log pt1979@cumin1002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin1002"
[00:42:59] <logmsgbot>	 !log pt1979@cumin1002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin1002"
[00:43:00] <logmsgbot>	 !log pt1979@cumin1002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1253.eqiad.wmnet with OS bookworm
[00:43:13] <wikibugs>	 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: Q2:rack/setup/install db125[0-4] - https://phabricator.wikimedia.org/T380083#10510857 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1002 for host db1253.eqiad.wmnet with OS bookworm completed: - db1253 (**PASS**)   - Rem...
[00:44:14] <wikibugs>	 (03PS1) 10Zabe: hooks: Stop using deprecated DatabaseBlock method [extensions/TorBlock] (wmf/1.44.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1115542
[00:45:01] <wikibugs>	 (03CR) 10Zabe: [C:03+2] hooks: Stop using deprecated DatabaseBlock method [extensions/TorBlock] (wmf/1.44.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1115542 (owner: 10Zabe)
[00:45:08] <wikibugs>	 (03CR) 10Zabe: [C:03+2] Restrict editing on ruwikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115146 (https://phabricator.wikimedia.org/T382805) (owner: 10Zabe)
[00:45:58] <wikibugs>	 (03Merged) 10jenkins-bot: Restrict editing on ruwikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115146 (https://phabricator.wikimedia.org/T382805) (owner: 10Zabe)
[00:46:40] <jinxer-wm>	 RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1146:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1146 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown
[00:49:31] <wikibugs>	 (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1115540 (owner: 10TrainBranchBot)
[00:49:31] <jinxer-wm>	 FIRING: [2x] CirrusSearchHighOldGCFrequency: Elasticsearch instance elastic1071-production-search-omega-eqiad is running the old gc excessively - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Stuck_in_old_GC_hell - https://grafana.wikimedia.org/d/000000462/elasticsearch-memory - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchHighOldGCFrequency
[00:50:33] <logmsgbot>	 !log pt1979@cumin1002 START - Cookbook sre.hosts.downtime for 2:00:00 on db1254.eqiad.wmnet with reason: host reimage
[00:50:53] <wikibugs>	 (03Merged) 10jenkins-bot: hooks: Stop using deprecated DatabaseBlock method [extensions/TorBlock] (wmf/1.44.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1115542 (owner: 10Zabe)
[00:51:35] <logmsgbot>	 !log zabe@deploy2002 Started scap sync-world: Backport for [[gerrit:1115542|hooks: Stop using deprecated DatabaseBlock method]], [[gerrit:1115146|Restrict editing on ruwikimedia (T382805)]]
[00:51:40] <stashbot>	 T382805: WMRU wiki (ru.wikimedia.org) restricting access - https://phabricator.wikimedia.org/T382805
[00:53:28] <jinxer-wm>	 FIRING: [2x] KubernetesAPINotScrapable: k8s-staging@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable
[00:53:29] <logmsgbot>	 !log pt1979@cumin1002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1254.eqiad.wmnet with reason: host reimage
[00:54:21] <logmsgbot>	 !log zabe@deploy2002 zabe: Backport for [[gerrit:1115542|hooks: Stop using deprecated DatabaseBlock method]], [[gerrit:1115146|Restrict editing on ruwikimedia (T382805)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
[00:56:19] <logmsgbot>	 !log zabe@deploy2002 zabe: Continuing with sync
[00:56:56] <jinxer-wm>	 FIRING: CalicoTyphaDown: Too few (0) calico-typha replicas running - https://wikitech.wikimedia.org/wiki/Calico#Typha" - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoTyphaDown
[00:56:57] <jinxer-wm>	 FIRING: CalicoKubeControllersDown: Calico Kubernetes Controllers not running - https://wikitech.wikimedia.org/wiki/Calico#Kube_Controllers" - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoKubeControllersDown
[01:02:43] <logmsgbot>	 !log zabe@deploy2002 Finished scap sync-world: Backport for [[gerrit:1115542|hooks: Stop using deprecated DatabaseBlock method]], [[gerrit:1115146|Restrict editing on ruwikimedia (T382805)]] (duration: 11m 08s)
[01:02:48] <stashbot>	 T382805: WMRU wiki (ru.wikimedia.org) restricting access - https://phabricator.wikimedia.org/T382805
[01:04:38] <wikibugs>	 (03CR) 10BCornwall: [C:03+1] Add aws validation key for lp.email ssl cert generation [dns] - 10https://gerrit.wikimedia.org/r/1115494 (https://phabricator.wikimedia.org/T384931) (owner: 10Dwisehaupt)
[01:08:39] <wikibugs>	 (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1115543
[01:09:30] <wikibugs>	 (03CR) 10Andrew Bogott: [C:03+2] Revert "nova: temporary mod of live_migration rules" [puppet] - 10https://gerrit.wikimedia.org/r/1115507 (owner: 10Andrew Bogott)
[01:11:03] <logmsgbot>	 !log pt1979@cumin1002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin1002"
[01:12:05] <logmsgbot>	 !log pt1979@cumin1002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin1002"
[01:12:06] <logmsgbot>	 !log pt1979@cumin1002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1254.eqiad.wmnet with OS bookworm
[01:12:13] <wikibugs>	 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: Q2:rack/setup/install db125[0-4] - https://phabricator.wikimedia.org/T380083#10510891 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1002 for host db1254.eqiad.wmnet with OS bookworm completed: - db1254 (**PASS**)   - Rem...
[01:13:55] <wikibugs>	 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: Q2:rack/setup/install db125[0-4] - https://phabricator.wikimedia.org/T380083#10510895 (10Papaul)
[01:14:32] <wikibugs>	 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: Q2:rack/setup/install db125[0-4] - https://phabricator.wikimedia.org/T380083#10510897 (10Papaul) 05Open→03Resolved a:03Papaul @Marostegui  this is complete
[01:19:59] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops, 10fundraising-tech-ops: Q1:rack/setup/install franio100[1-3] - https://phabricator.wikimedia.org/T367820#10510915 (10Papaul) @VRiley-WMF these servers are already racked what left to do is just connect the mgmt cable and 2 DAC cables using the 10G nics first nic to the first s...
[01:26:29] <wikibugs>	 (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1115543 (owner: 10TrainBranchBot)
[01:46:28] <icinga-wm>	 PROBLEM - Disk space on releases1003 is CRITICAL: DISK CRITICAL - /srv/docker/overlay2/b6389289f85d9d4e7027f3c9d1b815b3cb57b6919da6a314d706f2fd76e433ea/merged is not accessible: Permission denied https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=releases1003&var-datasource=eqiad+prometheus/ops
[01:50:56] <wikibugs>	 (03PS1) 10Zabe: manage-dblist: strtolower() visibility input [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115558
[01:52:20] <wikibugs>	 (03CR) 10Zabe: [C:03+2] manage-dblist: strtolower() visibility input [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115558 (owner: 10Zabe)
[01:53:39] <wikibugs>	 (03Merged) 10jenkins-bot: manage-dblist: strtolower() visibility input [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115558 (owner: 10Zabe)
[02:06:28] <icinga-wm>	 RECOVERY - Disk space on releases1003 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=releases1003&var-datasource=eqiad+prometheus/ops
[02:15:26] <wikibugs>	 (03PS1) 10Zabe: Prepare kncwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115567 (https://phabricator.wikimedia.org/T385181)
[02:22:06] <wikibugs>	 (03CR) 10Zabe: [C:03+2] Prepare kncwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115567 (https://phabricator.wikimedia.org/T385181) (owner: 10Zabe)
[02:22:52] <wikibugs>	 (03Merged) 10jenkins-bot: Prepare kncwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115567 (https://phabricator.wikimedia.org/T385181) (owner: 10Zabe)
[02:23:15] <logmsgbot>	 !log zabe@deploy2002 Started scap sync-world: Backport for [[gerrit:1115567|Prepare kncwiki (T385181)]]
[02:23:21] <stashbot>	 T385181: Create Wikipedia Central Kanuri - https://phabricator.wikimedia.org/T385181
[02:25:56] <logmsgbot>	 !log zabe@deploy2002 zabe: Backport for [[gerrit:1115567|Prepare kncwiki (T385181)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
[02:26:21] <logmsgbot>	 !log zabe@deploy2002 zabe: Continuing with sync
[02:32:25] <logmsgbot>	 !log zabe@deploy2002 Finished scap sync-world: Backport for [[gerrit:1115567|Prepare kncwiki (T385181)]] (duration: 09m 10s)
[02:32:30] <stashbot>	 T385181: Create Wikipedia Central Kanuri - https://phabricator.wikimedia.org/T385181
[02:34:08] <wikibugs>	 (03PS1) 10Zabe: Activate kncwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115577 (https://phabricator.wikimedia.org/T385181)
[02:35:35] <wikibugs>	 (03CR) 10Zabe: [C:03+2] Activate kncwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115577 (https://phabricator.wikimedia.org/T385181) (owner: 10Zabe)
[02:36:42] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[02:37:38] <wikibugs>	 (03Merged) 10jenkins-bot: Activate kncwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115577 (https://phabricator.wikimedia.org/T385181) (owner: 10Zabe)
[02:37:58] <logmsgbot>	 !log zabe@deploy2002 Started scap sync-world: Backport for [[gerrit:1115577|Activate kncwiki (T385181)]]
[02:38:03] <stashbot>	 T385181: Create Wikipedia Central Kanuri - https://phabricator.wikimedia.org/T385181
[02:40:43] <logmsgbot>	 !log zabe@deploy2002 zabe: Backport for [[gerrit:1115577|Activate kncwiki (T385181)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
[02:42:33] <logmsgbot>	 !log zabe@deploy2002 zabe: Continuing with sync
[02:49:00] <logmsgbot>	 !log zabe@deploy2002 Finished scap sync-world: Backport for [[gerrit:1115577|Activate kncwiki (T385181)]] (duration: 11m 02s)
[02:49:06] <stashbot>	 T385181: Create Wikipedia Central Kanuri - https://phabricator.wikimedia.org/T385181
[02:49:41] <wikibugs>	 (03PS1) 10Zabe: Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115582
[02:49:41] <wikibugs>	 (03CR) 10Zabe: [C:03+2] Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115582 (owner: 10Zabe)
[02:50:40] <wikibugs>	 (03Merged) 10jenkins-bot: Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115582 (owner: 10Zabe)
[03:01:42] <jinxer-wm>	 RESOLVED: JobUnavailable: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[03:04:50] <wikibugs>	 (03PS1) 10Zabe: configure kncwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115590 (https://phabricator.wikimedia.org/T385185)
[03:05:35] <wikibugs>	 (03CR) 10CI reject: [V:04-1] configure kncwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115590 (https://phabricator.wikimedia.org/T385185) (owner: 10Zabe)
[03:06:45] <wikibugs>	 (03PS2) 10Zabe: configure kncwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115590 (https://phabricator.wikimedia.org/T385185)
[03:07:26] <wikibugs>	 (03CR) 10CI reject: [V:04-1] configure kncwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115590 (https://phabricator.wikimedia.org/T385185) (owner: 10Zabe)
[03:08:01] <wikibugs>	 (03PS3) 10Zabe: configure kncwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115590 (https://phabricator.wikimedia.org/T385185)
[03:10:15] <wikibugs>	 (03CR) 10Zabe: [C:03+2] configure kncwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115590 (https://phabricator.wikimedia.org/T385185) (owner: 10Zabe)
[03:10:58] <wikibugs>	 (03Merged) 10jenkins-bot: configure kncwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115590 (https://phabricator.wikimedia.org/T385185) (owner: 10Zabe)
[03:11:34] <logmsgbot>	 !log zabe@deploy2002 Started scap sync-world: Backport for [[gerrit:1115582|Update interwiki cache]], [[gerrit:1115590|configure kncwiki (T385185)]]
[03:11:40] <stashbot>	 T385185: Post-creation work for kncwiki - https://phabricator.wikimedia.org/T385185
[03:14:20] <logmsgbot>	 !log zabe@deploy2002 zabe: Backport for [[gerrit:1115582|Update interwiki cache]], [[gerrit:1115590|configure kncwiki (T385185)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
[03:15:40] <logmsgbot>	 !log zabe@deploy2002 zabe: Continuing with sync
[03:22:03] <logmsgbot>	 !log zabe@deploy2002 Finished scap sync-world: Backport for [[gerrit:1115582|Update interwiki cache]], [[gerrit:1115590|configure kncwiki (T385185)]] (duration: 10m 28s)
[03:22:08] <stashbot>	 T385185: Post-creation work for kncwiki - https://phabricator.wikimedia.org/T385185
[03:34:26] <wikibugs>	 10ops-eqiad, 06DC-Ops: ManagementSSHDown - https://phabricator.wikimedia.org/T385251 (10phaultfinder) 03NEW
[03:35:20] <icinga-wm>	 PROBLEM - Disk space on grafana2001 is CRITICAL: DISK CRITICAL - free space: / 239MiB (1% inode=36%): /tmp 239MiB (1% inode=36%): /var/tmp 239MiB (1% inode=36%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=grafana2001&var-datasource=codfw+prometheus/ops
[03:55:20] <icinga-wm>	 RECOVERY - Disk space on grafana2001 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=grafana2001&var-datasource=codfw+prometheus/ops
[04:50:54] <icinga-wm>	 PROBLEM - BGP status on cr3-ulsfo is CRITICAL: BGP CRITICAL - AS1299/IPv6: Connect - Arelion, AS1299/IPv4: Connect - Arelion https://wikitech.wikimedia.org/wiki/Network_monitoring%23BGP_status
[04:51:50] <icinga-wm>	 PROBLEM - Host mr1-ulsfo.oob is DOWN: PING CRITICAL - Packet loss = 100%
[04:52:07] <jinxer-wm>	 FIRING: CirrusSearchHighOldGCFrequency: Elasticsearch instance elastic1096-production-search-omega-eqiad is running the old gc excessively - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Stuck_in_old_GC_hell - https://grafana.wikimedia.org/d/000000462/elasticsearch-memory - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchHighOldGCFrequency
[04:53:28] <jinxer-wm>	 FIRING: [2x] KubernetesAPINotScrapable: k8s-staging@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable
[04:56:52] <icinga-wm>	 RECOVERY - Host mr1-ulsfo.oob is UP: PING OK - Packet loss = 0%, RTA = 77.25 ms
[04:56:56] <jinxer-wm>	 FIRING: CalicoTyphaDown: Too few (0) calico-typha replicas running - https://wikitech.wikimedia.org/wiki/Calico#Typha" - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoTyphaDown
[04:56:58] <jinxer-wm>	 FIRING: CalicoKubeControllersDown: Calico Kubernetes Controllers not running - https://wikitech.wikimedia.org/wiki/Calico#Kube_Controllers" - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoKubeControllersDown
[05:47:17] <logmsgbot>	 !log marostegui@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1189.eqiad.wmnet with reason: Maintenance
[06:08:02] <wikibugs>	 (03PS1) 10Marostegui: check_private_data_report: Add Federico's email. [puppet] - 10https://gerrit.wikimedia.org/r/1115654
[06:10:01] <marostegui>	 !log Deploy schema change on s5 codfw with replication enabled dbmaint T385183
[06:10:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:10:06] <stashbot>	 T385183: Run the T383561 ALTERs on wikifunctionswiki - https://phabricator.wikimedia.org/T385183
[06:10:49] <logmsgbot>	 !log marostegui@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2209.codfw.wmnet with reason: Maintenance
[06:30:23] <wikibugs>	 10ops-eqiad, 06SRE, 06Data-Persistence, 06DC-Ops: Q2:rack/setup/install db125[0-4] - https://phabricator.wikimedia.org/T380083#10511085 (10Marostegui) Thank you!
[06:31:18] <wikibugs>	 (03PS1) 10KartikMistry: Update Recommendation API to 2025-01-17-163931-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115668 (https://phabricator.wikimedia.org/T381888)
[06:38:34] <wikibugs>	 (03PS1) 10Marostegui: Revert "db2172: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1115675
[06:39:18] <wikibugs>	 (03CR) 10Marostegui: [C:03+2] Revert "db2172: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/1115675 (owner: 10Marostegui)
[07:00:05] <jouncebot>	 Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250131T0700)
[07:24:11] <wikibugs>	 (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1115444 (https://phabricator.wikimedia.org/T350794) (owner: 10AOkoth)
[07:30:53] <logmsgbot>	 !log jmm@cumin2002 START - Cookbook sre.hosts.reimage for host ganeti2029.codfw.wmnet with OS bookworm
[07:30:58] <wikibugs>	 06SRE, 10Ganeti, 06Infrastructure-Foundations: Update remaining Ganeti servers in codfw to Bookworm - https://phabricator.wikimedia.org/T382508#10511134 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti2029.codfw.wmnet with OS bookworm
[07:35:20] <icinga-wm>	 PROBLEM - Disk space on grafana2001 is CRITICAL: DISK CRITICAL - free space: / 237MiB (1% inode=36%): /tmp 237MiB (1% inode=36%): /var/tmp 237MiB (1% inode=36%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=grafana2001&var-datasource=codfw+prometheus/ops
[07:46:29] <wikibugs>	 (03CR) 10AOkoth: [C:03+2] os_updates: open up rsync to staging kube pods [puppet] - 10https://gerrit.wikimedia.org/r/1115444 (https://phabricator.wikimedia.org/T350794) (owner: 10AOkoth)
[07:52:32] <logmsgbot>	 !log jmm@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2029.codfw.wmnet with reason: host reimage
[07:55:20] <icinga-wm>	 RECOVERY - Disk space on grafana2001 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=grafana2001&var-datasource=codfw+prometheus/ops
[07:56:20] <logmsgbot>	 !log jmm@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2029.codfw.wmnet with reason: host reimage
[08:00:04] <jouncebot>	 Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250131T0800)
[08:01:29] <wikibugs>	 (03CR) 10Jelto: [C:03+1] kubernetes-publish-sa-cert: Don't fail when no certs in etcd [puppet] - 10https://gerrit.wikimedia.org/r/1115433 (https://phabricator.wikimedia.org/T341984) (owner: 10JMeybohm)
[08:11:28] <wikibugs>	 06SRE, 06Infrastructure-Foundations, 10netops, 10observability, and 3 others: Prevent BGP alerts triggering when K8s host maintenance is being done - https://phabricator.wikimedia.org/T384731#10511163 (10fgiunchedi) Yes we would be able to [[ https://prometheus.io/docs/alerting/latest/configuration/#inhibi...
[08:14:43] <wikibugs>	 (03CR) 10Brouberol: [V:03+1 C:03+2] Update the airflow deb version to downgrade confluent-kafka [puppet] - 10https://gerrit.wikimedia.org/r/1115515 (owner: 10Brouberol)
[08:16:15] <logmsgbot>	 !log jmm@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2029.codfw.wmnet with OS bookworm
[08:16:25] <wikibugs>	 06SRE, 10Ganeti, 06Infrastructure-Foundations: Update remaining Ganeti servers in codfw to Bookworm - https://phabricator.wikimedia.org/T382508#10511166 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti2029.codfw.wmnet with OS bookworm completed: - ganeti202...
[08:16:42] <wikibugs>	 (03CR) 10Elukey: "I rechecked the knative PR and it relates to https://github.com/knative/serving/pull/13398, that is probably worth to backport as well.." [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1115394 (https://phabricator.wikimedia.org/T369493) (owner: 10Elukey)
[08:22:47] <logmsgbot>	 !log jmm@cumin2002 START - Cookbook sre.hosts.reboot-single for host ganeti2029.codfw.wmnet
[08:31:12] <logmsgbot>	 !log jmm@cumin2002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2029.codfw.wmnet
[08:33:42] <wikibugs>	 (03PS3) 10Elukey: knative: backport patches from 1.8.x release [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1115394 (https://phabricator.wikimedia.org/T369493)
[08:34:04] <wikibugs>	 (03CR) 10KartikMistry: [C:03+2] Update Recommendation API to 2025-01-17-163931-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115668 (https://phabricator.wikimedia.org/T381888) (owner: 10KartikMistry)
[08:34:05] <wikibugs>	 (03CR) 10Elukey: "Added the extra patch, rebuilt the images, they look ok!" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1115394 (https://phabricator.wikimedia.org/T369493) (owner: 10Elukey)
[08:34:59] <logmsgbot>	 !log jmm@cumin2002 START - Cookbook sre.ganeti.addnode for new host ganeti2029.codfw.wmnet to cluster codfw and group A
[08:35:13] <wikibugs>	 (03Merged) 10jenkins-bot: Update Recommendation API to 2025-01-17-163931-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115668 (https://phabricator.wikimedia.org/T381888) (owner: 10KartikMistry)
[08:36:07] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C:03+2] Update webrequest_sampled_live turnilo config [puppet] - 10https://gerrit.wikimedia.org/r/1115457 (https://phabricator.wikimedia.org/T383900) (owner: 10Joal)
[08:36:08] <wikibugs>	 (03CR) 10Fabfur: benthos: send data to eventgate too (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1115113 (https://phabricator.wikimedia.org/T383392) (owner: 10Fabfur)
[08:37:00] <logmsgbot>	 !log kartik@deploy2002 helmfile [ml-staging-codfw] 'sync' command on namespace 'recommendation-api-ng' for release 'main' .
[08:37:14] <wikibugs>	 (03PS5) 10Fabfur: benthos: send data to eventgate too [puppet] - 10https://gerrit.wikimedia.org/r/1115113 (https://phabricator.wikimedia.org/T383392)
[08:37:47] <logmsgbot>	 !log jmm@cumin2002 END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2029.codfw.wmnet to cluster codfw and group A
[08:40:01] <wikibugs>	 (03CR) 10JMeybohm: [C:03+1] "Yeah, I think it's the correct way to ensure the mw instance gets to see the 'right' host." [puppet] - 10https://gerrit.wikimedia.org/r/1115416 (https://phabricator.wikimedia.org/T384329) (owner: 10Brouberol)
[08:41:53] <wikibugs>	 (03CR) 10Fabfur: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1115113 (https://phabricator.wikimedia.org/T383392) (owner: 10Fabfur)
[08:42:59] <wikibugs>	 (03CR) 10Brouberol: [C:03+2] "Thanks to the both of you!" [puppet] - 10https://gerrit.wikimedia.org/r/1115416 (https://phabricator.wikimedia.org/T384329) (owner: 10Brouberol)
[08:43:01] <wikibugs>	 06SRE, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Add x-analytics nocookie=1 and x-tls-sess to webrequest-sampled-live stream - https://phabricator.wikimedia.org/T383900#10511213 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This is done! Fields show up in the kafka topic...
[08:43:46] <wikibugs>	 (03CR) 10JMeybohm: k8s.pool-depool-node: Add support to downtime/remove downtime (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1114000 (owner: 10JMeybohm)
[08:50:58] <wikibugs>	 06SRE, 10Ganeti, 06Infrastructure-Foundations: Update remaining Ganeti servers in codfw to Bookworm - https://phabricator.wikimedia.org/T382508#10511228 (10MoritzMuehlenhoff)
[08:51:43] <wikibugs>	 06SRE, 10Ganeti, 06Infrastructure-Foundations: Update remaining Ganeti servers in codfw to Bookworm - https://phabricator.wikimedia.org/T382508#10511229 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff All Ganeti nodes in codfw have been upgraded to Bookworm (and also migrated to nftab...
[08:52:07] <jinxer-wm>	 FIRING: CirrusSearchHighOldGCFrequency: Elasticsearch instance elastic1096-production-search-omega-eqiad is running the old gc excessively - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Stuck_in_old_GC_hell - https://grafana.wikimedia.org/d/000000462/elasticsearch-memory - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchHighOldGCFrequency
[08:52:48] <moritzm>	 !log rebalance codfw/A following OS updates T382508
[08:52:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:52:53] <stashbot>	 T382508: Update remaining Ganeti servers in codfw to Bookworm - https://phabricator.wikimedia.org/T382508
[08:53:28] <jinxer-wm>	 FIRING: [2x] KubernetesAPINotScrapable: k8s-staging@codfw is failing to scrape the k8s api - https://phabricator.wikimedia.org/T343529 - TODO - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPINotScrapable
[08:56:56] <jinxer-wm>	 FIRING: CalicoTyphaDown: Too few (0) calico-typha replicas running - https://wikitech.wikimedia.org/wiki/Calico#Typha" - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoTyphaDown
[08:56:58] <jinxer-wm>	 FIRING: CalicoKubeControllersDown: Calico Kubernetes Controllers not running - https://wikitech.wikimedia.org/wiki/Calico#Kube_Controllers" - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoKubeControllersDown
[09:05:30] <logmsgbot>	 !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
[09:05:35] <wikibugs>	 (03PS1) 10Jelto: trafficserver: switch query-main and query service to wikikube [puppet] - 10https://gerrit.wikimedia.org/r/1115766 (https://phabricator.wikimedia.org/T350793)
[09:05:53] <wikibugs>	 (03PS1) 10JMeybohm: Add interative.ask_yesno [software/pywmflib] - 10https://gerrit.wikimedia.org/r/1115767
[09:06:10] <logmsgbot>	 !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
[09:10:55] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add interative.ask_yesno [software/pywmflib] - 10https://gerrit.wikimedia.org/r/1115767 (owner: 10JMeybohm)
[09:13:54] <icinga-wm>	 PROBLEM - BFD status on cr2-magru is CRITICAL: Down: 1 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status
[09:14:08] <icinga-wm>	 PROBLEM - BFD status on cr2-eqdfw is CRITICAL: Down: 1 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status
[09:14:54] <icinga-wm>	 RECOVERY - BFD status on cr2-magru is OK: UP: 3 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status
[09:14:54] <wikibugs>	 (03PS1) 10Jcrespo: dbbackups: Decommission db2139 [puppet] - 10https://gerrit.wikimedia.org/r/1115774 (https://phabricator.wikimedia.org/T383971)
[09:15:08] <icinga-wm>	 RECOVERY - BFD status on cr2-eqdfw is OK: UP: 16 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status
[09:16:39] <logmsgbot>	 !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
[09:17:16] <logmsgbot>	 !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
[09:20:29] <wikibugs>	 (03PS3) 10JMeybohm: k8s.wipe-cluster: Improvements for k8s 1.31 upgrade [cookbooks] - 10https://gerrit.wikimedia.org/r/1115380 (https://phabricator.wikimedia.org/T341984)
[09:21:49] <wikibugs>	 (03PS1) 10Brouberol: envoy: adjust the noc config to use both SNI and a custom HTTP host header [puppet] - 10https://gerrit.wikimedia.org/r/1115776 (https://phabricator.wikimedia.org/T384329)
[09:24:35] <wikibugs>	 (03PS2) 10Kevin Bazira: ml-services: update article-country staging config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115359 (https://phabricator.wikimedia.org/T382295)
[09:24:41] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: PDU sensor over limit - https://phabricator.wikimedia.org/T383383#10511260 (10phaultfinder)
[09:26:21] <wikibugs>	 (03CR) 10CI reject: [V:04-1] k8s.wipe-cluster: Improvements for k8s 1.31 upgrade [cookbooks] - 10https://gerrit.wikimedia.org/r/1115380 (https://phabricator.wikimedia.org/T341984) (owner: 10JMeybohm)
[09:27:49] <wikibugs>	 (03PS1) 10Brouberol: airflow-analytics: replace the mw-misc by the noc listener [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115777 (https://phabricator.wikimedia.org/T384329)
[09:28:55] <wikibugs>	 (03CR) 10Urbanecm: [C:03+1] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1113997 (https://phabricator.wikimedia.org/T325990) (owner: 10Sergio Gimeno)
[09:29:50] <wikibugs>	 (03PS8) 10JMeybohm: k8s::client: Allow for install of all kubectl versions [puppet] - 10https://gerrit.wikimedia.org/r/1114970 (https://phabricator.wikimedia.org/T341984)
[09:29:50] <wikibugs>	 (03PS1) 10JMeybohm: kubernetes::master: Update kubectl alternative entry [puppet] - 10https://gerrit.wikimedia.org/r/1115778 (https://phabricator.wikimedia.org/T341984)
[09:32:56] <wikibugs>	 (03PS1) 10Slyngshede: Switch to CAS 7.1 - Test [dns] - 10https://gerrit.wikimedia.org/r/1115780
[09:35:49] <wikibugs>	 (03CR) 10Slyngshede: [C:03+2] Switch to CAS 7.1 - Test [dns] - 10https://gerrit.wikimedia.org/r/1115780 (owner: 10Slyngshede)
[09:35:57] <logmsgbot>	 !log slyngshede@dns1004 START - running authdns-update
[09:37:49] <logmsgbot>	 !log slyngshede@dns1004 END - running authdns-update
[09:38:00] <wikibugs>	 (03PS2) 10Jcrespo: dbbackups: Decommission db2139 [puppet] - 10https://gerrit.wikimedia.org/r/1115774 (https://phabricator.wikimedia.org/T383971)
[09:38:01] <wikibugs>	 (03PS1) 10Jelto: wikidata-query-gui: bump query-gui image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115786 (https://phabricator.wikimedia.org/T350793)
[09:39:03] <wikibugs>	 (03CR) 10JMeybohm: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1115778 (https://phabricator.wikimedia.org/T341984) (owner: 10JMeybohm)
[09:39:37] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: PDU sensor over limit - https://phabricator.wikimedia.org/T383383#10511284 (10phaultfinder)
[09:40:13] <wikibugs>	 (03PS2) 10Brouberol: airflow-analytics: replace the mw-misc by the noc listener [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115777 (https://phabricator.wikimedia.org/T384329)
[09:40:17] <logmsgbot>	 !log jynus@cumin1002 START - Cookbook sre.hosts.decommission for hosts db2139.codfw.wmnet
[09:40:26] <wikibugs>	 (03CR) 10Jcrespo: [C:03+2] dbbackups: Decommission db2139 [puppet] - 10https://gerrit.wikimedia.org/r/1115774 (https://phabricator.wikimedia.org/T383971) (owner: 10Jcrespo)
[09:40:57] <wikibugs>	 (03PS2) 10JMeybohm: Add interative.ask_yesno [software/pywmflib] - 10https://gerrit.wikimedia.org/r/1115767
[09:41:45] <logmsgbot>	 !log marostegui@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1163.eqiad.wmnet with reason: Maintenance
[09:41:52] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depooling db1163 (T384592)', diff saved to https://phabricator.wikimedia.org/P72897 and previous config saved to /var/cache/conftool/dbconfig/20250131-094152-marostegui.json
[09:41:57] <stashbot>	 T384592: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592
[09:43:03] <wikibugs>	 (03CR) 10Jelto: [C:03+1] "lgtm!" [puppet] - 10https://gerrit.wikimedia.org/r/1115778 (https://phabricator.wikimedia.org/T341984) (owner: 10JMeybohm)
[09:47:54] <wikibugs>	 (03CR) 10Gkyziridis: [C:03+1] "LGTM!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115359 (https://phabricator.wikimedia.org/T382295) (owner: 10Kevin Bazira)
[09:49:14] <wikibugs>	 (03PS1) 10Urbanecm: [Growth] Increase minimum tasks per topic to 2000 for eswiki, frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115791 (https://phabricator.wikimedia.org/T378527)
[09:49:18] <wikibugs>	 (03CR) 10JMeybohm: [C:03+2] kubernetes::master: Update kubectl alternative entry [puppet] - 10https://gerrit.wikimedia.org/r/1115778 (https://phabricator.wikimedia.org/T341984) (owner: 10JMeybohm)
[09:49:22] <wikibugs>	 (03CR) 10JMeybohm: [C:03+2] k8s::client: Allow for install of all kubectl versions [puppet] - 10https://gerrit.wikimedia.org/r/1114970 (https://phabricator.wikimedia.org/T341984) (owner: 10JMeybohm)
[09:51:00] <wikibugs>	 (03PS2) 10Urbanecm: [Growth] Increase minimum tasks per topic to 2000 for eswiki, frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115791 (https://phabricator.wikimedia.org/T378527)
[09:51:58] <wikibugs>	 (03CR) 10Urbanecm: [C:03+1] "this has happened. let's deploy this on Monday." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1113984 (https://phabricator.wikimedia.org/T383714) (owner: 10Cyndywikime)
[09:53:32] <wikibugs>	 (03CR) 10JMeybohm: [C:03+2] kubernetes-publish-sa-cert: Don't fail when no certs in etcd [puppet] - 10https://gerrit.wikimedia.org/r/1115433 (https://phabricator.wikimedia.org/T341984) (owner: 10JMeybohm)
[09:54:08] <wikibugs>	 (03CR) 10Urbanecm: [C:03+1] beta: increase growth tasks lookahead size (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1113997 (https://phabricator.wikimedia.org/T325990) (owner: 10Sergio Gimeno)
[09:54:10] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, February 03 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1113997 (https://phabricator.wikimedia.org/T325990) (owner: 10Sergio Gimeno)
[09:54:24] <wikibugs>	 (03CR) 10Urbanecm: beta: increase growth tasks lookahead size [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1113997 (https://phabricator.wikimedia.org/T325990) (owner: 10Sergio Gimeno)
[10:03:56] <logmsgbot>	 !log jynus@cumin1002 START - Cookbook sre.dns.netbox
[10:06:50] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depool db1246', diff saved to https://phabricator.wikimedia.org/P72898 and previous config saved to /var/cache/conftool/dbconfig/20250131-100650-marostegui.json
[10:06:59] <logmsgbot>	 !log root@cumin1002 START - Cookbook sre.mysql.upgrade for db1246.eqiad.wmnet
[10:08:06] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depool db2238', diff saved to https://phabricator.wikimedia.org/P72899 and previous config saved to /var/cache/conftool/dbconfig/20250131-100806-marostegui.json
[10:08:16] <logmsgbot>	 !log root@cumin1002 START - Cookbook sre.mysql.upgrade for db2238.codfw.wmnet
[10:09:25] <wikibugs>	 (03PS1) 10JMeybohm: Revert "kubernetes::master: Update kubectl alternative entry" [puppet] - 10https://gerrit.wikimedia.org/r/1115798
[10:09:27] <wikibugs>	 (03PS1) 10JMeybohm: Revert "k8s::client: Allow for install of all kubectl versions" [puppet] - 10https://gerrit.wikimedia.org/r/1115799
[10:09:27] <wikibugs>	 (03CR) 10AOkoth: [C:03+1] wikidata-query-gui: bump query-gui image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115786 (https://phabricator.wikimedia.org/T350793) (owner: 10Jelto)
[10:09:56] <wikibugs>	 (03Abandoned) 10JMeybohm: Revert "kubernetes::master: Update kubectl alternative entry" [puppet] - 10https://gerrit.wikimedia.org/r/1115798 (owner: 10JMeybohm)
[10:11:18] <wikibugs>	 (03CR) 10JMeybohm: "Reverting for now as it makes puppet fail on deploy hosts." [puppet] - 10https://gerrit.wikimedia.org/r/1115799 (owner: 10JMeybohm)
[10:11:30] <wikibugs>	 06SRE, 10MW-on-K8s, 06serviceops: mwscript-k8s does not support short maintenance script names - https://phabricator.wikimedia.org/T385238#10511338 (10Lucas_Werkmeister_WMDE) →14Duplicate dup:03T380533
[10:11:53] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Revert "k8s::client: Allow for install of all kubectl versions" [puppet] - 10https://gerrit.wikimedia.org/r/1115799 (owner: 10JMeybohm)
[10:12:22] <wikibugs>	 (03PS2) 10JMeybohm: Revert "k8s::client: Allow for install of all kubectl versions" [puppet] - 10https://gerrit.wikimedia.org/r/1115799
[10:12:40] <logmsgbot>	 !log root@cumin1002 END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1246.eqiad.wmnet
[10:12:43] <wikibugs>	 (03PS1) 10Slyngshede: C:apereo_cas Specify encryption algorithms for CAS 7.1 [puppet] - 10https://gerrit.wikimedia.org/r/1115801 (https://phabricator.wikimedia.org/T372892)
[10:13:12] <wikibugs>	 (03PS3) 10JMeybohm: Revert "k8s::client: Allow for install of all kubectl versions" [puppet] - 10https://gerrit.wikimedia.org/r/1115799 (https://phabricator.wikimedia.org/T341984)
[10:13:23] <logmsgbot>	 !log root@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1246.eqiad.wmnet with reason: Index rebuild
[10:13:45] <logmsgbot>	 !log root@cumin1002 END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2238.codfw.wmnet
[10:14:07] <logmsgbot>	 !log root@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2238.codfw.wmnet with reason: Index rebuild
[10:15:28] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Revert "k8s::client: Allow for install of all kubectl versions" [puppet] - 10https://gerrit.wikimedia.org/r/1115799 (https://phabricator.wikimedia.org/T341984) (owner: 10JMeybohm)
[10:15:56] <logmsgbot>	 !log jynus@cumin1002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2139.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002"
[10:16:19] <logmsgbot>	 !log jynus@cumin1002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2139.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1002"
[10:16:19] <logmsgbot>	 !log jynus@cumin1002 END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
[10:16:20] <logmsgbot>	 !log jynus@cumin1002 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2139.codfw.wmnet
[10:17:13] <wikibugs>	 (03PS4) 10JMeybohm: Revert "k8s::client: Allow for install of all kubectl versions" [puppet] - 10https://gerrit.wikimedia.org/r/1115799 (https://phabricator.wikimedia.org/T341984)
[10:19:38] <wikibugs>	 (03CR) 10Slyngshede: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/4913/co" [puppet] - 10https://gerrit.wikimedia.org/r/1115801 (https://phabricator.wikimedia.org/T372892) (owner: 10Slyngshede)
[10:19:53] <wikibugs>	 10ops-codfw, 10Data-Persistence-Backup, 10database-backups, 06DBA, 06DC-Ops: decommission db2139 - https://phabricator.wikimedia.org/T383971#10511368 (10jcrespo) a:05jcrespo→03None
[10:20:06] <wikibugs>	 10ops-codfw, 10Data-Persistence-Backup, 10database-backups, 06DBA, 06DC-Ops: decommission db2139 - https://phabricator.wikimedia.org/T383971#10511372 (10jcrespo)
[10:20:19] <wikibugs>	 10ops-codfw, 10Data-Persistence-Backup, 10database-backups, 06DBA, 06DC-Ops: decommission db2139 - https://phabricator.wikimedia.org/T383971#10511373 (10jcrespo) 05In progress→03Open
[10:20:27] <wikibugs>	 10ops-codfw, 10Data-Persistence-Backup, 10database-backups, 06DBA, 06DC-Ops: decommission db2139 - https://phabricator.wikimedia.org/T383971#10511375 (10jcrespo) p:05Medium→03Triage
[10:20:48] <wikibugs>	 (03CR) 10Clément Goubert: [C:03+1] envoy: adjust the noc config to use both SNI and a custom HTTP host header [puppet] - 10https://gerrit.wikimedia.org/r/1115776 (https://phabricator.wikimedia.org/T384329) (owner: 10Brouberol)
[10:21:39] <wikibugs>	 (03CR) 10Brouberol: [C:03+2] envoy: adjust the noc config to use both SNI and a custom HTTP host header [puppet] - 10https://gerrit.wikimedia.org/r/1115776 (https://phabricator.wikimedia.org/T384329) (owner: 10Brouberol)
[10:24:36] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: PDU sensor over limit - https://phabricator.wikimedia.org/T383383#10511384 (10phaultfinder)
[10:25:36] <wikibugs>	 (03CR) 10Slyngshede: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/4914/co" [puppet] - 10https://gerrit.wikimedia.org/r/1115801 (https://phabricator.wikimedia.org/T372892) (owner: 10Slyngshede)
[10:25:37] <wikibugs>	 (03CR) 10JMeybohm: [C:03+2] Revert "k8s::client: Allow for install of all kubectl versions" [puppet] - 10https://gerrit.wikimedia.org/r/1115799 (https://phabricator.wikimedia.org/T341984) (owner: 10JMeybohm)
[10:26:06] <wikibugs>	 (03PS5) 10JMeybohm: Revert "k8s::client: Allow for install of all kubectl versions" [puppet] - 10https://gerrit.wikimedia.org/r/1115799 (https://phabricator.wikimedia.org/T341984)
[10:28:23] <wikibugs>	 (03CR) 10Federico Ceratto: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1115654 (owner: 10Marostegui)
[10:28:24] <wikibugs>	 (03CR) 10Jelto: [C:03+1] Revert "k8s::client: Allow for install of all kubectl versions" [puppet] - 10https://gerrit.wikimedia.org/r/1115799 (https://phabricator.wikimedia.org/T341984) (owner: 10JMeybohm)
[10:30:44] <wikibugs>	 (03CR) 10JMeybohm: [C:03+2] Revert "k8s::client: Allow for install of all kubectl versions" [puppet] - 10https://gerrit.wikimedia.org/r/1115799 (https://phabricator.wikimedia.org/T341984) (owner: 10JMeybohm)
[10:31:37] <wikibugs>	 (03CR) 10Slyngshede: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/4915/co" [puppet] - 10https://gerrit.wikimedia.org/r/1115801 (https://phabricator.wikimedia.org/T372892) (owner: 10Slyngshede)
[10:32:07] <wikibugs>	 (03PS1) 10JMeybohm: Revert^2 "k8s::client: Allow for install of all kubectl versions" [puppet] - 10https://gerrit.wikimedia.org/r/1115803
[10:33:37] <wikibugs>	 06SRE, 10Deployments, 10MW-on-K8s, 06serviceops: mwscript-k8s gives Invalid value: "<script>": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character - https://phabricator.wikimedia.org/T380533#10511408 (10Urbanecm_...
[10:36:14] <wikibugs>	 (03PS2) 10Slyngshede: C:apereo_cas Specify encryption algorithms for CAS 7.1 [puppet] - 10https://gerrit.wikimedia.org/r/1115801 (https://phabricator.wikimedia.org/T372892)
[10:37:02] <wikibugs>	 (03CR) 10Slyngshede: [V:03+1] "PCC SUCCESS (NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/4916/console" [puppet] - 10https://gerrit.wikimedia.org/r/1115801 (https://phabricator.wikimedia.org/T372892) (owner: 10Slyngshede)
[10:39:37] <wikibugs>	 (03CR) 10Brouberol: mediwiki-dumps-legacy: Create helmfile deployment of a suspended job (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1114001 (https://phabricator.wikimedia.org/T352650) (owner: 10Btullis)
[10:40:52] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): [C:03+1] trafficserver: switch query-main and query service to wikikube [puppet] - 10https://gerrit.wikimedia.org/r/1115766 (https://phabricator.wikimedia.org/T350793) (owner: 10Jelto)
[10:47:40] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1163 (T384592)', diff saved to https://phabricator.wikimedia.org/P72900 and previous config saved to /var/cache/conftool/dbconfig/20250131-104739-marostegui.json
[10:47:45] <stashbot>	 T384592: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592
[10:49:32] <wikibugs>	 (03CR) 10Kevin Bazira: [C:03+2] "thanks for the review, George :)" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115359 (https://phabricator.wikimedia.org/T382295) (owner: 10Kevin Bazira)
[10:50:38] <wikibugs>	 (03CR) 10Brouberol: "The change looks good. Could we add a fixture files with dumps enabled, to get a meaningful CI diff?" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1104605 (https://phabricator.wikimedia.org/T352650) (owner: 10Giuseppe Lavagetto)
[10:51:07] <wikibugs>	 (03Merged) 10jenkins-bot: ml-services: update article-country staging config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115359 (https://phabricator.wikimedia.org/T382295) (owner: 10Kevin Bazira)
[10:52:02] <wikibugs>	 (03PS1) 10Muehlenhoff: logstash: Grant access to cn=ops [puppet] - 10https://gerrit.wikimedia.org/r/1115808 (https://phabricator.wikimedia.org/T376790)
[10:53:24] <logmsgbot>	 !log akosiaris@deploy2002 helmfile [staging] START helmfile.d/services/mw-api-int: apply
[10:53:33] <logmsgbot>	 !log akosiaris@deploy2002 helmfile [staging] DONE helmfile.d/services/mw-api-int: apply
[11:02:04] <logmsgbot>	 !log kevinbazira@deploy2002 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' .
[11:02:19] <wikibugs>	 (03CR) 10Cyndywikime: [C:03+1] [Growth] Increase minimum tasks per topic to 2000 for eswiki, frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115791 (https://phabricator.wikimedia.org/T378527) (owner: 10Urbanecm)
[11:02:46] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P72901 and previous config saved to /var/cache/conftool/dbconfig/20250131-110246-marostegui.json
[11:06:59] <wikibugs>	 (03PS9) 10Brouberol: mediwiki-dumps-legacy: Create helmfile deployment of a suspended job [deployment-charts] - 10https://gerrit.wikimedia.org/r/1114001 (https://phabricator.wikimedia.org/T352650) (owner: 10Btullis)
[11:08:22] <wikibugs>	 (03PS9) 10Brouberol: mediawiki: Add support for dumps suspended job [deployment-charts] - 10https://gerrit.wikimedia.org/r/1104605 (https://phabricator.wikimedia.org/T352650) (owner: 10Giuseppe Lavagetto)
[11:08:22] <wikibugs>	 (03PS10) 10Brouberol: mediwiki-dumps-legacy: Create helmfile deployment of a suspended job [deployment-charts] - 10https://gerrit.wikimedia.org/r/1114001 (https://phabricator.wikimedia.org/T352650) (owner: 10Btullis)
[11:08:52] <logmsgbot>	 !log jayme@cumin1002 END (FAIL) - Cookbook sre.k8s.wipe-cluster (exit_code=99) Wipe the K8s cluster staging-codfw: Kubernetes upgrade
[11:11:16] <wikibugs>	 (03CR) 10Brouberol: [C:03+1] "I've taken the liberty to contribute such a fixture. The change looks good, at least to my un-educated eyes." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1104605 (https://phabricator.wikimedia.org/T352650) (owner: 10Giuseppe Lavagetto)
[11:12:01] <wikibugs>	 (03PS4) 10JMeybohm: k8s.wipe-cluster: Improvements for k8s 1.31 upgrade [cookbooks] - 10https://gerrit.wikimedia.org/r/1115380 (https://phabricator.wikimedia.org/T341984)
[11:15:58] <wikibugs>	 (03PS1) 10Muehlenhoff: Enable logstash-access for production bitu [puppet] - 10https://gerrit.wikimedia.org/r/1115811 (https://phabricator.wikimedia.org/T376790)
[11:17:53] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P72903 and previous config saved to /var/cache/conftool/dbconfig/20250131-111753-marostegui.json
[11:17:58] <wikibugs>	 (03PS5) 10JMeybohm: k8s.wipe-cluster: Improvements for k8s 1.31 upgrade [cookbooks] - 10https://gerrit.wikimedia.org/r/1115380 (https://phabricator.wikimedia.org/T341984)
[11:20:05] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C:03+1] logstash: Grant access to cn=ops [puppet] - 10https://gerrit.wikimedia.org/r/1115808 (https://phabricator.wikimedia.org/T376790) (owner: 10Muehlenhoff)
[11:23:44] <wikibugs>	 (03CR) 10CI reject: [V:04-1] k8s.wipe-cluster: Improvements for k8s 1.31 upgrade [cookbooks] - 10https://gerrit.wikimedia.org/r/1115380 (https://phabricator.wikimedia.org/T341984) (owner: 10JMeybohm)
[11:24:44] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: PDU sensor over limit - https://phabricator.wikimedia.org/T383383#10511489 (10phaultfinder)
[11:26:15] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Set db1223 with weight 0 T385051', diff saved to https://phabricator.wikimedia.org/P72904 and previous config saved to /var/cache/conftool/dbconfig/20250131-112614-root.json
[11:26:20] <stashbot>	 T385051: Switchover s3 master (db1189 -> db1223) - https://phabricator.wikimedia.org/T385051
[11:26:29] <logmsgbot>	 !log marostegui@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 T385051
[11:26:53] <wikibugs>	 (03CR) 10Marostegui: [C:03+2] mariadb: Promote db1223 to s3 master [puppet] - 10https://gerrit.wikimedia.org/r/1115050 (https://phabricator.wikimedia.org/T385051) (owner: 10Gerrit maintenance bot)
[11:27:21] <logmsgbot>	 !log jayme@cumin1002 START - Cookbook sre.k8s.wipe-cluster Wipe the K8s cluster staging-codfw: Kubernetes upgrade
[11:29:04] <marostegui>	 !log Starting s3 eqiad failover from db1189 to db1223 - T385051
[11:29:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:29:21] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Promote db1223 to s3 primary T385051', diff saved to https://phabricator.wikimedia.org/P72905 and previous config saved to /var/cache/conftool/dbconfig/20250131-112920-root.json
[11:30:11] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depool db1189 T385051', diff saved to https://phabricator.wikimedia.org/P72906 and previous config saved to /var/cache/conftool/dbconfig/20250131-113011-marostegui.json
[11:31:20] <logmsgbot>	 !log root@cumin1002 START - Cookbook sre.mysql.upgrade for db1189.eqiad.wmnet
[11:33:00] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1163 (T384592)', diff saved to https://phabricator.wikimedia.org/P72907 and previous config saved to /var/cache/conftool/dbconfig/20250131-113300-marostegui.json
[11:33:01] <logmsgbot>	 !log marostegui@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: maintenance
[11:33:06] <stashbot>	 T384592: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592
[11:33:15] <logmsgbot>	 !log marostegui@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1169.eqiad.wmnet with reason: Maintenance
[11:33:22] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depooling db1169 (T384592)', diff saved to https://phabricator.wikimedia.org/P72908 and previous config saved to /var/cache/conftool/dbconfig/20250131-113321-marostegui.json
[11:33:43] <marostegui>	 !log Upgrade mariadb in dbstore1008 and rebuild tables on s1, s5 and s7 T384818
[11:33:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:33:47] <stashbot>	 T384818: Upgrade dbstore* hosts to 10.6.20 and rebuild tables - https://phabricator.wikimedia.org/T384818
[11:36:10] <logmsgbot>	 !log marostegui@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on dbstore1008.eqiad.wmnet with reason: maintenance
[11:37:32] <logmsgbot>	 !log jayme@cumin1002 END (FAIL) - Cookbook sre.k8s.wipe-cluster (exit_code=99) Wipe the K8s cluster staging-codfw: Kubernetes upgrade
[11:38:03] <logmsgbot>	 !log root@cumin1002 END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1189.eqiad.wmnet
[11:38:08] <icinga-wm>	 PROBLEM - Host db1189 #page is DOWN: PING CRITICAL - Packet loss = 100%
[11:38:09] <icinga-wm>	 RECOVERY - Host db1189 #page is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms
[11:38:18] <marostegui>	 downtime failed I guess....
[11:38:23] <claime>	 x)
[11:38:31] <jynus>	 why did it last 1 second?
[11:38:43] <marostegui>	 Because it rebooted for an upgrade
[11:39:07] <marostegui>	 Created silence ID 130a94c6-ac58-4af1-94d4-5d8a88f2ff94
[11:39:13] <marostegui>	 I guess it didn't go through
[11:39:13] <wikibugs>	 (03PS6) 10JMeybohm: k8s.wipe-cluster: Improvements for k8s 1.31 upgrade [cookbooks] - 10https://gerrit.wikimedia.org/r/1115380 (https://phabricator.wikimedia.org/T341984)
[11:39:31] <marostegui>	 Deleted silence ID 130a94c6-ac58-4af1-94d4-5d8a88f2ff94
[11:40:12] <logmsgbot>	 !log root@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1189.eqiad.wmnet with reason: Index rebuild
[11:46:41] <wikibugs>	 (03CR) 10CI reject: [V:04-1] k8s.wipe-cluster: Improvements for k8s 1.31 upgrade [cookbooks] - 10https://gerrit.wikimedia.org/r/1115380 (https://phabricator.wikimedia.org/T341984) (owner: 10JMeybohm)
[11:47:03] <wikibugs>	 (03CR) 10Marostegui: [C:03+2] check_private_data_report: Add Federico's email. [puppet] - 10https://gerrit.wikimedia.org/r/1115654 (owner: 10Marostegui)
[11:53:54] <wikibugs>	 (03PS7) 10JMeybohm: k8s.wipe-cluster: Improvements for k8s 1.31 upgrade [cookbooks] - 10https://gerrit.wikimedia.org/r/1115380 (https://phabricator.wikimedia.org/T341984)
[11:59:06] <logmsgbot>	 !log jayme@cumin1002 START - Cookbook sre.k8s.wipe-cluster Wipe the K8s cluster staging-codfw: Kubernetes upgrade
[12:00:05] <jouncebot>	 Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250131T0800)
[12:00:05] <jouncebot>	 jelto, arnoldokoth, and mutante: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for GitLab version upgrades deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20250131T1200).
[12:02:43] <Lucas_WMDE>	 I’d like to test a patch on mwdebug (bare-metal, probably) – any objections?
[12:02:48] <Lucas_WMDE>	 (cc tgr|away ^^)
[12:03:54] <logmsgbot>	 !log jayme@cumin1002 END (FAIL) - Cookbook sre.k8s.wipe-cluster (exit_code=99) Wipe the K8s cluster staging-codfw: Kubernetes upgrade
[12:04:46] <logmsgbot>	 !log jayme@cumin1002 START - Cookbook sre.k8s.wipe-cluster Wipe the K8s cluster staging-codfw: Kubernetes upgrade
[12:07:50] * Lucas_WMDE testing on mwdebug2002
[12:08:17] <wikibugs>	 06SRE, 06Infrastructure-Foundations, 10netops, 10observability, and 3 others: Prevent BGP alerts triggering when K8s host maintenance is being done - https://phabricator.wikimedia.org/T384731#10511648 (10cmooney) >>! In T384731#10511163, @fgiunchedi wrote: > Yes we would be able to [[ https://prometheus.io...
[12:08:23] <logmsgbot>	 !log jayme@cumin1002 END (FAIL) - Cookbook sre.k8s.wipe-cluster (exit_code=99) Wipe the K8s cluster staging-codfw: Kubernetes upgrade
[12:08:42] <wikibugs>	 (03PS8) 10JMeybohm: k8s.wipe-cluster: Improvements for k8s 1.31 upgrade [cookbooks] - 10https://gerrit.wikimedia.org/r/1115380 (https://phabricator.wikimedia.org/T341984)
[12:10:08] <logmsgbot>	 !log jayme@cumin1002 START - Cookbook sre.k8s.wipe-cluster Wipe the K8s cluster staging-codfw: Kubernetes upgrade
[12:11:52] * Lucas_WMDE done testing https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/1115832/2
[12:12:01] <Lucas_WMDE>	 wait, one sec
[12:12:11] <Lucas_WMDE>	 I should’ve done the scap pull to reset mwdebug2002 before saying I’m done
[12:12:13] <Lucas_WMDE>	 *now* I’m done :)
[12:13:21] <Lucas_WMDE>	 thcipriani, jeena: I would like to emergency deploy a backport https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/1115832/2 to wmf.14 to fix a client-side regression that seems to be annoying quite a few users (https://phabricator.wikimedia.org/T385261)
[12:13:25] <Lucas_WMDE>	 (cc tgr|away)
[12:14:49] <tgr|away>	 thanks for handling that Lucas_WMDE
[12:15:54] * Lucas_WMDE waits a few minutes for objections and will otherwise proceed soon
[12:16:01] <moritzm>	 !log rebalance codfw/D following OS updates T382508
[12:16:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:16:06] <stashbot>	 T382508: Update remaining Ganeti servers in codfw to Bookworm - https://phabricator.wikimedia.org/T382508
[12:16:21] <wikibugs>	 (03PS1) 10Lucas Werkmeister (WMDE): ForeignApi: Do not share AJAX options between requests [extensions/CentralAuth] (wmf/1.44.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1115833 (https://phabricator.wikimedia.org/T385261)
[12:16:29] <Lucas_WMDE>	 tgr|away: thanks for identifying and fixing the issue!
[12:20:17] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by lucaswerkmeister-wmde@deploy2002 using scap backport" [extensions/CentralAuth] (wmf/1.44.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1115833 (https://phabricator.wikimedia.org/T385261) (owner: 10Lucas Werkmeister (WMDE))
[12:20:28] <Lucas_WMDE>	 the master change made it through gate-and-submit, so I’ve +2ed the backport
[12:20:44] <Lucas_WMDE>	 that still leaves a few minutes for someone to object during the backport gate-and-submit build ^^
[12:20:56] <Amir1>	 go ahead
[12:24:12] <Lucas_WMDE>	 huh, the #2 error in logspam-watch looks like PHP 8.1 stuff
[12:24:21] <Lucas_WMDE>	 I wouldn’t have expected that yet at the current rollout stage ^^
[12:24:40] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: PDU sensor over limit - https://phabricator.wikimedia.org/T383383#10511744 (10phaultfinder)
[12:26:09] <logmsgbot>	 !log jayme@cumin1002 END (FAIL) - Cookbook sre.k8s.wipe-cluster (exit_code=99) Wipe the K8s cluster staging-codfw: Kubernetes upgrade
[12:26:22] <Lucas_WMDE>	 (T384738)
[12:26:22] <stashbot>	 T384738: TypeError: preg_match() expects parameter 2 to be string, null given - https://phabricator.wikimedia.org/T384738
[12:26:52] <Reedy>	 just bad timing :P
[12:27:13] <Reedy>	 If it wasn't a friday, I'd probably do a few more backports
[12:27:18] <Lucas_WMDE>	 I’m confused, which PHP version throws a TypeError for preg_match arg#2 null? https://3v4l.org/uvEhk#veol
[12:27:22] <Lucas_WMDE>	 anyway. not related to what I’m currently doing ^^
[12:28:24] <wikibugs>	 (03Merged) 10jenkins-bot: ForeignApi: Do not share AJAX options between requests [extensions/CentralAuth] (wmf/1.44.0-wmf.14) - 10https://gerrit.wikimedia.org/r/1115833 (https://phabricator.wikimedia.org/T385261) (owner: 10Lucas Werkmeister (WMDE))
[12:28:40] <logmsgbot>	 !log lucaswerkmeister-wmde@deploy2002 Started scap sync-world: Backport for [[gerrit:1115833|ForeignApi: Do not share AJAX options between requests (T385261)]]
[12:28:45] <stashbot>	 T385261: Cannot attach Wikipedia article to Wikidata item from the Wikipedia - https://phabricator.wikimedia.org/T385261
[12:29:17] <Lucas_WMDE>	 Reedy: that task isn’t even a PHP 8.1 error, it turns out (logstash says it was on PHP 7.4.33)
[12:29:27] <Lucas_WMDE>	 I have no idea how the error could even happen but I’ll let the other people on that task figure it out
[12:31:35] <logmsgbot>	 !log lucaswerkmeister-wmde@deploy2002 lucaswerkmeister-wmde: Backport for [[gerrit:1115833|ForeignApi: Do not share AJAX options between requests (T385261)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
[12:31:41] <Lucas_WMDE>	 testing…
[12:32:10] <Lucas_WMDE>	 works!
[12:32:11] <logmsgbot>	 !log lucaswerkmeister-wmde@deploy2002 lucaswerkmeister-wmde: Continuing with sync
[12:33:01] <wikibugs>	 (03CR) 10Urbanecm: beta: increase growth tasks lookahead size (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1113997 (https://phabricator.wikimedia.org/T325990) (owner: 10Sergio Gimeno)
[12:37:52] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1169 (T384592)', diff saved to https://phabricator.wikimedia.org/P72910 and previous config saved to /var/cache/conftool/dbconfig/20250131-123751-marostegui.json
[12:37:57] <stashbot>	 T384592: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592
[12:38:30] <logmsgbot>	 !log lucaswerkmeister-wmde@deploy2002 Finished scap sync-world: Backport for [[gerrit:1115833|ForeignApi: Do not share AJAX options between requests (T385261)]] (duration: 09m 49s)
[12:38:34] <stashbot>	 T385261: Cannot attach Wikipedia article to Wikidata item from the Wikipedia - https://phabricator.wikimedia.org/T385261
[12:39:49] <urbanecm>	 Lucas_WMDE: ok to sneak in a beta-only change? :)
[12:40:00] <urbanecm>	 (https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1113997, if you want to do something else too)
[12:40:01] <Lucas_WMDE>	 as far as I’m concerned, sure
[12:40:03] * Lucas_WMDE done deploying
[12:40:22] <urbanecm>	 ack
[12:40:24] <wikibugs>	 (03CR) 10Urbanecm: [C:03+2] "beta-only, no-op for prod" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1113997 (https://phabricator.wikimedia.org/T325990) (owner: 10Sergio Gimeno)
[12:40:26] <Lucas_WMDE>	 yeah I think if it only touches -labs.php that should be safe enough IMHO
[12:40:33] <urbanecm>	 yep yep
[12:42:53] <wikibugs>	 (03PS3) 10Urbanecm: beta: increase growth tasks lookahead size [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1113997 (https://phabricator.wikimedia.org/T325990) (owner: 10Sergio Gimeno)
[12:42:57] <wikibugs>	 (03CR) 10Urbanecm: beta: increase growth tasks lookahead size [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1113997 (https://phabricator.wikimedia.org/T325990) (owner: 10Sergio Gimeno)
[12:42:59] <wikibugs>	 (03CR) 10Urbanecm: [C:03+2] beta: increase growth tasks lookahead size [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1113997 (https://phabricator.wikimedia.org/T325990) (owner: 10Sergio Gimeno)
[12:44:02] <wikibugs>	 (03Merged) 10jenkins-bot: beta: increase growth tasks lookahead size [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1113997 (https://phabricator.wikimedia.org/T325990) (owner: 10Sergio Gimeno)
[12:44:57] * urbanecm done
[12:46:37] <wikibugs>	 (03CR) 10Cathal Mooney: [C:03+2] Network: add qos and sflow config for configure-switch-interfaces [cookbooks] - 10https://gerrit.wikimedia.org/r/1115109 (https://phabricator.wikimedia.org/T379549) (owner: 10Cathal Mooney)
[12:46:58] <wikibugs>	 (03CR) 10Cathal Mooney: [C:03+2] Class-of-service: don't insert comment with host name under cos/ints [homer/public] - 10https://gerrit.wikimedia.org/r/1115134 (https://phabricator.wikimedia.org/T379549) (owner: 10Cathal Mooney)
[12:52:07] <jinxer-wm>	 FIRING: CirrusSearchHighOldGCFrequency: Elasticsearch instance elastic1096-production-search-omega-eqiad is running the old gc excessively - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Stuck_in_old_GC_hell - https://grafana.wikimedia.org/d/000000462/elasticsearch-memory - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchHighOldGCFrequency
[12:52:41] <wikibugs>	 (03Merged) 10jenkins-bot: Network: add qos and sflow config for configure-switch-interfaces [cookbooks] - 10https://gerrit.wikimedia.org/r/1115109 (https://phabricator.wikimedia.org/T379549) (owner: 10Cathal Mooney)
[12:52:42] <wikibugs>	 (03Merged) 10jenkins-bot: Class-of-service: don't insert comment with host name under cos/ints [homer/public] - 10https://gerrit.wikimedia.org/r/1115134 (https://phabricator.wikimedia.org/T379549) (owner: 10Cathal Mooney)
[12:52:59] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P72911 and previous config saved to /var/cache/conftool/dbconfig/20250131-125258-marostegui.json
[12:54:40] <wikibugs>	 (03PS1) 10Phuedx: tests: Assert event stream configs have valid samples [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115839
[12:55:21] <icinga-wm>	 PROBLEM - Disk space on grafana2001 is CRITICAL: DISK CRITICAL - free space: / 220MiB (1% inode=36%): /tmp 220MiB (1% inode=36%): /var/tmp 220MiB (1% inode=36%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=grafana2001&var-datasource=codfw+prometheus/ops
[12:57:57] <jynus>	 ooops
[12:59:51] <jynus>	 something filled disk at 20:30 yesterday
[13:02:26] <jynus>	 !log ran apt clean on grafana2001
[13:02:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:02:38] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-presto1014 - https://phabricator.wikimedia.org/T382984#10511917 (10Papaul) The disk should be arriving today. I had to give Dell more details before they can agree to send out the disk.
[13:04:18] <jynus>	 ^ claime topranks do you know who owns grafana? I cleaned cache but will need more care, logs and service on only 15 GB is too low
[13:04:49] <wikibugs>	 (03CR) 10JMeybohm: "As we're not able to build helm 3.11 from the 3.17 source tree, updating the binary package name is the way to go here" [debs/helm3] - 10https://gerrit.wikimedia.org/r/1115388 (https://phabricator.wikimedia.org/T341984) (owner: 10Jelto)
[13:04:51] <topranks>	 jynus: observability I would have thought 
[13:04:53] <claime>	 jynus: o11y?
[13:05:05] <urbanecm>	 !log Restart mw-debug.codfw.pinkunicorn (verifying fix for T384919)
[13:05:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:05:10] <stashbot>	 T384919: SUL3 prevents GrowthExperiments from accessing CommunityConfiguration on testwiki - https://phabricator.wikimedia.org/T384919
[13:05:28] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations: Q2:rack/setup/install ganeti105[34].eqiad.wmnet - https://phabricator.wikimedia.org/T381576#10511942 (10Papaul) The servers are failing provision, I will take another look at it later.
[13:06:22] <wikibugs>	 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, 10cloud-services-team (Hardware): Relocate cloudnet1007-dev and cloudnet1008-dev to new racks and rename - https://phabricator.wikimedia.org/T382412#10511956 (10Papaul) a:05VRiley-WMF→03Andrew
[13:07:30] <jynus>	 I think filippo is not around
[13:08:06] <jynus>	 in any case, the heads up was also in case it alerts again
[13:08:07] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P72912 and previous config saved to /var/cache/conftool/dbconfig/20250131-130806-marostegui.json
[13:10:53] <claime>	 huh it's a vm so no LVM
[13:11:22] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db1246 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72913 and previous config saved to /var/cache/conftool/dbconfig/20250131-131121-root.json
[13:11:32] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db2238 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72914 and previous config saved to /var/cache/conftool/dbconfig/20250131-131132-root.json
[13:12:04] <logmsgbot>	 !log jelto@cumin1002 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Test GitLab 17.7 on Replica
[13:12:11] <claime>	 I'll create a task for them to grow the disk, I'm not doing that on a friday
[13:13:17] <jynus>	 yeah, it doesn't look an immediate issue. was just commenting that I hadn't completely fixed the issue
[13:15:21] <icinga-wm>	 RECOVERY - Disk space on grafana2001 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=grafana2001&var-datasource=codfw+prometheus/ops
[13:16:27] <claime>	 https://phabricator.wikimedia.org/T385282 created
[13:16:50] <wikibugs>	 (03PS1) 10Muehlenhoff: Add stub secrets for bookworm replica role [labs/private] - 10https://gerrit.wikimedia.org/r/1115846 (https://phabricator.wikimedia.org/T381565)
[13:19:57] <wikibugs>	 (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Add stub secrets for bookworm replica role [labs/private] - 10https://gerrit.wikimedia.org/r/1115846 (https://phabricator.wikimedia.org/T381565) (owner: 10Muehlenhoff)
[13:20:00] <logmsgbot>	 !log jelto@cumin1002 END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Test GitLab 17.7 on Replica
[13:20:50] <wikibugs>	 (03PS1) 10AOkoth: miscweb: enable egress policy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115848 (https://phabricator.wikimedia.org/T350794)
[13:23:13] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1169 (T384592)', diff saved to https://phabricator.wikimedia.org/P72915 and previous config saved to /var/cache/conftool/dbconfig/20250131-132313-marostegui.json
[13:23:19] <stashbot>	 T384592: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592
[13:23:28] <logmsgbot>	 !log marostegui@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1186.eqiad.wmnet with reason: Maintenance
[13:23:35] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depooling db1186 (T384592)', diff saved to https://phabricator.wikimedia.org/P72916 and previous config saved to /var/cache/conftool/dbconfig/20250131-132335-marostegui.json
[13:23:55] <wikibugs>	 (03PS1) 10Muehlenhoff: Make maps-test2002 a bookworm maps replica [puppet] - 10https://gerrit.wikimedia.org/r/1115850 (https://phabricator.wikimedia.org/T381565)
[13:24:39] <wikibugs>	 (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1115850 (https://phabricator.wikimedia.org/T381565) (owner: 10Muehlenhoff)
[13:26:27] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db1246 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72917 and previous config saved to /var/cache/conftool/dbconfig/20250131-132626-root.json
[13:26:38] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db2238 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72918 and previous config saved to /var/cache/conftool/dbconfig/20250131-132637-root.json
[13:27:21] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[13:27:25] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[13:27:53] <wikibugs>	 (03CR) 10Jelto: "yes that's needed for egress traffic, but see my comment in-line" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115848 (https://phabricator.wikimedia.org/T350794) (owner: 10AOkoth)
[13:28:05] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[13:28:11] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[13:29:07] <wikibugs>	 (03PS1) 10Jcrespo: dbprov: Add temporary configuration to backup to archival [puppet] - 10https://gerrit.wikimedia.org/r/1115853 (https://phabricator.wikimedia.org/T376627)
[13:31:55] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[13:32:04] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[13:32:17] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[13:37:52] <wikibugs>	 (03PS1) 10Brouberol: airflow: mirror datahub configuration from airflow hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115855
[13:38:39] <logmsgbot>	 !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
[13:39:26] <wikibugs>	 (03CR) 10Jcrespo: [C:03+2] dbprov: Add temporary configuration to backup to archival [puppet] - 10https://gerrit.wikimedia.org/r/1115853 (https://phabricator.wikimedia.org/T376627) (owner: 10Jcrespo)
[13:39:57] <logmsgbot>	 !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
[13:40:04] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[13:41:04] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[13:41:32] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db1246 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72919 and previous config saved to /var/cache/conftool/dbconfig/20250131-134132-root.json
[13:41:38] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[13:41:43] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db2238 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72920 and previous config saved to /var/cache/conftool/dbconfig/20250131-134142-root.json
[13:42:29] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[13:43:24] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[13:44:21] <wikibugs>	 (03PS2) 10Brouberol: airflow: mirror datahub configuration from airflow hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115855
[13:47:39] <wikibugs>	 (03PS1) 10Jcrespo: Revert "dbprov: Add temporary configuration to backup to archival" [puppet] - 10https://gerrit.wikimedia.org/r/1115860
[13:50:02] <wikibugs>	 (03CR) 10Jcrespo: [C:03+2] Revert "dbprov: Add temporary configuration to backup to archival" [puppet] - 10https://gerrit.wikimedia.org/r/1115860 (owner: 10Jcrespo)
[13:52:13] <wikibugs>	 (03PS3) 10Brouberol: airflow: mirror datahub configuration from airflow hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115855
[13:53:29] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[13:53:58] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[13:54:34] <wikibugs>	 (03PS4) 10Brouberol: airflow: mirror datahub configuration from airflow hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115855 (https://phabricator.wikimedia.org/T380619)
[13:54:44] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[13:56:05] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[13:56:38] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db1246 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72921 and previous config saved to /var/cache/conftool/dbconfig/20250131-135637-root.json
[13:56:48] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db2238 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72922 and previous config saved to /var/cache/conftool/dbconfig/20250131-135648-root.json
[13:57:06] <wikibugs>	 (03PS1) 10Muehlenhoff: Enable maps-test2003 to maps-test2006 as additional maps bookworm replicas [puppet] - 10https://gerrit.wikimedia.org/r/1115863 (https://phabricator.wikimedia.org/T381565)
[13:58:17] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[14:04:16] <wikibugs>	 (03CR) 10Btullis: [C:03+1] airflow: mirror datahub configuration from airflow hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115855 (https://phabricator.wikimedia.org/T380619) (owner: 10Brouberol)
[14:04:43] <wikibugs>	 (03CR) 10Brouberol: [C:03+2] airflow: mirror datahub configuration from airflow hosts [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115855 (https://phabricator.wikimedia.org/T380619) (owner: 10Brouberol)
[14:04:44] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[14:04:47] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[14:04:53] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[14:04:58] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[14:05:30] <logmsgbot>	 !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
[14:06:01] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[14:06:12] <logmsgbot>	 !log brouberol@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
[14:10:14] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[14:11:43] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db1246 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72923 and previous config saved to /var/cache/conftool/dbconfig/20250131-141143-root.json
[14:11:54] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db2238 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72924 and previous config saved to /var/cache/conftool/dbconfig/20250131-141153-root.json
[14:16:41] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[14:16:45] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[14:17:12] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[14:18:41] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[14:19:28] <wikibugs>	 (03PS1) 10Phuedx: tests: Assert event stream configs have valid samples [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115839
[14:19:29] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[14:19:32] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[14:19:39] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[14:19:52] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[14:20:23] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[14:21:03] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[14:26:46] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[14:27:05] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db1189 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72925 and previous config saved to /var/cache/conftool/dbconfig/20250131-142705-root.json
[14:27:10] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[14:28:41] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'apply'.
[14:29:21] <logmsgbot>	 !log jayme@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
[14:34:29] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1186 (T384592)', diff saved to https://phabricator.wikimedia.org/P72926 and previous config saved to /var/cache/conftool/dbconfig/20250131-143428-marostegui.json
[14:34:34] <stashbot>	 T384592: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592
[14:36:42] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[14:41:04] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depool db1233 db2225 T385084', diff saved to https://phabricator.wikimedia.org/P72927 and previous config saved to /var/cache/conftool/dbconfig/20250131-144103-marostegui.json
[14:41:09] <stashbot>	 T385084: Upgrade and rebuild s2 - https://phabricator.wikimedia.org/T385084
[14:41:24] <logmsgbot>	 !log root@cumin1002 START - Cookbook sre.mysql.upgrade for db1233.eqiad.wmnet
[14:41:48] <logmsgbot>	 !log root@cumin1002 START - Cookbook sre.mysql.upgrade for db2225.codfw.wmnet
[14:42:11] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db1189 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72928 and previous config saved to /var/cache/conftool/dbconfig/20250131-144210-root.json
[14:47:19] <logmsgbot>	 !log root@cumin1002 END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db1233.eqiad.wmnet
[14:47:22] <icinga-wm>	 PROBLEM - Host db1233 #page is DOWN: PING CRITICAL - Packet loss = 100%
[14:47:31] <logmsgbot>	 !log root@cumin1002 END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2225.codfw.wmnet
[14:47:31] <claime>	 !incidents
[14:47:32] <sirenbot>	 5642 (RESOLVED)  Host db1189 (paged) - PING  - Packet loss = 100%
[14:47:33] <marostegui>	 Another silence that didn't go thru apparently
[14:47:34] <icinga-wm>	 RECOVERY - Host db1233 #page is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms
[14:47:49] <claime>	 !incidents
[14:47:49] <sirenbot>	 5643 (UNACKED)  Host db1233 (paged) - PING  - Packet loss = 100%
[14:47:50] <sirenbot>	 5642 (RESOLVED)  Host db1189 (paged) - PING  - Packet loss = 100%
[14:47:55] <claime>	 !resolve 5643
[14:47:56] <sirenbot>	 5643 (RESOLVED)  Host db1233 (paged) - PING  - Packet loss = 100%
[14:47:59] <logmsgbot>	 !log root@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2225.codfw.wmnet with reason: Index rebuild
[14:48:05] <topranks>	 thanks claime 
[14:48:08] <claime>	 np
[14:48:11] <marostegui>	 The cookbook is issuing the silences, and maybe removing them too quickly
[14:48:37] <logmsgbot>	 !log root@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1233.eqiad.wmnet with reason: Index rebuild
[14:49:35] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P72929 and previous config saved to /var/cache/conftool/dbconfig/20250131-144934-marostegui.json
[14:57:16] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db1189 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72930 and previous config saved to /var/cache/conftool/dbconfig/20250131-145715-root.json
[14:58:08] <wikibugs>	 10ops-codfw, 06SRE, 06Data-Persistence, 06DC-Ops: Q2:rack/setup/install db2243 - https://phabricator.wikimedia.org/T382425#10512375 (10Jhancock.wm) a:03Jhancock.wm
[15:01:42] <jinxer-wm>	 RESOLVED: JobUnavailable: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[15:02:59] <wikibugs>	 (03PS1) 10Kamila Součková: benthos: update Benthos version; switch to buster [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1115883 (https://phabricator.wikimedia.org/T385210)
[15:03:06] <wikibugs>	 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, 10cloud-services-team (Hardware): Relocate cloudnet1007-dev and cloudnet1008-dev to new racks and rename - https://phabricator.wikimedia.org/T382412#10512386 (10cmooney) >>! In T382412#10499217, @VRiley-WMF wrote: > The servers were getting the IP address from p...
[15:04:42] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P72931 and previous config saved to /var/cache/conftool/dbconfig/20250131-150441-marostegui.json
[15:07:38] <wikibugs>	 10ops-eqiad, 06SRE, 10Cloud-VPS, 06DC-Ops, 10cloud-services-team (Hardware): Relocate cloudnet1007-dev and cloudnet1008-dev to new racks and rename - https://phabricator.wikimedia.org/T382412#10512402 (10cmooney) @Andrew @aborrero remember we have these static routes on the cloudsw pointing to the cloudg...
[15:11:57] <logmsgbot>	 !log fnegri@cumin1002 conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet,service=s1
[15:11:59] <logmsgbot>	 !log fnegri@cumin1002 conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet,service=s3
[15:12:21] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db1189 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72933 and previous config saved to /var/cache/conftool/dbconfig/20250131-151221-root.json
[15:12:24] <logmsgbot>	 !log jhancock@cumin2002 START - Cookbook sre.dns.netbox
[15:13:01] <logmsgbot>	 !log fnegri@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1013.eqiad.wmnet with reason: Rebooting clouddb1013 T384946
[15:15:25] <logmsgbot>	 !log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
[15:16:31] <logmsgbot>	 !log jhancock@cumin2002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding db2243 to codfw - jhancock@cumin2002"
[15:16:36] <logmsgbot>	 !log jhancock@cumin2002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding db2243 to codfw - jhancock@cumin2002"
[15:16:36] <logmsgbot>	 !log jhancock@cumin2002 END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
[15:18:14] <logmsgbot>	 !log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
[15:18:41] <logmsgbot>	 !log jhancock@cumin2002 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db2243.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
[15:19:03] <logmsgbot>	 !log jhancock@cumin2002 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
[15:19:48] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1186 (T384592)', diff saved to https://phabricator.wikimedia.org/P72934 and previous config saved to /var/cache/conftool/dbconfig/20250131-151947-marostegui.json
[15:19:53] <stashbot>	 T384592: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592
[15:19:58] <logmsgbot>	 !log fnegri@cumin1002 START - Cookbook sre.hosts.reboot-single for host clouddb1013.eqiad.wmnet
[15:20:03] <logmsgbot>	 !log marostegui@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1195.eqiad.wmnet with reason: Maintenance
[15:20:09] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depooling db1195 (T384592)', diff saved to https://phabricator.wikimedia.org/P72935 and previous config saved to /var/cache/conftool/dbconfig/20250131-152008-marostegui.json
[15:20:19] <logmsgbot>	 !log jhancock@cumin2002 START - Cookbook sre.network.configure-switch-interfaces for host db2243
[15:20:34] <logmsgbot>	 !log jhancock@cumin2002 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db2243
[15:21:47] <logmsgbot>	 !log jhancock@cumin2002 START - Cookbook sre.network.configure-switch-interfaces for host puppetserver2004
[15:21:56] <logmsgbot>	 !log jhancock@cumin2002 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host puppetserver2004
[15:23:05] <wikibugs>	 (03CR) 10Hnowlan: benthos: update Benthos version; switch to buster (032 comments) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1115883 (https://phabricator.wikimedia.org/T385210) (owner: 10Kamila Součková)
[15:23:10] <thcipriani>	 Lucas_WMDE: thanks for another Friday fix <3
[15:23:15] <logmsgbot>	 !log fnegri@cumin1002 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddb1013.eqiad.wmnet
[15:23:24] <Lucas_WMDE>	 np :)
[15:23:25] <icinga-wm>	 PROBLEM - MariaDB Replica SQL: s1 on clouddb1013 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
[15:23:25] <icinga-wm>	 PROBLEM - MariaDB Replica IO: s1 on clouddb1013 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
[15:23:25] <icinga-wm>	 PROBLEM - MariaDB Replica SQL: s3 on clouddb1013 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
[15:23:29] <icinga-wm>	 PROBLEM - mysqld processes on clouddb1013 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting
[15:23:33] <icinga-wm>	 PROBLEM - MariaDB Replica IO: s3 on clouddb1013 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
[15:24:01] <icinga-wm>	 PROBLEM - MariaDB read only s1 on clouddb1013 is CRITICAL: Could not connect to localhost:3311 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only
[15:24:01] <icinga-wm>	 PROBLEM - MariaDB read only wikireplica-s1 on clouddb1013 is CRITICAL: Could not connect to localhost:3311 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only
[15:24:29] <icinga-wm>	 RECOVERY - mysqld processes on clouddb1013 is OK: PROCS OK: 2 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting
[15:25:01] <icinga-wm>	 RECOVERY - MariaDB read only s1 on clouddb1013 is OK: Version 10.6.20-MariaDB, Uptime 46s, read_only: True, event_scheduler: False, 4023.26 QPS, connection latency: 0.016402s, query latency: 0.000351s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only
[15:25:01] <icinga-wm>	 RECOVERY - MariaDB read only wikireplica-s1 on clouddb1013 is OK: Version 10.6.20-MariaDB, Uptime 46s, read_only: True, event_scheduler: False, 3997.56 QPS, connection latency: 0.022397s, query latency: 0.000410s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only
[15:25:01] <wikibugs>	 (03PS1) 10Kamila Součková: benthos: udpate chart's modules [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115885 (https://phabricator.wikimedia.org/T385210)
[15:25:25] <icinga-wm>	 RECOVERY - MariaDB Replica IO: s1 on clouddb1013 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
[15:25:25] <icinga-wm>	 RECOVERY - MariaDB Replica SQL: s1 on clouddb1013 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
[15:25:25] <icinga-wm>	 RECOVERY - MariaDB Replica SQL: s3 on clouddb1013 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
[15:25:33] <icinga-wm>	 RECOVERY - MariaDB Replica IO: s3 on clouddb1013 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
[15:27:27] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db1189 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72936 and previous config saved to /var/cache/conftool/dbconfig/20250131-152726-root.json
[15:30:05] <logmsgbot>	 !log fnegri@cumin1002 conftool action : set/pooled=yes; selector: name=clouddb1013.eqiad.wmnet,service=s3
[15:30:16] <logmsgbot>	 !log fnegri@cumin1002 conftool action : set/pooled=yes; selector: name=clouddb1013.eqiad.wmnet,service=s1
[15:31:09] <wikibugs>	 (03PS2) 10Kamila Součková: benthos: update Benthos version; switch to bookworm [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1115883 (https://phabricator.wikimedia.org/T385210)
[15:31:19] <wikibugs>	 (03CR) 10Kamila Součková: benthos: update Benthos version; switch to bookworm (032 comments) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1115883 (https://phabricator.wikimedia.org/T385210) (owner: 10Kamila Součková)
[15:31:39] <wikibugs>	 (03CR) 10Hnowlan: [C:03+1] benthos: update Benthos version; switch to bookworm [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1115883 (https://phabricator.wikimedia.org/T385210) (owner: 10Kamila Součková)
[15:34:23] <wikibugs>	 (03PS1) 10Alexandros Kosiaris: Add group{0,1,2} and pretrain releases in mw-api-int staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115889 (https://phabricator.wikimedia.org/T384944)
[15:36:38] <wikibugs>	 (03CR) 10Kamila Součková: [C:03+2] benthos: update Benthos version; switch to bookworm [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1115883 (https://phabricator.wikimedia.org/T385210) (owner: 10Kamila Součková)
[15:36:51] <wikibugs>	 (03CR) 10Kamila Součková: [V:03+2 C:03+2] benthos: update Benthos version; switch to bookworm [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/1115883 (https://phabricator.wikimedia.org/T385210) (owner: 10Kamila Součková)
[15:38:00] <wikibugs>	 (03PS2) 10Kamila Součková: benthos: update chart's modules [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115885 (https://phabricator.wikimedia.org/T385210)
[15:51:20] <wikibugs>	 (03CR) 10Kamila Součková: "This is it for now, more to come once we have k8s-aux in codfw. Thank you!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115885 (https://phabricator.wikimedia.org/T385210) (owner: 10Kamila Součková)
[16:02:53] <wikibugs>	 (03CR) 10Jdlrobson: tests: Assert event stream configs have valid samples (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115839 (owner: 10Phuedx)
[16:24:27] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1195 (T384592)', diff saved to https://phabricator.wikimedia.org/P72939 and previous config saved to /var/cache/conftool/dbconfig/20250131-162426-marostegui.json
[16:24:32] <stashbot>	 T384592: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592
[16:39:34] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P72940 and previous config saved to /var/cache/conftool/dbconfig/20250131-163933-marostegui.json
[16:39:40] <wikibugs>	 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.9 point update - https://phabricator.wikimedia.org/T383537#10512598 (10MoritzMuehlenhoff)
[16:42:57] <icinga-wm>	 PROBLEM - mailman list info on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring
[16:43:37] <icinga-wm>	 PROBLEM - mailman archives on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring
[16:44:27] <icinga-wm>	 RECOVERY - mailman archives on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 53367 bytes in 0.111 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring
[16:44:47] <icinga-wm>	 RECOVERY - mailman list info on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 8922 bytes in 0.200 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring
[16:47:18] <wikibugs>	 (03CR) 10Hnowlan: [C:03+1] benthos: update chart's modules [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115885 (https://phabricator.wikimedia.org/T385210) (owner: 10Kamila Součková)
[16:52:07] <jinxer-wm>	 FIRING: CirrusSearchHighOldGCFrequency: Elasticsearch instance elastic1096-production-search-omega-eqiad is running the old gc excessively - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Stuck_in_old_GC_hell - https://grafana.wikimedia.org/d/000000462/elasticsearch-memory - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchHighOldGCFrequency
[16:54:41] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1195', diff saved to https://phabricator.wikimedia.org/P72941 and previous config saved to /var/cache/conftool/dbconfig/20250131-165440-marostegui.json
[16:58:47] <wikibugs>	 (03PS3) 10Arturo Borrero Gonzalez: prometheus-node-kernel-messages: add logic to ignore messages [puppet] - 10https://gerrit.wikimedia.org/r/1115391 (https://phabricator.wikimedia.org/T380960)
[17:03:21] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: prometheus-node-kernel-messages: add logic to ignore messages (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1115391 (https://phabricator.wikimedia.org/T380960) (owner: 10Arturo Borrero Gonzalez)
[17:09:48] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1195 (T384592)', diff saved to https://phabricator.wikimedia.org/P72942 and previous config saved to /var/cache/conftool/dbconfig/20250131-170947-marostegui.json
[17:09:53] <stashbot>	 T384592: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592
[17:10:03] <logmsgbot>	 !log marostegui@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1196.eqiad.wmnet with reason: Maintenance
[17:10:21] <logmsgbot>	 !log marostegui@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
[17:10:28] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depooling db1196 (T384592)', diff saved to https://phabricator.wikimedia.org/P72943 and previous config saved to /var/cache/conftool/dbconfig/20250131-171027-marostegui.json
[17:25:52] <logmsgbot>	 !log cmooney@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mr1-magru,mr1-magru IPv6,mr1-magru.oob,mr1-magru.oob IPv6 with reason: emergency reboot
[17:40:07] <wikibugs>	 (03PS1) 10Hnowlan: jobqueue: bump ThumbnailRender concurrency [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115899 (https://phabricator.wikimedia.org/T385273)
[17:47:07] <jinxer-wm>	 RESOLVED: CirrusSearchHighOldGCFrequency: Elasticsearch instance elastic1096-production-search-omega-eqiad is running the old gc excessively - https://wikitech.wikimedia.org/wiki/Search/Elasticsearch_Administration#Stuck_in_old_GC_hell - https://grafana.wikimedia.org/d/000000462/elasticsearch-memory - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchHighOldGCFrequency
[17:52:10] <logmsgbot>	 !log jhathaway@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on sretest1001.eqiad.wmnet with reason: T381919
[17:52:15] <stashbot>	 T381919: Supermicro: unable to set boot order after using Redfish to boot once - https://phabricator.wikimedia.org/T381919
[17:56:55] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db2225 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72945 and previous config saved to /var/cache/conftool/dbconfig/20250131-175654-root.json
[17:57:02] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db1233 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72946 and previous config saved to /var/cache/conftool/dbconfig/20250131-175701-root.json
[17:59:46] <logmsgbot>	 !log jhathaway@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on sretest1002.eqiad.wmnet with reason: T381919
[17:59:51] <stashbot>	 T381919: Supermicro: unable to set boot order after using Redfish to boot once - https://phabricator.wikimedia.org/T381919
[18:12:00] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db2225 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72947 and previous config saved to /var/cache/conftool/dbconfig/20250131-181200-root.json
[18:12:07] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db1233 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72948 and previous config saved to /var/cache/conftool/dbconfig/20250131-181206-root.json
[18:16:13] <icinga-wm>	 PROBLEM - BGP status on cr3-ulsfo is CRITICAL: BGP CRITICAL - AS64605/IPv6: Active - Anycast, AS64605/IPv4: Active - Anycast https://wikitech.wikimedia.org/wiki/Network_monitoring%23BGP_status
[18:16:41] <wikibugs>	 (03PS2) 10AOkoth: miscweb: enable egress policy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115848 (https://phabricator.wikimedia.org/T350794)
[18:20:45] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1196 (T384592)', diff saved to https://phabricator.wikimedia.org/P72949 and previous config saved to /var/cache/conftool/dbconfig/20250131-182044-marostegui.json
[18:20:50] <stashbot>	 T384592: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592
[18:23:20] <wikibugs>	 (03CR) 10Dwisehaupt: [C:03+2] Add aws validation key for lp.email ssl cert generation [dns] - 10https://gerrit.wikimedia.org/r/1115494 (https://phabricator.wikimedia.org/T384931) (owner: 10Dwisehaupt)
[18:23:25] <wikibugs>	 (03PS3) 10AOkoth: miscweb: enable egress policy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115848 (https://phabricator.wikimedia.org/T350794)
[18:23:27] <wikibugs>	 (03PS3) 10Dwisehaupt: Add aws validation key for lp.email ssl cert generation [dns] - 10https://gerrit.wikimedia.org/r/1115494 (https://phabricator.wikimedia.org/T384931)
[18:24:21] <wikibugs>	 (03CR) 10Dwisehaupt: "recheck" [dns] - 10https://gerrit.wikimedia.org/r/1115494 (https://phabricator.wikimedia.org/T384931) (owner: 10Dwisehaupt)
[18:26:34] <logmsgbot>	 !log dwisehaupt@dns1004 START - running authdns-update
[18:27:03] <wikibugs>	 (03PS4) 10AOkoth: miscweb: enable egress policy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115848 (https://phabricator.wikimedia.org/T350794)
[18:27:05] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db2225 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72950 and previous config saved to /var/cache/conftool/dbconfig/20250131-182705-root.json
[18:27:12] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db1233 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72951 and previous config saved to /var/cache/conftool/dbconfig/20250131-182712-root.json
[18:28:30] <logmsgbot>	 !log dwisehaupt@dns1004 END - running authdns-update
[18:29:35] <wikibugs>	 (03CR) 10AOkoth: miscweb: enable egress policy (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115848 (https://phabricator.wikimedia.org/T350794) (owner: 10AOkoth)
[18:35:52] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P72952 and previous config saved to /var/cache/conftool/dbconfig/20250131-183551-marostegui.json
[18:41:57] <icinga-wm>	 PROBLEM - mailman list info on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring
[18:42:11] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db2225 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72953 and previous config saved to /var/cache/conftool/dbconfig/20250131-184210-root.json
[18:42:18] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db1233 (re)pooling @ 75%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72954 and previous config saved to /var/cache/conftool/dbconfig/20250131-184217-root.json
[18:42:37] <icinga-wm>	 PROBLEM - mailman archives on lists1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Mailman/Monitoring
[18:43:27] <icinga-wm>	 RECOVERY - mailman archives on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 53367 bytes in 0.069 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring
[18:43:47] <icinga-wm>	 RECOVERY - mailman list info on lists1004 is OK: HTTP OK: HTTP/1.1 200 OK - 8997 bytes in 0.182 second response time https://wikitech.wikimedia.org/wiki/Mailman/Monitoring
[18:45:45] <icinga-wm>	 PROBLEM - Host ms-be2088 is DOWN: PING CRITICAL - Packet loss = 100%
[18:46:23] <icinga-wm>	 RECOVERY - Host ms-be2088 is UP: PING OK - Packet loss = 0%, RTA = 30.28 ms
[18:50:59] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P72955 and previous config saved to /var/cache/conftool/dbconfig/20250131-185058-marostegui.json
[18:56:36] <wikibugs>	 (03PS1) 10Mimurawil: Remove flag wgSecurePollSingleTransferableVoteEnabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115917 (https://phabricator.wikimedia.org/T376930)
[18:57:16] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db2225 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72956 and previous config saved to /var/cache/conftool/dbconfig/20250131-185715-root.json
[18:57:23] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'db1233 (re)pooling @ 100%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P72957 and previous config saved to /var/cache/conftool/dbconfig/20250131-185722-root.json
[19:06:05] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1196 (T384592)', diff saved to https://phabricator.wikimedia.org/P72958 and previous config saved to /var/cache/conftool/dbconfig/20250131-190605-marostegui.json
[19:06:11] <stashbot>	 T384592: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592
[19:06:21] <logmsgbot>	 !log marostegui@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1206.eqiad.wmnet with reason: Maintenance
[19:06:28] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depooling db1206 (T384592)', diff saved to https://phabricator.wikimedia.org/P72959 and previous config saved to /var/cache/conftool/dbconfig/20250131-190627-marostegui.json
[19:16:28] <jinxer-wm>	 FIRING: CertManagerCertNotReady: Certificate default/jayme-debug is not in a ready state (k8s-staging@codfw) - https://wikitech.wikimedia.org/wiki/Kubernetes/cert-manager - https://grafana.wikimedia.org/d/vo5tiJTnz?var-site=codfw&var-cluster=k8s-staging&var-namespace=default - https://alerts.wikimedia.org/?q=alertname%3DCertManagerCertNotReady
[19:19:07] <wikibugs>	 (03CR) 10AOkoth: "I'll merge this and test on staging... I'll revert if it doesn't work." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115848 (https://phabricator.wikimedia.org/T350794) (owner: 10AOkoth)
[19:19:19] <wikibugs>	 (03PS5) 10AOkoth: miscweb: enable egress policy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115848 (https://phabricator.wikimedia.org/T350794)
[19:22:08] <wikibugs>	 (03CR) 10AOkoth: [C:03+2] miscweb: enable egress policy [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115848 (https://phabricator.wikimedia.org/T350794) (owner: 10AOkoth)
[19:23:42] <logmsgbot>	 !log aokoth@deploy2002 helmfile [staging] START helmfile.d/services/miscweb: apply
[19:23:57] <logmsgbot>	 !log aokoth@deploy2002 helmfile [staging] DONE helmfile.d/services/miscweb: apply
[19:55:16] <wikibugs>	 (03PS1) 10AOkoth: miscweb: update values-os-reports env config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115944 (https://phabricator.wikimedia.org/T350794)
[19:59:34] <logmsgbot>	 !log jhathaway@cumin2002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919
[19:59:39] <stashbot>	 T381919: Supermicro: unable to set boot order after using Redfish to boot once - https://phabricator.wikimedia.org/T381919
[20:03:53] <wikibugs>	 (03PS2) 10AOkoth: miscweb: update values-os-reports env config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115944 (https://phabricator.wikimedia.org/T350794)
[20:43:53] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1206 (T384592)', diff saved to https://phabricator.wikimedia.org/P72960 and previous config saved to /var/cache/conftool/dbconfig/20250131-204353-marostegui.json
[20:43:58] <stashbot>	 T384592: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592
[20:59:00] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P72961 and previous config saved to /var/cache/conftool/dbconfig/20250131-205859-marostegui.json
[21:06:29] <wikibugs>	 10ops-codfw, 06DC-Ops: Recycling with Sprout - https://phabricator.wikimedia.org/T385326 (10Jhancock.wm) 03NEW
[21:14:07] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P72962 and previous config saved to /var/cache/conftool/dbconfig/20250131-211406-marostegui.json
[21:25:41] <wikibugs>	 10ops-codfw, 06DC-Ops: Recycling with Sprout - https://phabricator.wikimedia.org/T385326#10513744 (10Jhancock.wm)
[21:29:13] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1206 (T384592)', diff saved to https://phabricator.wikimedia.org/P72963 and previous config saved to /var/cache/conftool/dbconfig/20250131-212912-marostegui.json
[21:29:18] <stashbot>	 T384592: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592
[21:29:28] <logmsgbot>	 !log marostegui@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1207.eqiad.wmnet with reason: Maintenance
[21:29:35] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depooling db1207 (T384592)', diff saved to https://phabricator.wikimedia.org/P72964 and previous config saved to /var/cache/conftool/dbconfig/20250131-212934-marostegui.json
[21:37:41] <logmsgbot>	 !log mforns@deploy2002 Started deploy [airflow-dags/analytics@16e825c]: update commons impact metrics category allow list
[21:38:42] <logmsgbot>	 !log mforns@deploy2002 Finished deploy [airflow-dags/analytics@16e825c]: update commons impact metrics category allow list (duration: 01m 10s)
[22:10:37] <wikibugs>	 (03CR) 10Amdrel: [C:03+1] "This seems to have all occurrences of the key. Looks good to me." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115917 (https://phabricator.wikimedia.org/T376930) (owner: 10Mimurawil)
[22:43:57] <icinga-wm>	 PROBLEM - Kafka MirrorMaker main-codfw_to_main-eqiad max lag in last 10 minutes on alert1002 is CRITICAL: 1.008e+05 gt 1e+05 https://wikitech.wikimedia.org/wiki/Kafka/Administration https://grafana.wikimedia.org/d/000000521/kafka-mirrormaker?var-datasource=eqiad+prometheus/ops&var-lag_datasource=codfw+prometheus/ops&var-mirror_name=main-codfw_to_main-eqiad
[22:53:10] <wikibugs>	 (03PS1) 10Scott French: Enroll 10% of client sessions in PHP 8.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1115966 (https://phabricator.wikimedia.org/T383845)
[22:53:13] <wikibugs>	 (03PS1) 10Scott French: mw-api-int: serve 1% of traffic on PHP 8.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1115972 (https://phabricator.wikimedia.org/T383845)
[23:00:01] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1207 (T384592)', diff saved to https://phabricator.wikimedia.org/P72966 and previous config saved to /var/cache/conftool/dbconfig/20250131-230000-marostegui.json
[23:00:06] <stashbot>	 T384592: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592
[23:15:08] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P72967 and previous config saved to /var/cache/conftool/dbconfig/20250131-231507-marostegui.json
[23:16:28] <jinxer-wm>	 FIRING: CertManagerCertNotReady: Certificate default/jayme-debug is not in a ready state (k8s-staging@codfw) - https://wikitech.wikimedia.org/wiki/Kubernetes/cert-manager - https://grafana.wikimedia.org/d/vo5tiJTnz?var-site=codfw&var-cluster=k8s-staging&var-namespace=default - https://alerts.wikimedia.org/?q=alertname%3DCertManagerCertNotReady
[23:30:15] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P72968 and previous config saved to /var/cache/conftool/dbconfig/20250131-233014-marostegui.json
[23:45:22] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db1207 (T384592)', diff saved to https://phabricator.wikimedia.org/P72969 and previous config saved to /var/cache/conftool/dbconfig/20250131-234521-marostegui.json
[23:45:27] <stashbot>	 T384592: Add normalization columns to categorylinks table - https://phabricator.wikimedia.org/T384592
[23:45:37] <logmsgbot>	 !log marostegui@cumin1002 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7:00:00 on db1218.eqiad.wmnet with reason: Maintenance
[23:45:44] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depooling db1218 (T384592)', diff saved to https://phabricator.wikimedia.org/P72970 and previous config saved to /var/cache/conftool/dbconfig/20250131-234543-marostegui.json
[23:48:15] <icinga-wm>	 RECOVERY - Host analytics1073 is UP: PING WARNING - Packet loss = 60%, RTA = 0.25 ms
[23:48:45] <icinga-wm>	 PROBLEM - SSH on analytics1073 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring
[23:54:39] <icinga-wm>	 PROBLEM - Host analytics1073 is DOWN: PING CRITICAL - Packet loss = 100%