[00:01:51] !log marostegui@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2222 (T407997)', diff saved to https://phabricator.wikimedia.org/P84810 and previous config saved to /var/cache/conftool/dbconfig/20251105-000151-marostegui.json [00:01:55] T407997: Drop the afl_ip column and the afl_ip_timestamp index from the abuse_filter_log table - https://phabricator.wikimedia.org/T407997 [00:31:21] (03PS1) 10Samwilson: mediawiki tables-catalog: Add watchlist labels tables [puppet] - 10https://gerrit.wikimedia.org/r/1201842 (https://phabricator.wikimedia.org/T406843) [00:38:34] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1201843 [00:38:34] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1201843 (owner: 10TrainBranchBot) [00:45:35] (03PS1) 10Scott French: P:cache::haproxy: ensure x-requestctl is updated [puppet] - 10https://gerrit.wikimedia.org/r/1201844 (https://phabricator.wikimedia.org/T403220) [00:51:33] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1201843 (owner: 10TrainBranchBot) [00:56:08] (03PS1) 10BryanDavis: toolforge: Handle interwiki redirects in front proxy [puppet] - 10https://gerrit.wikimedia.org/r/1201847 (https://phabricator.wikimedia.org/T247432) [00:58:21] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-blazegraph-exporter-wdqs-blazegraph.service on wdqs2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:00:01] (03Abandoned) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1201319 (owner: 10TrainBranchBot) [01:06:27] (03CR) 10BryanDavis: "I tested a cut-and-paste version of this in toolsbeta which seemed to work as hoped." [puppet] - 10https://gerrit.wikimedia.org/r/1201847 (https://phabricator.wikimedia.org/T247432) (owner: 10BryanDavis) [01:08:43] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1201849 [01:08:43] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1201849 (owner: 10TrainBranchBot) [01:09:05] FIRING: [21x] CertAlmostExpired: Certificate for service cr2-eqsin.wikimedia.org:32767 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [01:09:05] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate default-staging-certificate.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Puppet#Renew_agent_certificate - TODO - https://alerts.wikimedia.org/?q=alertname%3DPuppetCertificateAboutToExpire [01:29:48] (03PS1) 10Tim Starling: admin: Add FIDO key for tstarling [puppet] - 10https://gerrit.wikimedia.org/r/1201850 [01:33:53] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1201849 (owner: 10TrainBranchBot) [01:40:16] (03CR) 10CDanis: [C:03+1] P:cache::haproxy: ensure x-requestctl is updated [puppet] - 10https://gerrit.wikimedia.org/r/1201844 (https://phabricator.wikimedia.org/T403220) (owner: 10Scott French) [01:43:50] (03PS1) 10CDanis: Add discovery-conftool-state to ignored stale texts [alerts] - 10https://gerrit.wikimedia.org/r/1201852 [01:50:35] (03PS1) 10Tim Starling: recentchanges: Fix watchlistactivity=all, i.e. seen/unseen conflict [core] (wmf/1.46.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1201853 (https://phabricator.wikimedia.org/T408167) [02:09:05] FIRING: [5x] SystemdUnitFailed: docker-reporter-kubernetes-dse_eqiad-images.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:13:21] FIRING: SystemdUnitFailed: user@11984.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:42:33] (03CR) 10TrainBranchBot: [C:03+2] "Approved by tstarling@deploy2002 using scap backport" [core] (wmf/1.46.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1201853 (https://phabricator.wikimedia.org/T408167) (owner: 10Tim Starling) [02:46:34] (03Merged) 10jenkins-bot: recentchanges: Fix watchlistactivity=all, i.e. seen/unseen conflict [core] (wmf/1.46.0-wmf.1) - 10https://gerrit.wikimedia.org/r/1201853 (https://phabricator.wikimedia.org/T408167) (owner: 10Tim Starling) [02:47:32] !log tstarling@deploy2002 Started scap sync-world: Backport for [[gerrit:1201853|recentchanges: Fix watchlistactivity=all, i.e. seen/unseen conflict (T408167)]] [02:47:35] T408167: Selecting both "Unseen changes" and "Seen changes" filters shows nothing at all - https://phabricator.wikimedia.org/T408167 [02:50:02] !log tstarling@deploy2002 tstarling: Backport for [[gerrit:1201853|recentchanges: Fix watchlistactivity=all, i.e. seen/unseen conflict (T408167)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [02:53:54] !log tstarling@deploy2002 tstarling: Continuing with sync [02:58:11] !log tstarling@deploy2002 Finished scap sync-world: Backport for [[gerrit:1201853|recentchanges: Fix watchlistactivity=all, i.e. seen/unseen conflict (T408167)]] (duration: 10m 39s) [02:58:14] T408167: Selecting both "Unseen changes" and "Seen changes" filters shows nothing at all - https://phabricator.wikimedia.org/T408167 [03:04:05] FIRING: [8x] JobUnavailable: Reduced availability for job cloud_dev_pdns_rec in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [04:45:12] 10ops-eqiad, 06SRE, 06DC-Ops: Degraded RAID on an-worker1199 - https://phabricator.wikimedia.org/T409060#11343263 (10VRiley-WMF) Resubmitted SR with TSR attached. [04:53:21] RESOLVED: SystemdUnitFailed: wmf_auto_restart_prometheus-blazegraph-exporter-wdqs-blazegraph.service on wdqs2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:08:21] FIRING: [10x] JobUnavailable: Reduced availability for job cloud_dev_pdns_rec in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [05:09:05] FIRING: [21x] CertAlmostExpired: Certificate for service cr2-eqsin.wikimedia.org:32767 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [05:09:05] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate default-staging-certificate.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Puppet#Renew_agent_certificate - TODO - https://alerts.wikimedia.org/?q=alertname%3DPuppetCertificateAboutToExpire [05:33:21] FIRING: [10x] JobUnavailable: Reduced availability for job cloud_dev_pdns_rec in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [05:43:39] 06SRE, 06collaboration-services, 06Infrastructure-Foundations, 10vrts, and 2 others: VRTS is spammed with bounce e-mails and is going to break - https://phabricator.wikimedia.org/T408632#11343310 (10Krd) My personal opinion is that we should disable notifications completely, but this perhaps isn't consensu... [06:09:05] FIRING: [5x] SystemdUnitFailed: docker-reporter-kubernetes-dse_eqiad-images.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:14:05] FIRING: SystemdUnitFailed: user@11984.service on build2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:17:11] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1158.eqiad.wmnet with reason: Maintenance [06:17:30] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance [06:17:37] !log marostegui@cumin1003 dbctl commit (dc=all): 'Depooling db1158 (T407997)', diff saved to https://phabricator.wikimedia.org/P84814 and previous config saved to /var/cache/conftool/dbconfig/20251105-061737-marostegui.json [06:17:41] T407997: Drop the afl_ip column and the afl_ip_timestamp index from the abuse_filter_log table - https://phabricator.wikimedia.org/T407997 [06:19:50] !log marostegui@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1158 (T407997)', diff saved to https://phabricator.wikimedia.org/P84815 and previous config saved to /var/cache/conftool/dbconfig/20251105-061950-marostegui.json [06:22:31] !log marostegui@cumin1003 dbctl commit (dc=all): 'Set db2191 with weight 0 T409168', diff saved to https://phabricator.wikimedia.org/P84816 and previous config saved to /var/cache/conftool/dbconfig/20251105-062230-marostegui.json [06:22:34] T409168: Switchover x1 master (db2215 -> db2191) - https://phabricator.wikimedia.org/T409168 [06:22:53] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 14 hosts with reason: Primary switchover x1 T409168 [06:23:28] (03CR) 10Marostegui: [C:03+2] mariadb: Promote db2191 to x1 master [puppet] - 10https://gerrit.wikimedia.org/r/1201593 (https://phabricator.wikimedia.org/T409168) (owner: 10Gerrit maintenance bot) [06:27:08] a!log Starting x1 codfw failover from db2215 to db2191 - T409168 [06:27:23] !log marostegui@cumin1003 dbctl commit (dc=all): 'Set x1 codfw as read-only for maintenance - T409168', diff saved to https://phabricator.wikimedia.org/P84817 and previous config saved to /var/cache/conftool/dbconfig/20251105-062723-marostegui.json [06:27:46] !log marostegui@cumin1003 dbctl commit (dc=all): 'Promote db2191 to x1 primary and set section read-write T409168', diff saved to https://phabricator.wikimedia.org/P84818 and previous config saved to /var/cache/conftool/dbconfig/20251105-062745-marostegui.json [06:27:49] T409168: Switchover x1 master (db2215 -> db2191) - https://phabricator.wikimedia.org/T409168 [06:28:15] (03CR) 10Marostegui: [C:03+2] wmnet: Update x1-master alias [dns] - 10https://gerrit.wikimedia.org/r/1201594 (https://phabricator.wikimedia.org/T409168) (owner: 10Gerrit maintenance bot) [06:28:21] !log marostegui@dns1006 START - running authdns-update [06:29:10] !log marostegui@dns1006 END - running authdns-update [06:29:21] !log marostegui@cumin1003 dbctl commit (dc=all): 'Depool db2215 T409168', diff saved to https://phabricator.wikimedia.org/P84819 and previous config saved to /var/cache/conftool/dbconfig/20251105-062920-marostegui.json [06:29:58] 06SRE, 06collaboration-services, 10Znuny: VRTS outbound emails not working - https://phabricator.wikimedia.org/T408967#11343349 (10Krd) All double checked now and appears correct, user reports that it still doesn't work. Please see: https://vrt-wiki.wikimedia.org/w/index.php?diff=134462 [06:30:10] !log marostegui@cumin1003 dbctl commit (dc=all): 'db2215 (re)pooling @ 10%: After switchover', diff saved to https://phabricator.wikimedia.org/P84820 and previous config saved to /var/cache/conftool/dbconfig/20251105-063009-root.json [06:32:03] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 14 hosts with reason: Primary switchover x1 T409168 [06:32:29] !log marostegui@cumin1003 DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 14 hosts with reason: Primary switchover x1 T409168 [06:35:01] !log marostegui@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P84821 and previous config saved to /var/cache/conftool/dbconfig/20251105-063458-marostegui.json [06:38:46] (03PS1) 10Marostegui: db1203: Migration to MariaDB 10.11 [puppet] - 10https://gerrit.wikimedia.org/r/1201868 [06:39:30] (03CR) 10Marostegui: [C:03+2] db1203: Migration to MariaDB 10.11 [puppet] - 10https://gerrit.wikimedia.org/r/1201868 (owner: 10Marostegui) [06:40:24] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1203.eqiad.wmnet with reason: Maintenance [06:40:28] !log marostegui@cumin1003 dbctl commit (dc=all): 'Depool db1203 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P84822 and previous config saved to /var/cache/conftool/dbconfig/20251105-064028-marostegui.json [06:41:31] (03PS1) 10Marostegui: mariadb: Decommission es1034 [puppet] - 10https://gerrit.wikimedia.org/r/1201870 (https://phabricator.wikimedia.org/T409025) [06:42:10] !log marostegui@cumin1003 START - Cookbook sre.hosts.decommission for hosts es1034.eqiad.wmnet [06:42:53] (03CR) 10Marostegui: [C:03+2] mariadb: Decommission es1034 [puppet] - 10https://gerrit.wikimedia.org/r/1201870 (https://phabricator.wikimedia.org/T409025) (owner: 10Marostegui) [06:45:16] !log marostegui@cumin1003 dbctl commit (dc=all): 'db2215 (re)pooling @ 25%: After switchover', diff saved to https://phabricator.wikimedia.org/P84823 and previous config saved to /var/cache/conftool/dbconfig/20251105-064515-root.json [06:48:22] !log marostegui@cumin1003 START - Cookbook sre.dns.netbox [06:48:30] !log marostegui@cumin1003 dbctl commit (dc=all): 'db1203 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P84824 and previous config saved to /var/cache/conftool/dbconfig/20251105-064829-root.json [06:49:18] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for VolkerE - https://phabricator.wikimedia.org/T406243#11343365 (10Volker_E) Hi @Ladsgroup et al, I've re-read the L3 and like to hereby formalize my agreement with it. Please note, that I can't resign the document. {F69903469} >>! In T406243#1127... [06:50:09] !log marostegui@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P84825 and previous config saved to /var/cache/conftool/dbconfig/20251105-065008-marostegui.json [06:51:48] !log marostegui@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1034.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" [06:52:10] !log marostegui@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1034.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" [06:52:10] !log marostegui@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [06:52:10] !log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es1034.eqiad.wmnet [06:52:36] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission es1034.eqiad.wmnet - https://phabricator.wikimedia.org/T409025#11343367 (10Marostegui) a:05Marostegui→03None [06:52:47] 10ops-eqiad, 06DBA, 06DC-Ops, 10decommission-hardware: decommission es1034.eqiad.wmnet - https://phabricator.wikimedia.org/T409025#11343372 (10Marostegui) This is ready for #dc-ops [07:00:04] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20251105T0700) [07:00:22] !log marostegui@cumin1003 dbctl commit (dc=all): 'db2215 (re)pooling @ 50%: After switchover', diff saved to https://phabricator.wikimedia.org/P84826 and previous config saved to /var/cache/conftool/dbconfig/20251105-070021-root.json [07:03:36] !log marostegui@cumin1003 dbctl commit (dc=all): 'db1203 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P84827 and previous config saved to /var/cache/conftool/dbconfig/20251105-070335-root.json [07:05:12] (03PS1) 10Marostegui: db2212: Migration to MariaDB 10.11 [puppet] - 10https://gerrit.wikimedia.org/r/1201898 (https://phabricator.wikimedia.org/T407463) [07:05:17] !log marostegui@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1158 (T407997)', diff saved to https://phabricator.wikimedia.org/P84828 and previous config saved to /var/cache/conftool/dbconfig/20251105-070516-marostegui.json [07:05:20] T407997: Drop the afl_ip column and the afl_ip_timestamp index from the abuse_filter_log table - https://phabricator.wikimedia.org/T407997 [07:05:30] (03PS1) 10Gerrit maintenance bot: mariadb: Promote db2212 to s1 master [puppet] - 10https://gerrit.wikimedia.org/r/1201900 (https://phabricator.wikimedia.org/T409255) [07:05:32] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1170.eqiad.wmnet with reason: Maintenance [07:05:35] (03PS1) 10Gerrit maintenance bot: wmnet: Update s1-master alias [dns] - 10https://gerrit.wikimedia.org/r/1201901 (https://phabricator.wikimedia.org/T409255) [07:05:40] !log marostegui@cumin1003 dbctl commit (dc=all): 'Depooling db1170 (T407997)', diff saved to https://phabricator.wikimedia.org/P84830 and previous config saved to /var/cache/conftool/dbconfig/20251105-070540-marostegui.json [07:06:04] (03CR) 10Marostegui: [C:03+2] db2212: Migration to MariaDB 10.11 [puppet] - 10https://gerrit.wikimedia.org/r/1201898 (https://phabricator.wikimedia.org/T407463) (owner: 10Marostegui) [07:07:03] !log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2212.codfw.wmnet with reason: Maintenance [07:07:08] !log marostegui@cumin1003 dbctl commit (dc=all): 'Depool db2212 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P84831 and previous config saved to /var/cache/conftool/dbconfig/20251105-070707-marostegui.json [07:12:50] (03PS1) 10Marostegui: instances.yaml: Add es1033 to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1201996 (https://phabricator.wikimedia.org/T409257) [07:13:26] (03CR) 10Marostegui: [C:03+2] instances.yaml: Add es1033 to dbctl [puppet] - 10https://gerrit.wikimedia.org/r/1201996 (https://phabricator.wikimedia.org/T409257) (owner: 10Marostegui) [07:15:10] !log marostegui@cumin1003 dbctl commit (dc=all): 'db2212 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P84832 and previous config saved to /var/cache/conftool/dbconfig/20251105-071510-root.json [07:15:28] !log marostegui@cumin1003 dbctl commit (dc=all): 'db2215 (re)pooling @ 75%: After switchover', diff saved to https://phabricator.wikimedia.org/P84833 and previous config saved to /var/cache/conftool/dbconfig/20251105-071527-root.json [07:16:06] !log marostegui@cumin1003 dbctl commit (dc=all): 'Add es1033 to es2 depooled T409257 T407472', diff saved to https://phabricator.wikimedia.org/P84834 and previous config saved to /var/cache/conftool/dbconfig/20251105-071605-marostegui.json [07:16:11] T409257: Move es1033 (es2 Debian Trixie) to es7 - https://phabricator.wikimedia.org/T409257 [07:16:11] T407472: Install a testing db with Debian Trixie - https://phabricator.wikimedia.org/T407472 [07:16:59] (03PS1) 10Marostegui: es1033: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1202002 (https://phabricator.wikimedia.org/T407472) [07:18:42] !log marostegui@cumin1003 dbctl commit (dc=all): 'db1203 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P84835 and previous config saved to /var/cache/conftool/dbconfig/20251105-071841-root.json [07:18:53] (03CR) 10Marostegui: [C:03+2] es1033: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/1202002 (https://phabricator.wikimedia.org/T407472) (owner: 10Marostegui) [07:21:46] !log marostegui@cumin1003 dbctl commit (dc=all): 'es1033 (re)pooling @ 1%: Testing Debian Trixie in es2', diff saved to https://phabricator.wikimedia.org/P84836 and previous config saved to /var/cache/conftool/dbconfig/20251105-072145-root.json [07:23:27] !log marostegui@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1170 (T407997)', diff saved to https://phabricator.wikimedia.org/P84837 and previous config saved to /var/cache/conftool/dbconfig/20251105-072326-marostegui.json [07:23:30] T407997: Drop the afl_ip column and the afl_ip_timestamp index from the abuse_filter_log table - https://phabricator.wikimedia.org/T407997 [07:30:16] !log marostegui@cumin1003 dbctl commit (dc=all): 'db2212 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P84838 and previous config saved to /var/cache/conftool/dbconfig/20251105-073016-root.json [07:30:34] !log marostegui@cumin1003 dbctl commit (dc=all): 'db2215 (re)pooling @ 100%: After switchover', diff saved to https://phabricator.wikimedia.org/P84839 and previous config saved to /var/cache/conftool/dbconfig/20251105-073033-root.json [07:33:48] !log marostegui@cumin1003 dbctl commit (dc=all): 'db1203 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P84840 and previous config saved to /var/cache/conftool/dbconfig/20251105-073347-root.json [07:36:51] !log marostegui@cumin1003 dbctl commit (dc=all): 'es1033 (re)pooling @ 2%: Testing Debian Trixie in es2', diff saved to https://phabricator.wikimedia.org/P84841 and previous config saved to /var/cache/conftool/dbconfig/20251105-073651-root.json [07:38:34] !log marostegui@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P84842 and previous config saved to /var/cache/conftool/dbconfig/20251105-073833-marostegui.json