[03:34:57] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s5.service on db1269:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:38:53] RESOLVED: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@s5.service on db1269:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:53:53] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@x3.service on db1154:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:24:41] Do I need to tag this in any further? https://phabricator.wikimedia.org/T435191 [07:25:15] slyngs: you can add DBA [07:25:34] slyngs: I should be able to have that completed today [07:26:10] Thank you :-) [07:26:35] As long as I get it in the queue correctly I'm happy. [07:27:53] slyngs: that DB is empty [07:28:00] Yes [07:28:28] All done [07:28:41] The fastest DBA on the planet. [07:28:48] haha [07:29:01] DBA-Man :-) [07:29:05] Thank you [07:29:10] XDDD [07:38:50] :) [09:11:55] federico3: will you decomm db-test? [09:12:01] The hosts I mean [09:13:17] I am fixing the topology now because it was looking a bit odd [09:13:54] ok [09:14:03] how odd? [09:14:30] eg: db-test2001 replicates from db-test1003 and it has db2902 under it [09:15:56] federico3: will you decomm db-test? [09:16:10] or you plan on keeping them? [09:17:44] unless we have a specific reason to keep them I'd decom them so we can remove them from puppet etc [09:18:07] +1 [09:20:48] I've left topology fixed and forced db1901 under its correct master and to catch up [09:37:35] oops, db2901 needs chasing, I'm looking at it [09:38:47] federico3: yeah and 2903 too? [09:39:02] there is no 2903 [09:39:15] like we had before, 3 hosts in eq and 2 in cod [09:39:42] maybe it makes sense, so we have the same on both DC [09:39:57] db1176 will be decommissioned "soon" and we can move db2230 to a proper production section [09:40:13] And have 901-903 on both dcs [09:40:43] ah ok, I can add it [09:40:49] thank you! [09:40:53] it will also make the puppet conf symmetrical [09:41:13] yeah, and we can get db2230's hw used properly [09:41:19] in some sX [09:46:26] I've pushed test-s4 to dbctl with for now just the masters: https://phabricator.wikimedia.org/P96249 [09:46:33] I am checking logs for errors [09:46:36] just in case [09:46:37] Amir1: ^ [09:46:51] ack [09:53:53] RESOLVED: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@x3.service on db1154:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:54:38] federico3: did you pick db2230 as the clone source for a reason? orchestrator has RecoverIntermediateMasterClusterFilters set for test-s4 [09:55:45] yeah, maybe we have to disable it too [09:56:05] nothing in logs with db1176 [09:56:10] (mediawiki logs) [09:56:18] Amir1: same, I dont see anything breaking at the moment in logstash [09:56:25] I am about to get in a meeting in 4 mins, but so far so good [10:08:48] federico3: maybe something like this would be better in the script : sudo transfer.py --type=xtrabackup --compress --verbose --stop-slave db2902.codfw.wmnet:/run/mysqld/mysqld.sock db2903.codfw.wmnet:/srv/sqldata [10:09:33] You could look up the socket with "select @@socket" on the source [10:12:15] That would then allow you to remove this line: ssh "$src.$srcdc.wmnet" -t "sudo systemctl stop mariadb.service" [10:19:36] the current version works ok and takes few seconds [10:38:20] db2230 was "2026-08-25 09:41:44 (3364s ago)" [10:40:25] s/was /was last seen/ .. That is more than a few seconds, did something break? [11:03:09] Amir1: https://phabricator.wikimedia.org/T435059#12251005 [11:03:14] is this something you need to fix? [11:31:55] sorry I was in a meeting [11:31:56] looking