[03:59:41] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@m5.service on db1217:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:59:56] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@m5.service on db1217:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:24:56] FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@m5.service on db1217:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:27:01] ^-- are these expected? [09:30:30] Taking a look [09:32:55] orchestrator last saw those 9h ago [09:33:08] It is a backup source [09:35:48] It rebooted [10:02:11] Emperor: "Requested system hardreset" shows in the log for "Timestamp = 2026-11-01 22:48:47" [10:06:12] sigh [10:11:30] It seems to have paged, but I don't know who as "more nothing " from rz.l at 0403 [10:13:42] cezmunsta: VO shows no incidents since the 29th [10:19:03] Oh, none have #page in the Icing alert, so I guess that answers that [10:19:17] yeah, I found the alerts in #wikimedia-operations but they're not p.aging [10:19:30] (presumably 'cos backup sources don't warrant waking people up) [10:19:54] Plus these are internal systems [10:20:32] I will create a ticket for DCops to take a look to see if there is anything to update etc [10:24:07] cool, ta. Shall I downtime the alert for a bit? [10:27:24] https://phabricator.wikimedia.org/T443208 - I have silenced that one [10:28:44] Thanks :) [10:29:01] (I fixed the other, which was rclone losing a race with a deletion) [10:29:41] RESOLVED: SystemdUnitFailed: swift_rclone_sync.service on ms-be2069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:29:41] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@m3.service on db1217:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:44:22] * cezmunsta was wondering where the other service alerts were [13:05:01] So…just to confirm (in the wake of another clock change), is the meeting in an hour? [13:05:50] urandom: yes, at 14:00 UTC [13:06:01] google calendar generally gets this right [13:07:04] I expected it to have drifted an hour for daylight confusion time [13:09:25] we're out of that now - Europe moved a weekish ago, and the US this weekend, so we're back in usual-sync [13:09:55] I think last week would have been confused, but you weren't here :) [13:11:07] Oh, right. Does the US move first in Spring? [13:11:51] That seems like the right approach for maximal confusion [13:21:36] yeah, I htink US is 14 March 2027, Europe is 28th [14:39:41] RESOLVED: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db1262:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:23:48] FIRING: [2x] MysqlReplicationLagPtHeartbeat: MySQL instance db1262:9104 has too large replication lag (4d 5h 56m 26s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1262&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [15:24:31] * cezmunsta silences ^ [17:06:12] hi dp!we're planning to repool eqiad a bit earlier than planned (Tuesday 14UTC) please let us know if you have any maintenance ongoing which might impact [18:11:14] jasmine_: not from a db point of view [18:13:18] tx! cc sylngs