[07:20:27] did someone touch backup2004 to fix its alarm or did it fix itself? that was a weird thing to fail... [07:20:40] (systemd-timedated) [07:20:44] nothing done from my end [07:22:41] db1171 is happier today [07:24:24] I will rerun bacula for dbprov1006, though [07:49:44] I need a quick restart [10:08:45] FIRING: MysqlReplicationThreadCountTooLow: MySQL instance db1172:9104 has replication issues. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1172&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationThreadCountTooLow [10:09:18] FIRING: MysqlReplicationLagPtHeartbeat: MySQL instance db1172:9104 has too large replication lag (23h 59m 18s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1172&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [10:20:59] arnaudb: is there a way to pause and increase the downtime schedule of your change? it seems to be running for long and it expiring, causing paging [10:21:55] or the script may need some changes to handle that, like renewing the expiration if it is about to expire [10:43:32] I'll increase the downtime! sorry for the spam :) [10:44:49] I extended 3 days to db1172 [10:45:38] but I guess it may happen to others? [11:10:44] I'll check after lunch [11:23:37] jynus: arnaudb: the db1172 is the revision table alter. I should have set the downtime to three day, I used the copy-paste version from Manuel didn't check if downtime needs adjustments, that was my mistake [11:25:58] Amir1: that doesn't seem to be on the maintenance map, that is why I thought it was arnaud's [11:26:32] no need to ping me, but make sure that is up to date [11:27:20] It actually bit me too before. I don't know why it doesn't add this maint to the map