[07:44:49] <jinxer-wm>	 FIRING: [2x] MysqlReplicationLagPtHeartbeat: MySQL instance db2189:9104 has too large replication lag (1d 23h 7m 28s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2189&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[07:45:05] <arnaudb>	 its catching up, I'll silence it for a few more hours
[09:49:49] <jinxer-wm>	 FIRING: [21x] MysqlReplicationLagPtHeartbeat: MySQL instance db1190:9104 has too large replication lag (13m 16s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[09:50:03] <jinxer-wm>	 FIRING: [21x] MysqlReplicationLagPtHeartbeat: MySQL instance db1190:9104 has too large replication lag (13m 16s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[09:54:49] <jinxer-wm>	 RESOLVED: [21x] MysqlReplicationLagPtHeartbeat: MySQL instance db1190:9104 has too large replication lag (13m 16s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[14:15:48] <jinxer-wm>	 FIRING: [2x] MysqlReplicationLagPtHeartbeat: MySQL instance db2136:9104 has too large replication lag (11m 1s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[14:20:48] <jinxer-wm>	 FIRING: [21x] MysqlReplicationLagPtHeartbeat: MySQL instance db1190:9104 has too large replication lag (15m 25s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[14:25:48] <jinxer-wm>	 RESOLVED: [21x] MysqlReplicationLagPtHeartbeat: MySQL instance db1190:9104 has too large replication lag (19m 25s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[15:49:22] <urandom>	 marostegui, arnaudb: fyi: on db1160 I installed linux-headers-6.1.0-20-amd64_6.1.85-1_amd64.deb  linux-headers-6.1.0-20-common_6.1.85-1_all.deb (via snapshot, copies in my homedir), and gcc-12 cpp-12 libcc1-0  linux-kbuild-6.1 linux-compiler-gcc-12-x86 (from bookworm, via apt)
[16:48:03] <arnaudb>	 ack, thanks
[16:50:48] <jinxer-wm>	 FIRING: [19x] MysqlReplicationLagPtHeartbeat: MySQL instance db1190:9104 has too large replication lag (11m 4s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[16:55:48] <jinxer-wm>	 FIRING: [21x] MysqlReplicationLagPtHeartbeat: MySQL instance db1190:9104 has too large replication lag (15m 48s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[17:00:48] <jinxer-wm>	 RESOLVED: [21x] MysqlReplicationLagPtHeartbeat: MySQL instance db1190:9104 has too large replication lag (20m 3s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[20:57:48] <jinxer-wm>	 FIRING: [13x] MysqlReplicationLagPtHeartbeat: MySQL instance db1199:9104 has too large replication lag (11m 7s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[21:02:48] <jinxer-wm>	 FIRING: [21x] MysqlReplicationLagPtHeartbeat: MySQL instance db1190:9104 has too large replication lag (15m 43s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[21:07:48] <jinxer-wm>	 RESOLVED: [21x] MysqlReplicationLagPtHeartbeat: MySQL instance db1190:9104 has too large replication lag (17m 43s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat