[14:49:40] PROBLEM - MariaDB sustained replica lag on s1 on db1154 is CRITICAL: 1.792e+04 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1154&var-port=13311 [15:00:48] FIRING: MysqlReplicationLag: MySQL instance db1154:13311 has too large replication lag (4h 8m 56s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1154&var-port=13311 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [15:45:48] RESOLVED: MysqlReplicationLag: MySQL instance db1154:13311 has too large replication lag (3m 38s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1154&var-port=13311 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [15:47:40] RECOVERY - MariaDB sustained replica lag on s1 on db1154 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1154&var-port=13311