[15:16:48] <jinxer-wm>	 FIRING: [8x] MysqlReplicationLagPtHeartbeat: MySQL instance db2177:9104 has too large replication lag (11m 25s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[15:17:04] <jinxer-wm>	 FIRING: SystemdUnitFailed: pt-heartbeat-wikimedia.service on db2209:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[15:21:48] <jinxer-wm>	 FIRING: [12x] MysqlReplicationLagPtHeartbeat: MySQL instance db2156:9104 has too large replication lag (15m 41s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[15:50:10] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s3 on db2209 is CRITICAL: 2444 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104
[16:05:08] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s3 on db2209 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2209&var-port=9104