[00:11:45] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on db2112 is CRITICAL: 107.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2112&var-port=9104
[00:12:45] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on db1106 is CRITICAL: 6.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1106&var-port=9104
[00:12:53] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on db1163 is CRITICAL: 2.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1163&var-port=9104
[00:13:09] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on db2116 is CRITICAL: 28.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2116&var-port=9104
[00:13:33] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on db2146 is CRITICAL: 35 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2146&var-port=9104
[00:14:51] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on db1106 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1106&var-port=9104
[00:14:59] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on db1163 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1163&var-port=9104
[00:17:17] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on db2116 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2116&var-port=9104
[00:17:37] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on db2146 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2146&var-port=9104
[00:17:59] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on db2112 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2112&var-port=9104
[02:24:15] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104
[02:28:35] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104
[06:56:24] <RhinosF1>	 Amir1: what script did you use for finding schema drifts?
[06:57:28] <Amir1>	 RhinosF1: I wrote one https://github.com/Ladsgroup/db-analyzor-tools/blob/master/db_drift_checker.py
[06:59:16] <RhinosF1>	 ty
[13:19:29] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104
[13:24:05] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104