[11:26:28] <dhinus>	 I'm very puzzled by this mariadb replication issue: https://phabricator.wikimedia.org/T386240#10548756
[11:26:45] <dhinus>	 I'm tempted to just recreate the replica from scratch, but I would really like to understand what's going on :)
[13:34:41] <Amir1>	 dhinus: I can take a look after I'm done with a really messy UBN
[13:40:08] <dhinus>	 Amir1: no rush, but any hint is appreciated!
[13:42:04] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db1227 is CRITICAL: 21.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1227&var-port=9104
[13:42:06] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db2182 is CRITICAL: 12 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2182&var-port=9104
[13:42:08] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db2168 is CRITICAL: 15.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2168&var-port=9104
[13:42:10] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db2221 is CRITICAL: 19 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2221&var-port=9104
[13:42:12] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db2208 is CRITICAL: 18 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2208&var-port=9104
[13:42:12] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db2220 is CRITICAL: 19.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2220&var-port=9104
[13:42:24] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db2150 is CRITICAL: 17.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2150&var-port=9104
[13:43:01] <Amir1>	 in orch it seems fine
[13:43:12] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s7 on db2220 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2220&var-port=9104
[13:43:13] <Amir1>	 probably some large transaction went throuh?
[13:44:06] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s7 on db2182 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2182&var-port=9104
[13:44:08] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s7 on db2168 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2168&var-port=9104
[13:44:08] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s7 on db2221 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2221&var-port=9104
[13:44:12] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s7 on db2208 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2208&var-port=9104
[13:44:26] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s7 on db2150 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2150&var-port=9104
[13:45:04] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s7 on db1227 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1227&var-port=9104