[05:53:48] <jinxer-wm>	 FIRING: MysqlReplicationLag: MySQL instance es1039:9104@es7 has too large replication lag (15h 58m 28s). Its replication source is es1035.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=es1039&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[05:53:48] <jinxer-wm>	 FIRING: MysqlReplicationLagPtHeartbeat: MySQL instance es1039:9104 has too large replication lag (15h 58m 34s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=es1039&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[05:58:48] <jinxer-wm>	 RESOLVED: MysqlReplicationLag: MySQL instance es1039:9104@es7 has too large replication lag (4h 38m 23s). Its replication source is es1035.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=es1039&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[05:58:48] <jinxer-wm>	 RESOLVED: MysqlReplicationLagPtHeartbeat: MySQL instance es1039:9104 has too large replication lag (2h 31m 17s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=es1039&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[12:02:15] <Emperor>	 I understand why some of these thanos backfills are taking ages - huge tegola container DBs :(
[12:07:17] <Emperor>	 26G and 25G on this one backend host
[12:08:33] <jynus>	 TIL tegola
[12:12:28] <Emperor>	 storage for map tiles
[12:13:10] <jynus>	 I didn't know it was called that
[14:30:25] <jynus>	 another thing I learned today is that you can run "systemctl stop /partition"
[14:30:30] <jynus>	 to umount
[14:34:25] <TheresNoTime>	 (huh TIL too)
[14:45:07] <jynus>	 and another thing I learned is that I cannot have the AC unit, the washing machine and my pc switched on at the same time
[14:47:49] <federico3>	 jynus: sounds like a very italian experience
[15:48:06] <jynus>	 I am going to restart backupmon, which will create a small unavailability
[15:48:30] <jynus>	 I don't think anyone here use that other than me, but giving a heads up just in case