[02:21:42] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s4 on db2219 is CRITICAL: 48.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2219&var-port=9104
[02:24:42] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s4 on db2219 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2219&var-port=9104
[07:48:13] <jynus>	 db1239 catching up replication now
[07:50:30] <marostegui>	 jynus: Nice!
[07:50:34] <marostegui>	 I will move it when it is in sync
[08:07:26] <marostegui>	 I am switching es5 master
[08:08:16] <marostegui>	 Done
[08:16:11] <marostegui>	 Amir1: I forgot about --check in the schema change, it only checks for replicas right? 
[10:06:05] <jynus>	 2 hosts are rebuilding tables on several sections: db1225 and db1216
[10:06:16] <jynus>	 should be done by the end of day
[13:05:07] <jynus>	 db1239:s2 caught up (s1 will do soon)
[13:06:17] <jynus>	 for db1216:s8 I need to rebuild a lot of tables, so it may need to wait until monday
[13:59:27] <Amir1>	 marostegui: it checks for replicas but if you run it with --dc-masters it runs the check on masters
[13:59:35] <Amir1>	 (granted the replicas variable is None)
[14:02:25] <Amir1>	 marostegui: If you asked that question one hour earlier, I would have answered you before sleep :D
[14:08:00] <jynus>	 any tips on how to handle db1216. the table rebuilding will finish in a few hours, but I am soon going to finish my day
[14:08:10] <jynus>	 I have downtimed it until monday
[14:30:30] <marostegui>	 jynus: Just leave it until monday then :)
[14:44:27] <jynus>	 will do, then
[14:44:31] <jynus>	 have a nice weekend
[14:45:19] <marostegui>	 you too!
[14:48:03] <marostegui>	 Amir1: Thanks, yeah, I figured earlier, but forgot to post it. :)
[14:48:05] <marostegui>	 Thank you!