[09:03:16] <volans>	 next delayed slaves broken replica expected at 10:05 (s2), 10:10 (s3), 10:15 (s4), 10:16 (s5), 10:17 (s6), 10:18 (s7), 10:19 (x1)
[09:03:39] <jynus>	 lol
[09:03:53] <jynus>	 I do not think it will be so smooth
[09:04:21] <jynus>	 m3 failed at 4-5am
[09:04:53] <volans>	 in the sense that is not 24h?
[09:05:08] <volans>	 *all times above are in GMT+1
[09:05:10] <jynus>	 this delayed replication do not work so well when there is very little load
[09:05:37] <jynus>	 I would change it, but it is not worth it
[09:06:36] <jynus>	 it seems that heartbeat is either run manually on the old slaves, or the new execution works as well
[09:07:40] <jynus>	 and the check works because it doesn't check for the executable, but for the table
[09:08:02] <jynus>	 so, here is the plan: execute it manually on the old slaves
[09:08:13] <jynus>	 no need for an extra check, because the current works
[09:08:27] <jynus>	 and if heartbeat fails, *WE* will notice
[09:08:51] <jynus>	 and then focus on the heartbeat execution/maintenance only on the new masters
[09:09:00] <jynus>	 which will be in production as sson as the failover
[09:09:28] <volans>	 +1, we can be human puppets for the old master's heartbeat :)
[09:09:29] <jynus>	 not migrate any critical system to heartbeat (like mediawiki) until the failover
[09:09:49] <jynus>	 an until we spend a few weeks testing it all
[09:10:14] <volans>	 make sense
[09:10:35] <jynus>	 bad things that will require solution: intermediate slaves lag
[09:11:11] <volans>	 in the sense that mediawiki will not see the delay right?
[09:11:14] <jynus>	 but given that we should only have 3 tiers a) during failover b) for WAN replication, which do not page
[09:11:26] <jynus>	 I think it is assumable
[09:12:03] <jynus>	 we will see, we will constantly tune between noise and alertness
[09:12:23] <jynus>	 mediawiki will continue using seconds behind master until a patch is applied
[09:12:35] <jynus>	 but that is ok, because there are other blockers for multi-tier slaves
[09:12:56] <jynus>	 for example, recently I discovered that slave delay is checked using the binlog
[09:13:13] <jynus>	 and that doesn't work for multi-tier slaves
[09:13:20] <volans>	 the position?
[09:13:23] <jynus>	 yes
[09:13:26] <volans>	 too bad
[09:13:29] <jynus>	 not for delay
[09:13:46] <jynus>	 but for "is this slave fresh enough after writing?"
[09:14:04] <jynus>	 it should have to change to either gtid, as we will
[09:14:08] <jynus>	 or heartbeat
[09:14:24] <jynus>	 (which would provide a pseudo-gtid)
[09:14:29] <volans>	 [s2 on dbstore2001]
[09:14:33] <volans>	 you already there?
[09:14:43] <jynus>	 doing
[09:17:09] <volans>	 if you need help let me know
[09:17:37] <jynus>	 it is ok, focus on your things
[09:17:57] <jynus>	 it is a waste of time 2 people doing do the same
[09:23:03] <jynus>	 so, how accurate that was to your prediction? :-)
[09:24:02] <volans>	 not at all :D
[09:24:20] <jynus>	 better than I expected
[09:24:40] <volans>	 you know, time is relative...
[09:24:57] <jynus>	 we didn't have into account relativistic errors
[09:25:03] <jynus>	 that was it
[09:25:34] <volans>	 next time I'll multiply by sqrt(1-v**2/c**2)
[09:26:45] <jynus>	 better way to accelerate servers -9.8m/s^2
[09:43:46] <volans>	 when you have a min: https://gerrit.wikimedia.org/r/#/c/274914/1
[09:45:23] <volans>	 if you think it's "too much" I can do only 2 in es1, I applied the principle that they can be repooled quickly (no replica to catch) and codfw is not yet prod
[09:45:33] <jynus>	 +1
[09:46:04] <jynus>	 the ones with load 1 are the masters, right?
[09:46:13] <jynus>	 "intermediate master"
[09:46:50] <volans>	 yes, I we changed topology so 2009 and 2005 are local masters in codfw for es3/2
[09:46:53] <jynus>	 it doesn't matter much, but I usually put those first in the config to avoid missunderstandings
[09:47:13] <jynus>	 e.g. we failover to codfw and delete the first line (real masters)
[09:47:33] <jynus>	 I hope you agree with that
[09:47:41] <volans>	 make sense, let me update it
[09:47:56] <jynus>	 let me comment on the patch so it is in written form
[09:49:56] <jynus>	 BTW and FYI, codfw.php is pending a large patch: https://gerrit.wikimedia.org/r/#/c/267659/
[09:50:56] <volans>	 good to know
[11:19:23] <volans>	 there is an easy way to get changed commits in submodule update directly from gerrit?
[11:19:52] <jynus>	 pfff
[11:20:11] <jynus>	 from the local machine, easier
[11:20:17] <volans>	 I guess not according to https://code.google.com/p/gerrit/issues/detail?id=1832
[11:21:09] <jynus>	 in theory, we should always point to HEAD, unless there is a reason not to
[11:21:48] <jynus>	 which, btw, I am not sure if you and mutante did after your last commits
[11:22:47] <jynus>	 it was not you, it was mutante
[11:22:54] <jynus>	 that did not update the main repo
[11:23:25] <jynus>	 not a big deal
[11:23:27] <volans>	 ok, I'll keep in mind
[11:24:51] <jynus>	 so, as puppet runs first on the hosts, and later on neon, the replication lag should get filled with decimals all around
[11:25:19] <volans>	 great
[11:25:21] <jynus>	 not that we need the decimals, but it is the confirmation that it is working
[11:48:10] <jynus>	 (for my own log) hosts than have not been fully heartbeat'ed: toolsdb (not active), dbstore[12]001 (delayed slave, it will take 24 hour to take effect), db1047 and db2002 (need a restart/change on replication filter)
[12:12:07] <jynus>	 db1048 and db2012 needs to white list heartbeat, too
[18:36:17] <volans>	 SSL proposals on the ticket... there is no TL;DR though... be prepared ;)