[10:21:14] * volans restarting replica on dbstore1001 skipping the various missing hitcounters
[10:31:24] <volans>	 jynus: given that you're around, I would like a second pair of eyes if you have a minute
[10:32:07] <jynus>	 alternatively, we stop replication and put a filter on %.hitcounter et al
[10:32:19] <jynus>	 yes?
[10:32:56] <volans>	 sanitarium:3311... I found a broken replica now and I was trying to fix it
[10:33:11] <jynus>	 mmmm
[10:33:15] <volans>	 so the first error was a duplicate key on revision, given that sanitarium too on enwiki
[10:33:19] <jynus>	 not the first time it happend
[10:33:25] <volans>	 has the primary key rev_page, rev_id
[10:33:39] <jynus>	 not sure if tokudb issue on insert
[10:34:03] <volans>	 I've checked the content was exactly the same of the existing one except for rev_text_id that is forced to 0 by the trigger
[10:34:07] <jynus>	 a key issue due to unsafe statements due to insert select
[10:34:15] <volans>	 so I decided to skip and restart... BUT
[10:34:30] <jynus>	 or revision is in a bad state
[10:34:37] <jynus>	 do not give it too much thought
[10:34:46] <jynus>	 because I have not yet reimported it
[10:34:46] <volans>	 the problem was not that
[10:34:50] <volans>	 was after the start slave :D
[10:34:54] <volans>	 Relay log read failure: Could not parse relay log event entry
[10:35:00] <jynus>	 ah, yes
[10:35:22] <volans>	 so I'd like a second pair of eyes before doing a change master to force the redownload of relay from the master
[10:35:24] <jynus>	 just reset slave all; change master (obviously, save de coords first
[10:35:51] <jynus>	 that is why I think it is probably a tokudb / version issue
[10:35:57] <jynus>	 that used to happen a lot
[10:36:02] <jynus>	 no only there
[10:36:18] <jynus>	 which means it could be <= .15 related
[10:36:32] <jynus>	 so basically you were doing what I did
[10:37:03] <volans>	 yep, just that one thing I cannot explain
[10:37:04] <jynus>	 just make sure you use the right master log
[10:37:28] <volans>	 on the mysqlbinlog on the master at that position there is one more query before the one that breaks
[10:37:52] <volans>	 I'm looking at Relay_Master_Log_File: db1057-bin.002016 Exec_Master_Log_Pos: 1035866585
[10:38:59] <jynus>	 db1057-bin.002016
[10:39:09] <jynus>	 not 18, which is the downloaded
[10:39:11] <jynus>	 right
[10:39:30] <volans>	 yes of course
[10:39:50] <jynus>	 this is one of the times in which we do not do things 100% well because they are going to be reimported anyway
[10:40:11] <jynus>	 there is like 20 reports of data differences
[10:40:23] <jynus>	 in part also because the filters
[10:40:59] <jynus>	 however, my plan to simplify both labs and dbstore management
[10:41:23] <jynus>	 is to get rid of toku- that would allow to simply copy raw table files
[10:42:35] <volans>	 yeah
[10:45:19] <volans>	 ok, I'll proceed :)
[10:45:49] <jynus>	 I cannot login now to phab, but I mention because I searched procurement agreement on dbstore-like task
[10:46:18] <jynus>	 that should fix it, at least for a few days
[10:46:44] <jynus>	 we must thing on upgrading that- if possible, directly to 10.1
[10:46:50] <jynus>	 *think
[10:47:09] <volans>	 it would help a lot, yes
[10:47:49] <jynus>	 the other thing I want to mention (despite logging)
[10:47:57] <jynus>	 was the x1-codfw
[10:48:19] <jynus>	 I found an unused piece of hardware and reimagined it
[10:48:26] <jynus>	 *reimaged
[10:48:46] <jynus>	 sorry you did db2008 for nothing
[10:49:00] <jynus>	 (actually, it helped)
[10:49:09] <volans>	 no prob at all :)
[10:49:28] <jynus>	 but db2033 has better, on warranty hard
[10:49:51] <jynus>	 and the more old machines we can get rid off, the better
[10:51:57] <jynus>	 on https://tendril.wikimedia.org/host/view/db1069.eqiad.wmnet/3311
[10:52:06] <jynus>	 you can see this happened a number of times
[10:52:20] <jynus>	 plus one time where update got "stuck"
[10:52:37] <volans>	 yes
[10:52:40] <jynus>	 (weekly replication lag)
[10:52:58] <volans>	 restarted, is recovering now, all looks good
[10:53:24] <jynus>	 my bet on bug
[10:53:54] <jynus>	 maybe only happens on 10 master
[10:54:23] <jynus>	 these are the things in which slave version > master
[10:54:27] <jynus>	 does matter
[10:54:50] <volans>	 yes, make sense
[10:55:13] <volans>	 although it already had all the relay logs locally
[10:56:00] <jynus>	 I do not have counter-argument to that :-)
[10:56:30] <jynus>	 but the fact is it is the only .15 enwiki slave
[10:57:21] <volans>	 together with labsdb1001 true
[10:59:30] <jynus>	 labsdb1001 is not a direct slave, does not count
[11:00:23] <jynus>	 bump, at your discretion, the priority of the sanitarium upgrade ticket
[11:00:32] <volans>	 yeah it's just a slave of sanitarium...
[11:01:23] <volans>	 it's already high :) (the ticket)