[08:03:45] <volans>	 jynus: are you working on dbstore1002? replica stopped on s1 for duplicate entry
[08:07:16] <volans>	 from backlog it seems your importing data from dbstore1002, not to it... but double checking
[08:15:15] <jynus>	 yeah, that is something else
[08:15:27] <jynus>	 but I am trying to fix it reimporting from the master
[08:16:31] <volans>	 what do you mean?
[08:17:17] <jynus>	 the table archive seems to be gone out of sync
[08:18:37] <jynus>	 probably because unsafe statements (insert..select) + STATEMENT replication + TokuDB + locks
[08:19:09] <volans>	 probably
[08:22:13] <jynus>	 there is a chance of out of band writes, too
[08:22:39] <jynus>	 that is the 1 of the 2 only slaves that is not read only, due to analytic usage
[08:23:38] <jynus>	 but the failures seem to have to do with the archive table, which is highly conflictive due to being used always as INSERT...SELECT
[08:24:00] <jynus>	 something that should be banned or ROW replication should be used
[08:42:52] <jynus>	 one thing you could help me with is db1047
[08:43:03] <volans>	 sure, tell me
[08:43:12] <jynus>	 I restarted it yesterday, due to full aria corruption
[08:43:21] <jynus>	 like the one on dbstore2002?
[08:43:25] <jynus>	 one of those
[08:43:36] <volans>	 great
[08:43:38] <jynus>	 that is ok, and in general everthing is ok
[08:43:41] <volans>	 saw in the backlog
[08:43:54] <jynus>	 I converted the table to tokudb, it should not be an issue again
[08:44:15] <jynus>	 the problem now is that replication is behind
[08:44:28] <jynus>	 it seems that write activity is lower than yesterday
[08:44:41] <jynus>	 so low that in 24 hours it is 22 hours behind
[08:44:53] <volans>	 not cool
[08:45:21] <volans>	 only s1? s2 is stopped
[08:45:23] <jynus>	 try to see why- tokudb issue? too much load? swapping? storage issue?
[08:45:30] <jynus>	 I stopped manually one of the 2
[08:45:38] <jynus>	 to see if I could catch up the other
[08:45:44] <volans>	 ok, looking
[08:46:10] <jynus>	 (e.g. using the buffer for only one database)
[08:46:34] <jynus>	 it could be a storage issue, that RAID had problems in the past
[08:46:51] <volans>	 I/O wait are around 18% for a start
[08:46:55] <jynus>	 the question is, whatever it takes, try to make it faster
[08:46:58] <volans>	 ok
[08:47:18] <jynus>	 no matter if it compromises the stability of the data, it is not a core production server
[08:47:25] <jynus>	 but it is used by analytics
[08:47:49] <jynus>	 try to do it without restarting the server, I did it yesterday and people were not very happy
[08:48:04] <jynus>	 if it is needed, it is ok, but we need to say so before
[08:48:31] <jynus>	 basically check the logs for something wrong like corruption, or RAID issues
[08:48:59] <jynus>	 either the iops are going somehere, or we have lost them
[08:49:57] <volans>	 so the current cache policy for 2 out of the 3 disks is WriteThrough instead of WriteBack, it might explain it
[08:50:53] <jynus>	 the reason for that is probably battery issues
[08:50:59] <jynus>	 which I saw in the past
[08:51:02] <jynus>	 override that
[08:51:29] <jynus>	 there is no original data there (and if it is, people should have a copy)
[08:51:36] <jynus>	 it may have been reseted on restart
[08:54:44] <volans>	 yes, battery for adapter 1 is faulty
[08:55:07] <jynus>	 there is a similar command on the troubleshooting section of mariadb
[08:55:33] <jynus>	 just override it, we will order a repacement of that in the next batch
[08:56:29] <jynus>	 I think I disabled the learning mode on all RAIDs also, but be aware of previous issues with that in the past
[08:59:27] <volans>	 done
[09:01:32] <volans>	 checking if recover more quickly
[09:01:34] <jynus>	 ok, now wait and see on tendril if io goes to similar levels/replication starts decreasing
[09:01:37] <jynus>	 :-)
[09:05:30] <jynus>	 TokuDB is an issue, but only 20% because it is TokuDB, and a 80% because it is not InnoDB
[09:06:33] <jynus>	 I know this is is probably hardware, but in the end, I do not think compression is worth all the issues
[09:08:36] <volans>	 lol, btw db1047 has also an LVM logical volume across 2 HW RAID virtual disks
[09:10:12] <volans>	 doesn't seems to have speed up much, I'll continue to look at other way to speed it up
[09:18:05] <volans>	 at least it inverted trend, starting to recover
[09:18:39] <jynus>	 dbstore1002 should be ok now
[09:18:55] <jynus>	 and replication going down
[15:29:56] <jynus>	 volans, dbstore1001 is at 91% disk usage
[15:30:21] <jynus>	 that is ok-ish because it is 700GB free
[15:31:17] <volans>	 ok
[15:31:18] <jynus>	 but we may want to check potential optimizations/disk saves
[15:31:25] <jynus>	 or a long-term plan
[15:32:51] <jynus>	 there is 353 GB unallocated, just in case
[15:48:38] <jynus>	 volans, you should be able to see: https://phabricator.wikimedia.org/T123379#2164999 and https://phabricator.wikimedia.org/T131363
[15:49:02] <jynus>	 you were asked to be on the loop, and probably will need to be involved a bit
[15:49:47] <volans>	 yes I've access
[15:50:35] <jynus>	 take a look at it, because they may ask question such as "is 7.6TB ok, or do we go for 11.5TB"
[15:50:59] <jynus>	 and it should not be blocked for such easy questions, you can answer those
[16:13:56] <jynus>	 If you have the time, could you add a 1 line to wikitech? "mysql servers should be added to X property on hiera:path"
[16:15:15] <volans>	 in the setup of new replicas?
[16:18:17] <jynus>	 anywhere, do not give it too much thought :-P
[16:18:50] <jynus>	 whereve you can tell me "jynus you did wrong because you didn't followed the conventions"
[16:19:07] <volans>	 lol
[16:19:24] * volans needs to look more at how the whole hiera works
[16:19:35] <jynus>	 didn't you modify hiera??
[16:20:24] <jynus>	 we should use it more, but every time I do somthing dirty I say "It is only temporary, so it is not worth it"
[16:20:39] <jynus>	 and then it stays like that for years
[16:21:50] <volans>	 yeah I modified it
[16:26:17] <volans>	 done: https://wikitech.wikimedia.org/wiki/Setting_up_a_MySQL_replica#Puppet
[16:28:54] <jynus>	 I will try to update the old coredb module there too at some point
[16:29:56] <jynus>	 the whole "topology in puppet" is deprecated
[16:30:42] <jynus>	 (we would not even want to mar*k the masters on puppet)
[16:31:46] <volans>	 true
[16:59:23] <volans>	 I've added the expose puppet cert to the coredb role too, it's just generating the files, they will not be referenced in the my.cnf for now
[17:02:30] <jynus>	 sure
[18:46:29] <volans>	 should we wait to restart the replica on s2 on db1047 until s1 catches fully up? I think they can run together at this point
[18:47:51] <jynus>	 sorry, I didn't control it, is it catching up?
[18:48:16] <jynus>	 if yes, just start it when you can (or I will do it tomorrow)
[18:48:24] <jynus>	 not a big deal
[18:50:26] <volans>	 is catching, but still very behind
[19:14:01] <volans>	 I've restarted slave on s2, I'll keep an eye on it, if it slows too much s1 I'll stop it again
[19:15:11] <jynus>	 it is ok, replication should not be a huge issue
[19:15:18] <jynus>	 lag I mean
[19:15:40] <jynus>	 there, analytics use it for long-running queries mostly
[19:16:18] <jynus>	 (I got worried when it got to 24 hours and going up)
[19:16:30] <volans>	 yep