[09:27:58] <jynus>	 I've seen you are depooling db1107 4 times this week, is there any issue with it?
[09:28:23] <marostegui>	 It is 10.4 host, I am pooling during the day and depooling at the end of my day
[09:28:34] <marostegui>	 I want to be consistent with my email :)
[09:28:44] <jynus>	 but no observed issue yet?
[09:28:47] <marostegui>	 nop
[09:28:50] <marostegui>	 all good
[09:29:16] <jynus>	 ok, I got worried it could have a real issue rather than just being careful
[09:29:26] <marostegui>	 no no, all is good
[09:29:37] <jynus>	 let me know if you want to pool it for longer and I can depoolit before I go
[09:29:46] <marostegui>	 will do, thanks :)
[09:30:00] <jynus>	 I belive peak time is around 6
[09:30:25] <marostegui>	 yeah, not worried about that yet, but about query plans and unexpected issues with MW and GTID for now
[09:30:40] <marostegui>	 once I am fully sure with that, I will leave it pooled for longer with low weight and all that
[09:30:54] <jynus>	 so this is relevant to the mw log conversation
[09:31:00] <jynus>	 but if it is only one host
[09:31:06] <jynus>	 there is this trick you can use
[09:31:19] <jynus>	 which is flushing the p_s stats
[09:31:54] <jynus>	 (it should be more reliable than tendril)
[09:32:21] <marostegui>	 I am also capturing queries on the log, by enabling it
[09:32:24] <marostegui>	 and analyzing those
[09:34:12] <jynus>	 I found it, sys.ps_truncate_all_tables()
[09:34:28] <jynus>	 just giving you more options, there is many ways to do the same :-D
[09:34:36] <marostegui>	 sure :-)
[09:34:36] <marostegui>	 thanks
[09:41:32] <jynus>	 interesting dashboard: more QPS doesn't necessarily mean higher latency: https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=MariaDB+read+only+s8
[09:42:29] <marostegui>	 we are mixing different kind of traffics there too, so maybe less QPS but heavier
[09:42:34] <jynus>	 yeah
[09:42:50] <jynus>	 I was worried at first when I saw a host with 30K QPS
[09:43:01] <marostegui>	 it only does main, db1126
[09:43:03] <jynus>	 but it turns out in some cases it is the least loaded one
[09:43:11] <jynus>	 api is probably meaner to mysql
[12:35:08] <jynus>	 I've just seen db1087 is at 88% /srv usage, but cannot see why if is different from the other hosts
[12:35:16] <jynus>	 *it
[12:37:03] <jynus>	 I am guessing row format
[12:58:32] <marostegui>	 db1087 is scheduled to be compressed, starting as soon as I am done with another alter on db1099
[12:59:43] <jynus>	 sorry, I didn't meant to stress you, I just didn't know at first why
[13:00:05] <jynus>	 I am surprised wikidata is now 3.2TB uncompressed
[13:00:48] <jynus>	 so my "backing up 2TB databases uncompressed" are already outdated
[13:00:59] <jynus>	 :-(
[13:04:27] <jynus>	 most of the increase in growth rate happened during november
[13:23:47] <jynus>	 https://i.imgur.com/aYYFcqQ.jpg
[13:24:08] <marostegui>	 XDD
[16:23:03] <jynus>	 working to resolve a 2011 bug now :-D
[19:04:00] <wikibugs>	 10DBA, 10Cloud-Services: Prepare and check storage layer for ngwikimedia - https://phabricator.wikimedia.org/T240772 (10Urbanecm) @Marostegui I've created the database now!