[14:16:31] <volans>	 I'm about to repool db1029 with 10% of the load
[14:16:39] <jynus>	 +1
[15:26:59] <volans>	 https://mariadb.com/kb/en/mariadb/performance-schema-system-variables/#performance_schema_digests_size
[15:27:24] <volans>	 Range: -1 to 200... default value (mariadb <= 10.0.11) 1000.... LOL
[15:47:13] <jynus>	 I spent some time tuning that some time ago, didn't touch them since then
[15:47:42] <jynus>	 it was taking by default like 10GB of memory due to max_connections
[15:47:59] <volans>	 for this specific config we have -1 on my.cnf that is autosizing
[15:48:05] <volans>	 and the default since 10.0.12
[15:54:22] <volans>	 if you agree I would like to change live max_allowed_packet, innodb_buffer_pool_dump_at_shutdown and innodb_buffer_pool_load_at_startup to have the same values of my.cnf (33554432, 1, 1)
[15:54:50] <jynus>	 sure
[15:55:38] <jynus>	 db1033 is almost ready to come back
[15:56:38] <volans>	 great
[15:57:28] <volans>	 didn't you want to keep db1079 name spare and start from db1080?
[15:57:34] <volans>	 for the new hosts
[16:00:54] <jynus>	 I did, but I do not want to bother chris just for that
[16:01:21] <jynus>	 it is not like we will run out of numbers
[16:01:23] <cmjohnson1>	 ugh..wish you would've told me I just did all the dns
[16:01:33] <volans>	 ok, saw the CR on -operations
[16:01:57] <jynus>	 Actually, I did https://phabricator.wikimedia.org/T131368#2269247
[16:02:14] <cmjohnson1>	 yep you did....sorry missed that
[16:02:25] <cmjohnson1>	 do you want me to fix it?
[16:02:28] <jynus>	 no
[16:02:44] <jynus>	 we just need to adjust the names on robs accounting
[16:03:28] <cmjohnson1>	 ok..sorry it was on teh procurement ticket I don't look at those for instructions..
[16:03:35] <jynus>	 no problem
[16:03:49] <jynus>	 if that would have caused issues, I would have commented it before
[16:04:32] <cmjohnson1>	 cool, thanks!
[16:05:01] <cmjohnson1>	 yeah I am also seeing racking instructions
[16:05:28] <cmjohnson1>	 I have 4 in each row..including D
[16:07:11] <volans>	 cmjohnson1: yes saw your physical disposition, I've already made a plan for which one to pick for each shard here: https://phabricator.wikimedia.org/T111992#2308149
[16:07:45] <cmjohnson1>	 cool...sorry for the miscommunication
[16:08:17] <jynus>	 no need to say sorry, as I said, if it was an issue, I would have pinged you
[16:08:34] <volans>	 and this just thanks to the details you had in your ticket, I was able to make it even before we know the names :)
[16:18:50] <cmjohnson1>	 volans: 1 small change to that racking order...I found out I am short 1 power outlet still in B1 so instead of asking for another server to decom I can move  2 to B3
[16:20:31] <volans>	 cmjohnson1: for me B3 is even better, quite a few shards have already servers on B1
[16:20:48] <cmjohnson1>	 cool...then db1085 and db1086
[16:21:00] <cmjohnson1>	 go to b3...i will update task
[16:22:07] <volans>	 if you could add there the mapping rack-name there for those it will save me to lookup all of them on racktables :-P
[16:22:14] <volans>	 if you have it at hand
[16:33:48] <volans>	 jynus: could you also increase a bit the weight of db1029 please? Probably like 4
[16:33:48] <Cyberpower678>	 volans, I transfered the rows of the old table which is 23.7 GiB in size to the three new tables which totals 9.7 GiB in size.
[16:34:12] <Cyberpower678>	 So now it's using 2.5 times less space.
[16:35:22] <volans>	 that's good, most of the saved space was in the indexes
[16:36:05] <jynus>	 I have some doubts about x1, it used to be 20 / 10
[16:36:16] <jynus>	 but I do not know if it was before or after failover
[16:36:20] <jynus>	 let me check the history
[16:37:13] <Cyberpower678>	 volans, now to test the most important query
[16:37:15] <Cyberpower678>	 SELECT pageid, externallinks_global.url_id, externallinks_global.paywall_id, url, archive_url, has_archive, live_state, unix_timestamp(last_deadCheck) AS last_deadCheck, archivable, archived, archive_failure, unix_timestamp(access_time) AS access_time, unix_timestamp(archive_time) AS archive_time, paywall_status, reviewed, notified
[16:37:15] <Cyberpower678>	 										 FROM externallinks_".WIKIPEDIA."
[16:37:15] <Cyberpower678>	 										 LEFT JOIN externallinks_global ON externallinks_global.url_id = externallinks_".WIKIPEDIA.".url_id
[16:37:15] <Cyberpower678>	 										 LEFT JOIN externallinks_paywall ON externallinks_global.paywall_id = externallinks_paywall.paywall_id
[16:37:18] <Cyberpower678>	 										 WHERE `pageid` = '{$this->commObject->pageid}';
[16:38:11] <jynus>	 Cyberpower678, plase use a paste for that: https://phabricator.wikimedia.org/paste/edit/form/14/
[16:38:56] <Cyberpower678>	 jynus, I generally use a paste if it's more than 5 lines
[16:40:43] <Cyberpower678>	 Wow.  Fast.  0.063 seconds to execute and deliver a query. :D
[17:06:38] <jynus>	 volans, I've checked and the usual load was 10-master/20-slave for x1
[17:06:59] <volans>	 it make sense
[17:07:14] <jynus>	 I will deploy now https://gerrit.wikimedia.org/r/#/c/289686/
[17:07:53] <volans>	 ok, 13GB of RAM used on a 64GB host... it will help to fill the buffer pool
[17:53:39] <volans>	 jynus: lag on db2037 started when the error log started to log:
[17:53:48] <volans>	 InnoDB: Space id 24 trying to reserve 3 extents actually reserved 256  reserve 1144 free 977 size 7313408 rounds 51 total_reserved 13056
[17:54:57] <volans>	 https://github.com/MariaDB/server/blob/10.0/storage/xtradb/fsp/fsp0fsp.cc#L2830
[17:55:05] <jynus>	 which version?
[17:55:32] <volans>	 10.0.22-1
[17:55:36] <volans>	 ours
[17:55:52] <volans>	 my link is for the HEAD of 10.0
[17:56:08] <jynus>	 yes was asking for ours
[17:56:42] <jynus>	 I think I've never found contention on extending data- my bet is on disk issues
[17:57:19] <volans>	 this should be the right one
[17:57:19] <volans>	 https://github.com/MariaDB/server/blob/d775ecdd010daad4dc6147fba58acd006bf2c60c/storage/xtradb/fsp/fsp0fsp.cc#L2812
[17:59:42] <jynus>	 I mean, we had way more intense writes before and didn't find this, this is coming from replication, single thread!
[18:04:25] <jynus>	 bah, not worth it
[18:04:50] <volans>	 what, investigating? :)