[01:55:49] <wikibugs>	 10DBA, 06Operations, 10Phabricator: Intermittent DB connectivity problem on phabricator, needs investigation - https://phabricator.wikimedia.org/T163507#3200523 (10mmodell)
[06:12:46] <wikibugs>	 10DBA, 07Schema-change: Truncate SHA-1 indexes - https://phabricator.wikimedia.org/T51190#3200704 (10jcrespo) So we go for full production?
[07:33:48] <jynus>	 so what are you into now?
[07:34:03] <marostegui>	 I am going to start recloning db1063
[07:34:05] <marostegui>	 so it is ready for next week
[07:34:13] <marostegui>	 there is an alter going on on db1065 (enwiki)
[07:34:13] <jynus>	 that is s5?
[07:34:16] <marostegui>	 yes
[07:34:27] <jynus>	 revision?
[07:34:31] <marostegui>	 yes
[07:34:39] <marostegui>	 Started it before I saw all the discussion
[07:34:42] <marostegui>	 about removing the index
[07:34:53] <jynus>	 it is ok
[07:34:57] <marostegui>	 btw, I am tempted to decom db2062, give more weight to db2071 and start analyze over the weekend
[07:35:07] <jynus>	 db2062 ?
[07:35:16] <marostegui>	 api s1 overloaded slaves
[07:35:16] <jynus>	 you mean depool?
[07:35:22] <marostegui>	 sorry yes
[07:35:23] <marostegui>	 depool
[07:35:24] <marostegui>	 :)
[07:35:33] <jynus>	 yeah, I was going to do one of those
[07:35:37] <marostegui>	 ah
[07:35:39] <jynus>	 not sure which one
[07:35:39] <marostegui>	 good :)
[07:35:48] <marostegui>	 either db2062 or 69
[07:35:50] <marostegui>	 i would sya
[07:36:20] <marostegui>	 say
[07:36:21] <jynus>	 anything big we should do before the weekend?
[07:36:39] <jynus>	 analyze on eqiad? upgrades?
[07:36:49] <marostegui>	 Have you seen this
[07:36:51] <marostegui>	 (give me a sec)
[07:37:16] <marostegui>	 https://phabricator.wikimedia.org/T163351#3200731
[07:37:34] <jynus>	 yeah, I commented it with tim
[07:37:40] <marostegui>	 ah sorry, missed it
[07:37:46] <jynus>	 I think it is happening on reboot
[07:38:00] <jynus>	 did you see my latest comment about it?
[07:38:25] <jynus>	 https://phabricator.wikimedia.org/T163495#3200807
[07:39:21] <wikibugs>	 10DBA, 06Operations, 13Patch-For-Review, 05codfw-rollout: codfw API slaves overloaded during the 2017-04-19 codfw switch - https://phabricator.wikimedia.org/T163351#3200825 (10jcrespo)
[07:39:29] <jynus>	 I would say we close T163351
[07:39:30] <stashbot>	 T163351: codfw API slaves overloaded during the 2017-04-19 codfw switch - https://phabricator.wikimedia.org/T163351
[07:39:40] <jynus>	 because the title technicaly is not true
[07:39:43] <jynus>	 anymore
[07:39:45] <marostegui>	 oh
[07:39:45] <marostegui>	 i didn't see that one
[07:39:45] <marostegui>	 that is interesting
[07:40:16] <jynus>	 and focus on T163495 and its children
[07:40:17] <stashbot>	 T163495: Mediawiki revision-related queries are failing with high rate for enwiki on codfw - https://phabricator.wikimedia.org/T163495
[07:40:35] <jynus>	 creating a ticket per query if necessary
[07:41:15] <marostegui>	 yeah
[07:41:16] <jynus>	 are you ok with that? (and copy all relevant info that is not yet fixed)
[07:41:20] <marostegui>	 sure sure
[07:41:31] <marostegui>	 makes sense
[07:41:35] <jynus>	 basically leaving that for the incident
[07:41:44] <jynus>	 and this new one for the fallout
[07:41:54] <marostegui>	 yes, sounds good to me :)
[07:41:58] <marostegui>	 better organized
[07:42:42] <jynus>	 basically, adding those last comments to the specific ticket
[07:43:08] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 07Schema-change: Truncate SHA-1 indexes - https://phabricator.wikimedia.org/T51190#3200827 (10jcrespo)
[07:43:32] <wikibugs__>	 07Blocked-on-schema-change, 10DBA, 07Schema-change: Truncate SHA-1 indexes - https://phabricator.wikimedia.org/T51190#553002 (10jcrespo) Tim confirmed ok to deploy.
[07:43:59] <jynus>	 that is T116557
[07:44:00] <stashbot>	 T116557: AFComputedVariable::compute query timeouts - https://phabricator.wikimedia.org/T116557
[07:44:31] <marostegui>	 nice, oct 2016
[07:44:34] <marostegui>	 2015!
[07:44:42] <jynus>	 there was another issue
[07:46:11] <jynus>	 maybe I should resolve that one and create a new one to avoid confusion
[07:46:35] <marostegui>	 yeah, because it looks pretty similar
[07:46:38] <marostegui>	 at a first glance
[07:46:47] <marostegui>	 at least to someone like me without the context of what happened there
[07:47:42] <jynus>	 I will leave it there
[07:47:50] <jynus>	 It is the same scope, and probably related
[07:47:59] <marostegui>	 ok
[08:31:48] <jynus>	 so I think I am going to focus on 3 things
[08:31:56] <jynus>	 the ANALYZE
[08:32:07] <jynus>	 the harder query killer
[08:32:19] <jynus>	 and I cannot remember the third one
[08:32:34] <jynus>	 yes, moving dbstore1001 topology
[08:32:59] <jynus>	 I will do all at once 
[08:33:12] <marostegui>	 oh, the harder query killer +10000
[08:33:22] <marostegui>	 I guess only testing and not deploying on friday no?
[08:33:34] <jynus>	 no fully deploy, of course
[08:33:40] <marostegui>	 :)
[08:33:41] <jynus>	 but start with some adnaves
[08:33:46] <jynus>	 *advances
[08:34:24] <marostegui>	 should we leave an analyze running on db1080 (i think it was that one?) and db1065? from s1, the hosts that got restarted and now do filesort?
[08:34:39] <jynus>	 ok
[08:34:54] <marostegui>	 I can do it, not asking you to!
[08:35:10] <jynus>	 I need to see if https://gerrit.wikimedia.org/r/338996 is up to date
[08:35:23] <marostegui>	 let me see
[08:35:26] <jynus>	 to move dbstore1001 to the new masters
[08:35:50] <marostegui>	 s4 is the only one we changed to so far and it is correct, db1068 is the new s4 master
[08:36:10] <jynus>	 yeah, the plan
[08:36:13] <jynus>	 not the current state
[08:36:30] <jynus>	 you are cloning 63 now, right?
[08:36:32] <marostegui>	 yep
[08:36:42] <marostegui>	 so we have it ready next week
[08:36:49] <marostegui>	 I mean, for next week
[08:37:14] <jynus>	 so I need to check 61, 62 and 54
[08:37:29] <jynus>	 to see if they are the planned ones and still the right ones
[08:38:22] <marostegui>	 db1054 for s2 is the planned one
[08:38:28] <marostegui>	 62 for s7
[08:38:32] <marostegui>	 and 61 for s6
[08:38:40] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 13Patch-For-Review: *_minor_mime are varbinary(32) on WMF sites, out of sync with varbinary(100) in MW core - https://phabricator.wikimedia.org/T73563#3200904 (10jcrespo)
[08:38:42] <wikibugs__>	 10DBA, 06Operations, 13Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3200902 (10jcrespo)
[08:38:43] <marostegui>	 is the last plan we had on: https://phabricator.wikimedia.org/T162133
[08:38:47] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 10Expiring-Watchlist-Items, 10MediaWiki-Watchlist, and 3 others: Add wl_id to watchlist tables on production dbs - https://phabricator.wikimedia.org/T130067#3200903 (10jcrespo)
[08:38:50] <wikibugs>	 07Blocked-on-schema-change, 10DBA, 10Wikimedia-Site-requests, 06Wikisource, and 4 others: Schema change for page content language - https://phabricator.wikimedia.org/T69223#3200905 (10jcrespo)
[08:39:02] <jynus>	 I will re-review it
[08:39:08] <marostegui>	 cool
[08:39:38] <jynus>	 good news is that s4 we solved it the hard way- recloning
[08:39:47] <jynus>	 and s2 and s6 are almost fully checked
[08:40:02] <marostegui>	 yeah, and we even did a switchover before next week, one task less
[08:40:38] <jynus>	 I think I will change dbstore1001 master in advance
[08:40:47] <marostegui>	 that sounds like a great idea
[08:40:51] <jynus>	 so I only have to deal with that once
[08:41:01] <jynus>	 instead of the tedious process every time
[08:41:15] <marostegui>	 none of those new masters need recloning?
[08:41:36] <jynus>	 s7 is the one that worries me
[08:41:52] <jynus>	 I am fairly confident about s2 and s6
[08:42:17] <jynus>	 if we want to be 100% sure we can clone s7 from the current master?
[08:42:18] <jynus>	 or
[08:42:23] <jynus>	 run pt-table checksum
[08:42:31] <jynus>	 I do not know
[08:42:44] <marostegui>	 why do you have doubts about s7?
[08:42:50] <marostegui>	 something rings a bell?
[08:42:52] <jynus>	 it is more
[08:43:09] <jynus>	 we have checked s2 and s6
[08:43:13] <jynus>	 and cloned s4
[08:43:29] <jynus>	 and I was mostly worried about s4 integrity
[08:43:54] <jynus>	 s7 is just unknown
[08:43:58] <marostegui>	 we can always mysqldump from the master yes
[08:46:02] <jynus>	 we can also consider do master upgrades
[08:46:09] <jynus>	 of kernel, I mean
[08:46:31] <marostegui>	 ah, yes
[08:46:38] <marostegui>	 that would make moritzm happy :)
[08:46:53] <jynus>	 while they are SPOF in reality it takes less time to reboot them if they had issues
[09:00:50] <jynus>	 are you touching db1051?
[09:00:56] <marostegui>	 nop
[09:01:10] <marostegui>	 something wrong with it?
[09:01:20] <jynus>	 I am cleaning up old downtimes/disables I may have setup
[09:02:01] <marostegui>	 ah
[09:02:06] <marostegui>	 I should probably do the same
[09:02:33] <jynus>	 I think I did all
[09:03:34] <marostegui>	 thanks!
[09:04:49] <jynus>	 db1040 free space: https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&panelId=17&fullscreen&orgId=1&var-server=db1040&var-network=eth0
[09:05:29] <marostegui>	 still around 100G
[09:05:42] <jynus>	 yeah, no problem storage-wise
[09:05:49] <jynus>	 more for the alerts
[09:06:01] <marostegui>	 yeah, I would downtime it for the weekend 
[09:06:07] <jynus>	 I think we can change on hiera the theshold
[09:06:14] <marostegui>	 if you are moving dbstore out of it it will have no slaves
[09:06:23] <marostegui>	 so nothing to worry much about other than the data itself
[09:06:33] <marostegui>	 and after checking it we can throw it away and one box less
[09:06:36] <marostegui>	 and 2u more!
[09:08:24] <jynus>	 you can give a look at puppet/hieradata/hosts/es2001.yaml for other cases
[09:09:22] <marostegui>	 ah
[09:09:27] <marostegui>	 I didn't know that :)
[09:10:07] <marostegui>	 so there would be the place to also change the threshold for dbstore1001 and the timeouts for the slave checks?
[09:10:53] <jynus>	 well, that works because it is programmed like that
[09:11:14] <jynus>	 remember the socket thing I always talk about but never do
[09:11:28] <jynus>	 it is because I had to add that functionality to puppet!
[09:11:38] <marostegui>	 haha
[09:13:00] <jynus>	 See an example on: https://wikitech.wikimedia.org/wiki/Puppet_coding#A_working_example:_deployment_host
[09:14:58] <marostegui>	 will take a look
[09:16:17] <jynus>	 I will upgrade kernel and mysql on db2062 so we have a base line to start with
[09:16:28] <marostegui>	 cool, 10.0.30?
[09:16:31] <jynus>	 yes
[09:16:39] <marostegui>	 ok!
[09:16:46] <marostegui>	 will you run the analyze too?
[09:16:49] <jynus>	 if it is a problem with the latest version, we want to know
[09:16:55] <marostegui>	 yes
[09:16:57] <jynus>	 I do not care if it is a problem with older versions
[09:17:06] <jynus>	 or how to fix it there
[09:17:19] <jynus>	 yes, upgrade, reboot and analyze
[09:17:22] <marostegui>	 great!
[09:17:33] <jynus>	 so we start from a common state
[09:18:07] <jynus>	 BTW
[09:18:19] <jynus>	 when you upgrade mariadb, remember to run puppet afterwards
[09:18:38] <jynus>	 as sometimes puppet overwrites package stuch
[09:18:48] <marostegui>	 ok!
[09:18:50] <jynus>	 right now it is init.d, in the future it could be systemd
[09:19:13] <marostegui>	 I am not going to upgrade db1063, will leave it running 10.0.29
[09:19:18] <jynus>	 if we did it right, not writing to the main config, we would not need it
[09:19:19] <marostegui>	 too soon to have a master with 10.0.30
[09:19:48] <jynus>	 but not running puppet caused issues in the past (not on dbs)
[09:20:21] <marostegui>	 ok, will do then
[10:07:43] <jynus>	 loooooooooooooooooooool
[10:09:01] <jynus>	 http://s2.quickmeme.com/img/9b/9b3ec2629194a2642105cfc73c1d3abfc7e7f4468871d26725b9dbb53786ca94.jpg
[10:09:56] <wikibugs>	 10DBA, 10AbuseFilter, 06Performance-Team, 13Patch-For-Review: AFComputedVariable::compute query timeouts - https://phabricator.wikimedia.org/T116557#3201018 (10jcrespo) This is madness: ``` root@db2062[enwiki]> EXPLAIN SELECT /* AFComputedVariable::{closure}  */  rev_user_text  FROM `revision` WHERE rev_pa...
[10:12:03] <jynus>	 now what do I do, do I just repool db1063 and wait if it fixed itself automagically?
[10:12:18] <jynus>	 *db2062
[10:20:00] <marostegui>	 mmmm
[10:20:09] <marostegui>	 can you restart mysql on db2062 and see if it happens?
[10:20:24] <jynus>	 do you have more use cases?
[10:20:33] <jynus>	 to test it was not just 1 case
[10:20:52] <marostegui>	 not really, but as I tested db1065 it wasn't filesorting and after a restart it did
[10:20:58] <marostegui>	 let's try the same thing with db2062 maybe
[10:21:32] <jynus>	 it does for rev_page = '17437940'
[10:21:52] <jynus>	 so it is highly dependent on the value and the internal state
[10:22:55] <jynus>	 I will run the ANALYZE AS expected
[10:23:30] <marostegui>	 https://phabricator.wikimedia.org/T163351#3200731
[10:23:36] <marostegui>	 this one filesorts too on db2062
[10:23:40] <jynus>	 yes
[10:23:43] <marostegui>	 yeah, I would say let's run it
[10:29:23] <wikibugs__>	 10DBA, 13Patch-For-Review, 07Performance: Reduce max execution time of interactive queries or a better detection and killing of bad query patterns - https://phabricator.wikimedia.org/T160984#3201055 (10jcrespo)
[11:33:42] <wikibugs>	 10DBA, 13Patch-For-Review, 07Performance: Reduce max execution time of interactive queries or a better detection and killing of bad query patterns - https://phabricator.wikimedia.org/T160984#3201116 (10jcrespo) Testing https://gerrit.wikimedia.org/r/346559 on db2062.
[11:36:05] <jynus>	 self reminder: killer events and semisync setup on the new masters
[11:49:50] <jynus>	 I am going to do some fake queries on db2062 for testing purposes
[11:49:55] <marostegui>	 ok
[11:49:59] <jynus>	 with wikiuser / wikiadmin
[11:50:01] <marostegui>	 I am going to analyze table on db1081 
[11:50:14] <marostegui>	 to leave it running the weekend
[11:50:15] <jynus>	 just a heads up it is me but may show up on long running queries monitoring
[11:51:17] <marostegui>	 ok :)
[11:52:03] <jynus>	 it doesn't work :-(
[11:53:52] <marostegui>	 the killer?
[11:55:15] <jynus>	 yeah, will be debugging
[11:55:59] <wikibugs__>	 10DBA, 13Patch-For-Review: Reclone db1063 to become a slave in s5 - https://phabricator.wikimedia.org/T163109#3201137 (10Marostegui) The scope of this ticket is done. db1063 is now replicating as slave **(that means binlog type = MIXED) so that needs to be changed before making it the master.** SSL and GTID ar...
[11:56:07] <wikibugs>	 10DBA, 13Patch-For-Review: Reclone db1063 to become a slave in s5 - https://phabricator.wikimedia.org/T163109#3201138 (10Marostegui) 05Open>03Resolved a:03Marostegui
[11:56:09] <wikibugs__>	 10DBA, 13Patch-For-Review, 05codfw-rollout: Analyze if we want to replace some masters in eqiad while it is not active - https://phabricator.wikimedia.org/T162133#3201140 (10Marostegui)
[11:56:39] <wikibugs>	 10DBA, 13Patch-For-Review, 05codfw-rollout: Analyze if we want to replace some masters in eqiad while it is not active - https://phabricator.wikimedia.org/T162133#3153482 (10Marostegui) db1063 is ready to be the master in s5.  **binlog needs to be changed to STATEMENT**
[11:56:48] <jynus>	 for some reason the event scheduler was disabled there
[11:57:13] <marostegui>	 on db2062? how come? we haven't touched it for months
[11:57:32] <jynus>	 I know
[11:57:53] <jynus>	 we need to check with cumin what is the state of that variable
[11:58:14] * volans at your service :)
[11:58:22] <marostegui>	 xddd
[11:58:27] <volans>	 what do you need?
[11:58:40] <jynus>	 it works perfectly now
[11:58:47] <marostegui>	 volans: do you have a higlight for the word cumin? XD
[11:59:14] <volans>	 marostegui: you'll never know ;)
[12:05:43] <jynus>	 it was on everywhere except on db2062 (already fixed)
[12:05:48] <jynus>	 and on db1041
[12:08:04] <hoo>	 jynus: Can you tell how much an index is used?
[12:08:23] <hoo>	 Because we have at least one on wb_terms (yes, that table) which is probably totally useless
[12:08:40] <hoo>	 It's only there in production, not in hte code itself
[12:08:54] <jynus>	 tell me the index and the server
[12:09:23] <hoo>	 wb_terms_entity_type and wb_terms_type (two even)
[12:09:29] <hoo>	 All s5 servers, wikidatawiki
[12:09:34] <jynus>	 table?
[12:09:38] <hoo>	 wb_terms
[12:11:18] <marostegui>	 Speaking of wb_terms, I was thinking about starting with: https://phabricator.wikimedia.org/T162539
[12:11:38] <marostegui>	 in eqiad so we can get it done by next week on production
[12:12:14] <hoo>	 That would be really appreciated
[12:13:18] <jynus>	 https://phabricator.wikimedia.org/P5306
[12:15:05] <hoo>	 jynus: tl;dr they're entirely unused?
[12:16:01] <jynus>	 https://phabricator.wikimedia.org/P5306#28415
[12:16:06] <jynus>	 as far as I can see
[12:18:26] <hoo>	 There are two values in wb_terms_entity_type and three in wb_terms_type
[12:18:41] <hoo>	 Just to show /how/ useless these are
[12:18:50] <hoo>	 I mean distinct values
[12:19:04] <jynus>	 yeah
[12:19:15] <jynus>	 I also see an unused wikidatawiki  | wb_terms    | tmp1
[12:19:25] <hoo>	 The whole table is just a big mess honestly
[12:19:54] <hoo>	 tmp1 should really be used on servers that get "regular" traffic
[12:21:26] <hoo>	 Shall I open a ticket for dropping wb_terms_entity_type and wb_terms_type?
[12:21:44] <hoo>	 They are not in our table definitions and I have no idea where they come from
[12:21:56] <marostegui>	 That would be nice, so we can alter it along with https://phabricator.wikimedia.org/T162539
[12:25:46] <hoo>	 https://phabricator.wikimedia.org/T163548
[12:25:58] <hoo>	 I'll try to see whether this was in Wikibase at some point
[12:26:06] <hoo>	 I really wonder how we screwed up so badly
[12:26:47] <jynus>	 no reason to ask that
[12:26:53] <jynus>	 we can only fix things
[12:27:04] <jynus>	 and look always forward
[12:27:05] <wikibugs>	 07Blocked-on-schema-change, 10Wikidata, 03Wikidata-Sprint: Deploy schema change for adding term_full_entity_id column to wb_terms table - https://phabricator.wikimedia.org/T162539#3201218 (10Marostegui) Let's do this alter along with dropping the indexes on: T163548
[12:27:23] <jynus>	 not need to blame or ask for guily
[12:27:34] <jynus>	 talking about guilty
[12:27:37] <jynus>	 O:-)
[12:27:51] <jynus>	 I filed
[12:28:26] <jynus>	 can you be more specific on T163548?
[12:28:27] <stashbot>	 T163548: Drop the useless wb_terms keys "wb_terms_entity_type" and "wb_terms_type" - https://phabricator.wikimedia.org/T163548
[12:28:49] <jynus>	 which tables and servers: (I do not need the names, just "all production wikidata servers"
[12:28:59] <jynus>	 or something like that
[12:29:17] <jynus>	 oh, I assume it has to be deployed first?
[12:29:52] <jynus>	 anyway, I filed T163544 can you bump that on priority?
[12:29:53] <stashbot>	 T163544: Wikibase\TermSqlIndex::getMatchingTerms , Wikibase\TermSqlIndex::fetchTerms have bad performance after codfw failover - https://phabricator.wikimedia.org/T163544
[12:30:10] <jynus>	 to have something done for the 3 May
[12:30:20] <jynus>	 so we minimize wikidata disruption
[12:33:37] <hoo>	 jynus: I looked into it… and it's not that easy(tm)
[12:33:43] <jynus>	 I know
[12:33:49] <jynus>	 i am not asking for a fix
[12:33:51] <hoo>	 that's actually how I stumbled upon the indexes
[12:33:53] <jynus>	 I am asking for a hack
[12:33:59] <marostegui>	 hoo: can you complete a bit the description of T163548 with jynus's suggestions?
[12:34:00] <stashbot>	 T163548: Drop the useless wb_terms keys "wb_terms_entity_type" and "wb_terms_type" - https://phabricator.wikimedia.org/T163548
[12:34:06] <jynus>	 I said some options like
[12:34:13] <jynus>	 reducing the limit size
[12:34:23] <jynus>	 or reducing the amount of queries done
[12:34:45] <jynus>	 I am not expecting miracles- but the only query that did worse thant those 2
[12:34:58] <hoo>	 marostegui: Updated it some more
[12:35:00] <jynus>	 brought down flow, echo and translations :-)
[12:35:22] <jynus>	 so I do not expect a fix, just something so we avoid the overload
[12:35:40] <wikibugs>	 10DBA, 10Wikidata, 07Schema-change: Drop the useless wb_terms keys "wb_terms_entity_type" and "wb_terms_type" - https://phabricator.wikimedia.org/T163548#3201243 (10hoo)
[12:35:41] <jynus>	 even if it is we disable X for 1 day, whatever it takes
[12:36:00] <hoo>	 Ugh… this is key Wikidata functionality
[12:36:17] <hoo>	 It wont hit on a warmed up server
[12:36:27] <hoo>	 but if the wb_terms indexes are not warm…
[12:37:56] <marostegui>	 hoo: to start dropping those indexes, do we need to wait something from your side? deploy stuff, change tables.sql or something else? 
[12:38:14] <hoo>	 marostegui: No, they're removed from Wikibase since 2014
[12:38:19] <hoo>	 and not used in any queries
[12:38:34] <jynus>	 I am not saying to disable wikidata, it was just an example
[12:38:46] <jynus>	 I am asking you, can you do something about it?
[12:38:48] <marostegui>	 hoo: great!
[12:39:01] <jynus>	 technically, I see 300 second queries
[12:39:04] <hoo>	 jynus: I know, no worries
[12:39:08] <jynus>	 those should not be there
[12:39:09] <hoo>	 but I can't think of anything
[12:39:12] <hoo>	 true
[12:39:29] <jynus>	 I see clear brute force options
[12:39:36] <jynus>	 which is reducing batch sizes
[12:39:47] <hoo>	 If that helps, we can sure do that
[12:39:55] <jynus>	 that is the only thing I am asking
[12:40:03] <hoo>	 *he says w/o checking the code first*
[12:40:06] <jynus>	 to minimize the issues on the next failocer
[12:40:13] <jynus>	 just give it a look please
[12:40:20] <hoo>	 Yeah, looking right now
[12:40:25] <hoo>	 thanks for all your support
[12:40:26] <jynus>	 but it is time-constrained (next failover is 3 May)
[12:41:35] <hoo>	 I need to talk to our PM to see how dirty I can go here… but I'll find a way
[12:44:12] <hoo>	 jynus: For the first query in https://phabricator.wikimedia.org/T163544 how many term_entity_id would be ok?
[12:44:22] <hoo>	 Right now we don't batch there at all, but I can add that rather easily
[12:46:38] <hoo>	 Would 10 be ok?
[12:49:08] <jynus>	 I would say 9
[12:49:18] <jynus>	 I thin 10 is the limit to start getting in some cases bad queries
[12:49:29] <jynus>	 (approximate cardinality)
[12:49:38] <hoo>	 Ok, that's fine
[12:49:41] <jynus>	 in most cases
[12:49:58] <jynus>	 I would assume it is a question of the total number of rows selected
[12:50:15] <jynus>	 a similar thing to page- some pages have thousands of revisions
[12:50:24] <jynus>	 and that slows down some queries
[13:00:26] <hoo>	 jynus: Would it be ok for you, if we were to only temporarily (for the DC switch) apply the fix for the second query in the ticket?
[13:00:41] <hoo>	 These queries should be replaced with elastic soon-ish anywa
[13:00:42] <hoo>	 y
[13:02:23] <wikibugs>	 10DBA, 13Patch-For-Review: codfw: Fix S4 commonswiki.templatelinks partitions - https://phabricator.wikimedia.org/T149079#3201321 (10Marostegui) This was done before the switchover ``` root@db2019.codfw.wmnet[commonswiki]> show create table templatelinks; +---------------+--------------------------------------...
[13:02:32] <wikibugs>	 10DBA, 13Patch-For-Review: codfw: Fix S4 commonswiki.templatelinks partitions - https://phabricator.wikimedia.org/T149079#3201323 (10Marostegui) 05Open>03Resolved a:03Marostegui
[13:08:18] <marostegui>	 i am doing somethin j*nus will love…cleaning up screen on neodymium, I think I removed like 8 or 9
[13:12:45] <wikibugs__>	 10DBA, 10Wikidata, 07Schema-change: Drop the useless wb_terms keys "wb_terms_entity_type" and "wb_terms_type" on "wb_terms" table - https://phabricator.wikimedia.org/T163548#3201342 (10Marostegui) a:03Marostegui
[14:28:07] <wikibugs>	 10DBA, 13Patch-For-Review: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416#3201504 (10Marostegui) db1067 is done: ``` root@neodymium:~# mysql -hdb1067 -A enwiki -e "show create table revision\G" --skip-ssl *************************** 1. row **...
[14:29:21] <wikibugs__>	 10DBA, 06Operations, 10ops-codfw: pdu phase inbalances: ps1-a3-codfw, ps1-c6-codfw, & ps1-d6-codfw - https://phabricator.wikimedia.org/T163339#3201510 (10Papaul) @Marostegui Anytime Monday at 9:30am works for me.
[14:30:45] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: pdu phase inbalances: ps1-a3-codfw, ps1-c6-codfw, & ps1-d6-codfw - https://phabricator.wikimedia.org/T163339#3201514 (10Marostegui) >>! In T163339#3201510, @Papaul wrote: > @Marostegui Anytime Monday at 9:30am works for me.   Let's do it Monday at 9:30AM then.
[14:38:30] <marostegui>	 jynus: you were only playing with db2062 for the query testing or with db2055 too?
[14:38:44] <jynus>	 only 62 because it is depooled
[14:38:48] <marostegui>	 ok: https://tendril.wikimedia.org/activity?research=0&labsusers=0
[14:38:50] <marostegui>	 check the first query
[14:38:53] <marostegui>	 :|
[14:39:09] <jynus>	 use https://tendril.wikimedia.org/activity?wikiadmin=0&research=0&labsusers=0
[14:39:26] <jynus>	 wikiadmin are queries only happening on vslow hosts
[14:39:32] <jynus>	 (mostly)
[14:39:45] <marostegui>	 ah :)
[14:39:45] <marostegui>	 pheeew
[14:39:53] <jynus>	 we let those run because only one runs per week
[14:39:57] <jynus>	 and only on those hosts
[14:40:09] * marostegui breaths normally again
[14:40:11] <jynus>	 and they have a hard limit of 14 hours
[14:40:14] <jynus>	 *24
[14:40:27] <jynus>	 they cannot be done by users
[14:41:11] <jynus>	 it is not always like that, but simplifying, think wikiuser as webrequests and wikiadmin as internal requests
[14:41:32] <marostegui>	 got it
[14:41:34] <marostegui>	 thanks :)
[14:41:57] <jynus>	 all those limits are on wikiuser because wikiadmin normally doesn create problems
[14:42:10] <jynus>	 however, sometimes someone runs preblematic queries manually
[14:42:17] <jynus>	 and we bark at them :-)
[14:42:20] <marostegui>	 haha
[14:42:30] <jynus>	 from terbium or tin or equivalent
[14:43:41] <jynus>	 I am now going to execute 5000 queries into db2062, don't freak out
[14:43:47] <marostegui>	 xdddddd
[14:44:10] <marostegui>	 I think I am going to log off in a bit to be honest
[14:44:16] <jynus>	 you do well
[14:44:47] <hoo>	 jynus: Is it ok for you if I quote parts of our conversation from ~1h ago in a ticket?
[14:44:54] <jynus>	 sure
[14:45:03] <hoo>	 Thanks
[14:45:11] <jynus>	 as long as I do not sound too idiotic :-)
[14:45:45] <jynus>	 for context- high contention on failover
[14:45:46] <hoo>	 You sure don't
[14:46:27] <jynus>	 worse offenders: language (temp. disabled), core (working on it) and wikidata
[14:46:51] <jynus>	 so more people are working on its respective fields
[16:57:35] <marostegui>	 jynus: https://www.percona.com/blog/2017/04/17/the-mysqlpump-utility/
[16:58:03] <marostegui>	 the parallel processing is a nice one
[16:59:07] <jynus>	 it requires stopping the slave
[16:59:24] <jynus>	 I think it does nothing that mydumper or our scripts don't do
[17:00:41] <jynus>	 what we need is facebook's incremental backups strategy
[17:01:16] <jynus>	 https://code.facebook.com/posts/1007323976059780/continuous-mysql-backup-validation-restoring-backups/
[17:03:20] <marostegui>	 yeah, that is the ideal situation 
[17:04:02] <jynus>	 it is not a lack of technology
[17:04:08] <jynus>	 it is a proper automatization
[17:04:52] <jynus>	 https://www.youtube.com/watch?v=Fe2oLZ4CWD4
[17:08:58] <jynus>	 as dependencies: primary keys everywhere and smaller revision table
[19:54:54] <wikibugs>	 10DBA, 06Operations, 10Wikimedia-General-or-Unknown: Spurious completely empty `image` table row on commonswiki - https://phabricator.wikimedia.org/T155769#3202432 (10jcrespo) >  (I volunteer)  Please don't.