[06:16:37] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.32-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), and 2 others: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273 (10Marostegui) 05stalled>03declined Let's decline this then (feel free to reopen if you thin... [06:21:20] 10DBA, 10MediaWiki-Database, 10Performance-Team (Radar), 10User-Marostegui: Replace parsercache keys to something more meaningful on db-XXXX.php - https://phabricator.wikimedia.org/T210725 (10Marostegui) [06:21:26] 10DBA, 10Operations, 10Patch-For-Review, 10codfw-rollout: [RFC] improve parsercache replication and sharding handling - https://phabricator.wikimedia.org/T133523 (10Marostegui) [06:23:50] 10DBA, 10Operations, 10decommission, 10ops-codfw: Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return) - https://phabricator.wikimedia.org/T209858 (10Marostegui) [07:15:41] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [07:15:48] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) [07:23:58] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) [07:24:05] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [08:08:28] 10DBA, 10MediaWiki-API, 10Patch-For-Review, 10Performance: plwiki API request is excessively slow when including a badrevid - https://phabricator.wikimedia.org/T140302 (10Marostegui) What should we do with this bug? Should https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/299018/ get some more reviewers... [08:08:36] 10DBA, 10MediaWiki-API, 10Patch-For-Review, 10Performance: plwiki API request is excessively slow when including a badrevid - https://phabricator.wikimedia.org/T140302 (10Marostegui) p:05Triage>03Normal [08:40:38] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) [08:40:44] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [08:41:04] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) s4 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1125 [] db1121 [] db1103 [] db1102 [] db1097 [] db10... [08:41:10] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) s4 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] db1125 [] db1121 [] db1103 [] db110... [08:41:22] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) [08:41:37] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [09:59:19] marostegui: hey, next week lots of changes to API database query patterns go live (Changing queries to use change_tag table instead of tag_summary) It should actually improve the performance but if something happened. Tell me [09:59:25] FYI mostly [09:59:50] Amir1: Is that a good idea considering how close we are to the Christmas holidays? [10:00:07] Amir1: Please monitor graphs closely [10:00:20] marostegui: it's part of the train *shrugs* [10:00:24] Sure thing [10:01:09] Amir1: Yeah, but "lots of changes to API database query patterns" sounds like something we have to worry about [10:01:46] When will that be released? [10:01:54] wmf.8 train [10:02:06] Tuesday 18th? [10:02:17] https://phabricator.wikimedia.org/T206662 [10:02:25] to group0 (test wikis) [10:02:36] then group1 at Wednesday [10:02:52] I think group1 would be the best to check as wikidata and commons are there [10:03:03] Yeah, group 0 will not make any difference really [10:03:08] but please monitor _closely_ [10:03:31] I am sure you've done your tests and all that, but we know that when we hit production data things might not go as planned [10:03:39] I will do my part but if something also happened there, it might be related [10:03:59] more of "where to look if things exploded" [10:04:39] If we are changing query patterns I would expect more load on the DBs, because of unexpected query plans and all that [10:04:42] Is it easy to revert? [10:04:54] as I said, it actually improves performance (that already happened in LogPager and UserContribs and RC itself) [10:05:14] I do trust you, I just don't trust the MySQL optimizer _) [10:05:16] :) [10:05:23] No one does :D [10:05:29] I prefer to be careful [10:05:31] and neither should [10:05:41] marostegui: yup, let me grab the gerrit patches [10:06:02] When we start hitting production we certainly go into the unknown [10:06:22] We have seen that unexpected queries plans multiple times [10:06:30] So better to be ready :) [10:07:06] And as we are close to Christmas holidays, coverage will be reduced, that's my worry :-) [10:08:27] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/473617 https://gerrit.wikimedia.org/r/c/mediawiki/core/+/476059 https://gerrit.wikimedia.org/r/c/mediawiki/core/+/476069 https://gerrit.wikimedia.org/r/c/mediawiki/core/+/476058 [10:08:38] marostegui: each one of them is for a different funcationality [10:08:51] one is for ApiQuery, one is for Special:Undelete [10:09:41] So you'll stop hitting tag_summary and instead go to change_tag? [10:16:12] marostegui: yup and given that change_tag is normalized and being used in lots of places already it would read it from memory [10:16:40] and stop loading tag_summary into memory (which we will drop it soon (TM)) [10:18:34] that is great [10:18:44] let's hope the new queries will behave as expected [10:33:18] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 2 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Banyek) In https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/477503/ I have a proposal about how to install... [10:43:11] 10DBA, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10MW-1.32-notes (WMF-deploy-2018-05-22 (1.32.0-wmf.5)), and 2 others: Clean up indexes of wb_terms table - https://phabricator.wikimedia.org/T194273 (10Lucas_Werkmeister_WMDE) [12:44:05] marostegui: how much disk does commonswiki have free? [12:51:26] And could we load a copy of wb_terms onto one of those test servers? [13:01:07] addshore: the master has 3.3T and has used 1.9T [13:01:27] addshore: we could copy wb_terms onto one of those, but it will take some time (even more than a day) [13:02:58] marostegui: do we need a ticket for the copy? [13:03:22] We want to experiment and see how much space setting up a normalized version would take [13:03:26] Yeah, if you can that'd be nice [13:25:42] Can do [13:26:39] cheers [14:51:36] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Banyek) @Milimetric could you be a little bit more specific please? As I understood currently you run the querie... [14:52:02] 10DBA, 10Operations, 10Availability (MediaWiki-MultiDC), 10Performance-Team (Radar): Investigate solutions for MySQL connection pooling - https://phabricator.wikimedia.org/T196378 (10Imarlier) @jcrespo Why would we need to deploy Mediawiki in order to repoint when the master is switched? Wouldn't the prox... [15:10:33] 10DBA, 10Operations, 10Availability (MediaWiki-MultiDC), 10Performance-Team (Radar): Investigate solutions for MySQL connection pooling - https://phabricator.wikimedia.org/T196378 (10Marostegui) >>! In T196378#4797793, @Imarlier wrote: > @jcrespo Why would we need to deploy Mediawiki in order to repoint wh... [15:48:57] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Milimetric) Thanks to @Banyek for the questions and a talk we just had over hangouts, we decided to go in two di... [15:49:04] 10DBA, 10MediaWiki-API, 10MW-1.33-notes (1.33.0-wmf.8; 2018-12-11), 10Patch-For-Review, 10Performance: plwiki API request is excessively slow when including a badrevid - https://phabricator.wikimedia.org/T140302 (10Anomie) 05Open>03Resolved a:03Anomie This should be resolved in 1.33.0-wmf.8. [15:59:47] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Milimetric) 2. exact query for a materialized view that would allow us to import the revision table into Hadoop.... [16:55:53] 10DBA, 10Patch-For-Review, 10cloud-services-team (Kanban): cloudvps: dedicated openstack database - https://phabricator.wikimedia.org/T202889 (10Andrew) My current understanding is: 1) db1073 is already only being used for WMCS uses 2) The recurring issue with connections is resolved So I'm not sure there'... [16:58:18] 10DBA, 10Patch-For-Review, 10cloud-services-team (Kanban): cloudvps: dedicated openstack database - https://phabricator.wikimedia.org/T202889 (10Marostegui) db1073 currently has wikitech db (but there's another task to follow up on its migration already) [18:06:32] 10DBA, 10Analytics, 10Data-Services, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10elukey) Opened a procurement task for 1 Cloudb replica in T211135. We are not planning to buy two hosts with the following assumption: 1) We import data onc... [18:39:32] 10DBA, 10Gerrit, 10Operations: Convert Gerrit's to use H2 as the database after 2.16 upgrade - https://phabricator.wikimedia.org/T211139 (10Paladox) [18:41:12] 10DBA, 10Gerrit, 10Operations: Convert Gerrit to use H2 as the database after 2.16 upgrade - https://phabricator.wikimedia.org/T211139 (10Paladox) [18:41:20] 10DBA, 10Gerrit, 10Operations: Convert Gerrit to use H2 as the database after 2.16 upgrade - https://phabricator.wikimedia.org/T211139 (10Paladox) [19:42:42] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Bstorm) To sum up discussion with @Banyek this morning: From the Cloud perspective this is a problem with the re... [19:58:46] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Milimetric) +1 to @Bstorm's framing of the problem. I was going to say the same thing, and withdraw my revision... [20:13:00] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Milimetric) And to follow up on my first bullet from before: 1. after more closely looking at our temporary sol... [20:17:35] 10DBA, 10Analytics, 10Data-Services, 10User-Elukey: Hardware for cloud db replicas for analytics usage - https://phabricator.wikimedia.org/T210749 (10Milimetric) The other reason for a single host instead of redundant hosts is this: our only critical use of the box is during the first few days of the month...