[06:25:28] 10DBA, 10Patch-For-Review: Drop valid_tag table - https://phabricator.wikimedia.org/T212254 (10Marostegui) There has been no errors since this rename, so I am going to go ahead and start dropping this table. [06:26:18] 10DBA, 10Patch-For-Review: Drop valid_tag table - https://phabricator.wikimedia.org/T212254 (10Marostegui) [06:51:20] 10DBA, 10Patch-For-Review: Drop valid_tag table - https://phabricator.wikimedia.org/T212254 (10Marostegui) [06:54:57] 10DBA, 10Patch-For-Review: Drop valid_tag table - https://phabricator.wikimedia.org/T212254 (10Marostegui) [07:22:47] 10DBA, 10Data-Services: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127 (10Marostegui) [07:39:35] 10DBA, 10CheckUser, 10MediaWiki-Database, 10Patch-For-Review, 10Wikimedia-production-error: DBQueryTimeoutError on Wikimedia Login's Special:CheckUser - https://phabricator.wikimedia.org/T212692 (10Marostegui) 05Open→03Resolved a:03Marostegui And after adding the same index the query works fine: `... [07:44:58] 10DBA, 10CheckUser, 10MediaWiki-Database, 10Patch-For-Review, 10Wikimedia-production-error: DBQueryTimeoutError on Wikimedia Login's Special:CheckUser - https://phabricator.wikimedia.org/T212692 (10Marostegui) After adding the index: ` root@db1078.eqiad.wmnet[loginwiki]> SELECT log_timestamp FROM `log... [07:52:09] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) enwiki has `KEY tmp_1 (rc_this_oldid),` which I am removing as part of this schema change so we only leave: ` KEY... [07:52:21] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [07:52:23] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) [08:12:15] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) s1 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] dbstore1001 [] db1124 [] db1119 [] db1118 [] db1114 []... [08:12:20] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) s1 eqiad progress [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1002 [] dbstore1001 [] db1124 [] db1119 []... [08:23:43] 10DBA, 10Data-Services: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127 (10Peachey88) [08:30:44] 10DBA: replication broken on db1124:3318 on wikidata.pagelinks - https://phabricator.wikimedia.org/T209521 (10Marostegui) [08:30:47] 10DBA: Recheck wikidatawiki.pagelinks table across all hosts in s8 - https://phabricator.wikimedia.org/T212574 (10Marostegui) 05Open→03Resolved This is all done: ` Execution ended, no differences found. ` I have also run a full check for: db1087 vs db1071 (master) db1092 db1099:3318 db1101:3318 db1104 db110... [08:36:49] 10DBA: Recheck wikidatawiki.pagelinks table across all hosts in s8 - https://phabricator.wikimedia.org/T212574 (10Banyek) awesome! [08:54:53] 10DBA, 10Patch-For-Review: Drop valid_tag table - https://phabricator.wikimedia.org/T212254 (10Marostegui) [09:15:54] 10Blocked-on-schema-change, 10DBA, 10MediaWiki-Database, 10Scoring-platform-team, and 2 others: Schema change for rc_this_oldid index - https://phabricator.wikimedia.org/T202167 (10Marostegui) [09:16:03] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping page.page_counter on wmf databases - https://phabricator.wikimedia.org/T86338 (10Marostegui) [10:54:50] banyek: remember to check if the backup for s7 is still running on codfw before running the alter on the master (it was running a couple of hours ago) [10:55:20] 👍 definetely, thanks [10:55:46] dbstore2001 still lags at s7 [10:55:55] I'll wait for it [10:55:57] check the backup table on zarcillo [10:56:05] maybe it is still lagging but the backup is done [10:56:19] just double check the table to see if it is ongoing still [10:56:34] a quick glance at tendril activity shows it is still running - just checked [10:57:07] Ok, I'll check it again after lunch, maybe it finishes until [10:57:56] last week it took around 7h [10:58:06] today it started at 2019-01-02 05:46:02 [10:58:17] 2018-12-26 07:43:50 | 2018-12-26 14:37:56 [10:58:21] ^ that is last week [10:58:39] so I guess it will be finished around 12:30 UTC [10:58:43] (Rough guess) [10:59:25] Well, we'll see it soon [10:59:35] you can of course proceeed with dbstore1002 and the other backup hosts in eqiad if they are done already [11:00:33] I'll keep doing codfw first [11:00:49] sure [11:01:10] I have a plenty of emails to check, and I'll read my ticket's until lunch, I want to clean up my backlog ;) [11:01:14] :) [13:49:04] 10DBA, 10Anti-Harassment, 10CheckUser, 10Stewards-and-global-tools, and 3 others: Strategy for how to change the "cu_changes" DB schema - https://phabricator.wikimedia.org/T212781 (10Rxy) p:05Triage→03Normal [13:51:34] 10DBA, 10Anti-Harassment, 10CheckUser, 10Stewards-and-global-tools, and 3 others: Strategy for how to change the "cu_changes" DB schema - https://phabricator.wikimedia.org/T212781 (10Rxy) [13:52:06] 10DBA, 10Anti-Harassment, 10CheckUser, 10Stewards-and-global-tools, and 3 others: Strategy for how to change the "cu_changes" DB schema - https://phabricator.wikimedia.org/T212781 (10Rxy) [13:52:24] 10DBA, 10Anti-Harassment, 10CheckUser, 10Stewards-and-global-tools, and 3 others: Strategy for how to change the "cu_changes" DB schema - https://phabricator.wikimedia.org/T212781 (10Rxy) [13:55:38] 10DBA, 10Anti-Harassment, 10CheckUser, 10Stewards-and-global-tools, and 3 others: Strategy for how to change the "cu_changes" DB schema - https://phabricator.wikimedia.org/T212781 (10Marostegui) What do you mean with a "single DB cluster"? I don't know much about the `cu_changes` table, but as far as I kno... [14:17:26] 10DBA, 10Anti-Harassment, 10CheckUser, 10Stewards-and-global-tools, and 3 others: Strategy for how to change the "cu_changes" DB schema - https://phabricator.wikimedia.org/T212781 (10Rxy) >>! In T212781#4848941, @Marostegui wrote: > What do you mean with a "single DB cluster"? I don't know much about the `... [14:19:55] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [14:27:28] 10DBA, 10Anti-Harassment, 10CheckUser, 10Stewards-and-global-tools, and 3 others: Strategy for how to change the "cu_changes" DB schema - https://phabricator.wikimedia.org/T212781 (10Marostegui) Thanks for clarifying that! So one big table holding the data for all the wikis is probably not a good idea as y... [14:42:35] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10bmansurov) [15:14:20] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [15:25:12] 10DBA, 10Anti-Harassment, 10CheckUser, 10Stewards-and-global-tools, and 3 others: Strategy for how to change the "cu_changes" DB schema - https://phabricator.wikimedia.org/T212781 (10Bawolff) > So one big table holding the data for all the wikis is probably not a good idea as you'd be creating just a big m... [15:32:22] 10DBA, 10Anti-Harassment, 10CheckUser, 10Stewards-and-global-tools, and 3 others: Strategy for how to change the "cu_changes" DB schema - https://phabricator.wikimedia.org/T212781 (10Marostegui) >>! In T212781#4849240, @Bawolff wrote: >> So one big table holding the data for all the wikis is probably not a... [15:43:14] 10DBA, 10Anti-Harassment, 10CheckUser, 10Stewards-and-global-tools, and 3 others: Strategy for how to change the "cu_changes" DB schema - https://phabricator.wikimedia.org/T212781 (10Rxy) Thanks for advice!. I'll just implement with plan 3. then if we can't wait to show the result, I'll switch to Plan 1.... [15:43:38] 10DBA, 10Anti-Harassment, 10CheckUser, 10Stewards-and-global-tools, and 4 others: Strategy for how to change the "cu_changes" DB schema - https://phabricator.wikimedia.org/T212781 (10Rxy) 05Open→03Resolved a:03Rxy [15:58:22] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10bmansurov) @Banyek I've increased the MySQL connection limit to 400 in the above patch. Does tha... [16:06:46] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10Banyek) @bmansurov as I see the memory usage on the m2 hosts, this could be good, but I am not s... [16:10:43] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Ladsgroup) [16:10:45] 10DBA, 10Patch-For-Review: Drop tag_summary table - https://phabricator.wikimedia.org/T212255 (10Ladsgroup) 05Stalled→03Open wmf.9 is everywhere \o/ [16:13:15] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10bmansurov) @Banyek thanks! For the context I requested the initial number to be 50 per {T205294}... [16:21:12] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10Banyek) >>! In T212154#4849377, @bmansurov wrote: > @Banyek thanks! For the context I requested... [16:24:07] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10bmansurov) @Banyek it's coming from T212154#4828960. Given we're having two [[ https://gerrit.wi... [16:25:58] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10Banyek) That makes sense, then I'll add a +1 and and LGTM to the patchm, but I won't mind, if yo... [16:28:56] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [16:34:58] bye for today! [16:37:13] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10bmansurov) @Banyek thanks. I'll wait for a comment from @Marostegui or @jcrespo. [16:39:11] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10Marostegui) 400 is way too much. The host itself has a 500 connections limit for everything. I d... [16:41:49] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10bmansurov) @mobrovac, @Pchelolo can we reduce 192 to something smaller? Maybe disable the servic... [16:53:16] 10DBA, 10Operations, 10Patch-For-Review: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases - https://phabricator.wikimedia.org/T162070 (10Dzahn) [16:57:33] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10Pchelolo) As a absolute minimum we'd need at least `4 + 6 + 2 == 12` connections just for the au... [18:58:00] 10DBA, 10Recommendation-API, 10Research, 10Core Platform Team Backlog (Watching / External), and 2 others: Recommendation API exceeds max_user_connections in MySQL - https://phabricator.wikimedia.org/T212154 (10bmansurov) @Marostegui does 50 sound good to you? [19:41:30] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Krinkle) > For instance, in recent changes patrolling when a change is marked as "patrolled", we can assume it is also "non dama... [19:43:15] 10DBA, 10Operations, 10Patch-For-Review: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases - https://phabricator.wikimedia.org/T162070 (10Dzahn) The only remaining uses of the mysql module are now limited to the statistics module and wikimetrics which i think are maintai... [19:44:08] 10DBA, 10Analytics, 10Analytics-Cluster, 10Operations: Cleanup or remove mysql puppet module; repurpose mariadb module to cover misc use cases - https://phabricator.wikimedia.org/T162070 (10Dzahn) [19:46:51] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4850136, @Krinkle wrote: >> For instance, in recent changes patrolling when a change is marked as "patrol... [19:56:49] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Krinkle) > The benefit of these integrations is that it will allow work deduplication. I would very much like the patrolling w... [20:09:41] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4850213, @Krinkle wrote: > >> The benefit of these integrations is that it will allow work deduplication... [20:16:46] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) [20:48:03] 10DBA, 10User-Banyek, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for ptwikipedia - https://phabricator.wikimedia.org/T211544 (10Andrew) [20:50:08] 10DBA, 10User-Banyek, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for ptwikipedia - https://phabricator.wikimedia.org/T211544 (10Zoranzoki21)