[00:13:32] 10DBA, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Banyek: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] - https://phabricator.wikimedia.org/T210478 (10Neil_P._Quinn_WMF) [06:32:06] banyek|away: I am re doing s4 backup, as the backup started yesterday, and the schema change on user table got in the middle so the table definition changed in between the backup run, so it failed [06:34:22] 10DBA, 10Analytics, 10Analytics-Kanban, 10Data-Services, and 3 others: Create materialized views on Wiki Replica hosts for better query performance - https://phabricator.wikimedia.org/T210693 (10Milimetric) the views in question here could be created a little faster than monthly. But still, even every two... [06:36:37] 10DBA, 10Patch-For-Review: db2057 storage crashed - https://phabricator.wikimedia.org/T212275 (10Marostegui) [06:36:43] 10DBA, 10Operations, 10ops-codfw: Upgrade db2057 firmware - https://phabricator.wikimedia.org/T212277 (10Marostegui) 05Open→03Resolved Host repooled Thanks @papaul! [06:40:22] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Marostegui) Sorry for not being clear enough - I mean from a pure DB point of view, the internal of those batches, as in: how many rows per DELETE would be issued on the database :-) Can we manually try to delete one... [06:40:56] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Marostegui) p:05Triage→03Normal [07:38:44] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Marostegui) Right, my question is answered at T198176#4724935 I think with that and your additional suggestion of waiting 30 seconds + maxlag of 3 we should be fine. However, it doesn't hurt to do a first manual run... [08:51:12] marostegui: oh, sorry about that, it happened on codfw? (I think I didn't ran the schema change on the backup host in eqiad yet) [08:51:22] yes, codfw [08:56:54] yeah, I was aware the eqiad backup only [09:52:44] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [10:16:50] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [10:32:59] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [10:35:02] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [10:56:03] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [11:12:51] marostegui: https://phabricator.wikimedia.org/P7935 [11:13:42] maybe try again with verbose? [11:14:03] I guess it was metadata locking? [11:14:31] let's see [11:14:33] check the processlist [11:14:36] whilst running it [11:14:39] to see if you see locks there [11:15:15] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10MarcoAurelio) @Marostegui Unfortunately I'll have to do all this by hand given that the pywiki script fails due to returning an error message which is not an error (T212356). I can find the page with most revisions a... [11:15:38] yeah [11:15:40] ```ERROR 1205: Lock wait timeout exceeded; try restarting transaction``` [11:16:05] 10DBA: Mass bigdeletion scheduled for sr.wikinews - https://phabricator.wikimedia.org/T212346 (10Marostegui) >>! In T212346#4837096, @MarcoAurelio wrote: > @Marostegui Unfortunately I'll have to do all this by hand given that the pywiki script fails due to returning an error message which is not an error (T21235... [11:16:09] There you go [11:16:41] I guess I'll try this again later [11:16:58] at least this not needs de/repooling [11:17:06] *doesn't need [11:17:11] Yeah, give it a couple of tries but if not, some masters are too busy for certain alters [11:17:38] I had the same issue with s4 for the page table [11:17:47] I tried a few times and then left it [11:18:06] Mark it as pending and it can be done once we have a window or a master failover [11:18:41] 👍 [11:18:45] https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1068&var-port=9104&from=1541030400000&to=1543622399999&panelId=25&fullscreen [11:19:13] it's funny how the open tables are increasing [12:20:01] 10DBA, 10Patch-For-Review: Drop tag_summary table - https://phabricator.wikimedia.org/T212255 (10Marostegui) 05Open→03Stalled Stalling until wmf.9 is deployed everywhere: https://tools.wmflabs.org/versions/ [12:20:04] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui) [12:28:48] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [12:28:58] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) [12:30:46] it finally succeeded? [12:33:32] yes [12:33:40] \o/ [12:33:59] and now I'll eat something because I am starving ;) [12:34:23] cool [12:37:55] 10DBA, 10Data-Services, 10Toolforge, 10Tracking: Certain tools users create multiple long running queries that take all memory and/or CPU from labsdb hosts, slowing it down and potentially crashing (tracking) - https://phabricator.wikimedia.org/T119601 (10Marostegui) [12:38:03] 10DBA, 10Data-Services, 10Tool-Quentinv57's-tools, 10Patch-For-Review: quentinv57-tools/tools/globalcontribs.php generates slow/complex SQL queries which impact server performance - https://phabricator.wikimedia.org/T194343 (10Marostegui) 05Open→03Resolved a:03jcrespo >>! In T194343#4788476, @MarcoAu... [14:00:48] marostegui: Fixing wb_terms table is now officially a goal for 2019: https://www.wikidata.org/wiki/Wikidata:Development_plan [14:01:03] It has a fancy name though "Storage infrastructure improvements" [14:01:46] https://media.giphy.com/media/2WxWfiavndgcM/200.gif [14:02:56] I like the blue bar, it will take a full year XDD [14:03:19] marostegui: I may ask you later about that sr.wikinews issue we have at hand, but now I need a siesta [14:03:37] Hauskatze: I will traveling in a bit, we can chat tomorrow if you like [14:03:48] marostegui: sure, no problem [14:04:01] Hauskatze: Have to travel to today to avoid tomorrow's RENFE strike :( [14:04:34] marostegui: yeah [14:05:17] always in the most convenient days ;) [14:05:51] yeah :( [14:06:10] Shouldn't have privatized RENFE [14:06:15] nor ENDESA, etc. [14:09:19] Yeah… [14:14:39] marostegui: have a nice travel and stay safe out there, see you tomorrow deo volente [14:21:27] safe travels [17:24:27] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) @daniel or anyone else from TechCom want to help close this out? Code review is on hold pending DBA review, so I we sho... [17:31:30] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10Marostegui) I cannot really provide more feedback on the code itself apart from what I commented about the queries. I'm not expe... [17:37:01] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10daniel) Moving to the RFC inbox, so TechCom will look at it during the next meeting. Since DBA have approved the plan, TechCom w... [17:46:53] 10DBA, 10Jade, 10Operations, 10TechCom-RFC, and 2 others: Introduce a new namespace for collaborative judgments about wiki entities - https://phabricator.wikimedia.org/T200297 (10awight) >>! In T200297#4838242, @Marostegui wrote: > I cannot really provide more feedback on the code itself apart from what I... [17:52:56] 10DBA, 10Jade, 10Scoring-platform-team: Review real-world query plans and performance for Jade - https://phabricator.wikimedia.org/T212435 (10awight) [17:53:15] 10DBA, 10Jade, 10Scoring-platform-team: Review real-world query plans and performance for Jade - https://phabricator.wikimedia.org/T212435 (10awight) [18:19:26] 10DBA, 10Jade, 10Scoring-platform-team: Review real-world query plans and performance for Jade - https://phabricator.wikimedia.org/T212435 (10Marostegui) p:05Triage→03Normal [19:26:03] 10DBA, 10Jade, 10Scoring-platform-team: Review real-world query plans and performance for Jade - https://phabricator.wikimedia.org/T212435 (10awight) [19:51:26] 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Cmjohnson) the tech came today to swap pc1007 system board and the new (refurbed) board is bad again. This will require another call into Dell and will not be fixed until after the holiday break. [20:23:17] 10DBA, 10Operations, 10ops-eqiad: rack/setup/install pc1007-pc1010 - https://phabricator.wikimedia.org/T207258 (10Marostegui) Terrible! Thanks for the heads up Chris!