[08:13:17] 10DBA, 10Performance-Team, 10Reading-Infrastructure-Team-Backlog, 10WikimediaEditorTasks, and 2 others: Performance review of Extension:WikimediaEditorTasks - https://phabricator.wikimedia.org/T218087 (10jcrespo) I briefly scanned https://www.mediawiki.org/wiki/Extension:WikimediaEditorTasks and its subpag... [08:46:04] 10DBA, 10MediaWiki-Database, 10WikimediaEditorTasks, 10Patch-For-Review, 10Reading-Infrastructure-Team-Backlog (Kanban): Choose DB/Cluster for WikimediaEditorTasks tables - https://phabricator.wikimedia.org/T218302 (10Marostegui) We'd need to check out the performance of `SELECT wetede_entity_id FROM wik... [08:55:25] 10DBA, 10MediaWiki-Database, 10WikimediaEditorTasks, 10Patch-For-Review, 10Reading-Infrastructure-Team-Backlog (Kanban): Choose DB/Cluster for WikimediaEditorTasks tables - https://phabricator.wikimedia.org/T218302 (10jcrespo) ` UPDATE wikimedia_editor_tasks_counts SET wetc_count = wetc_count + 1 WHERE w... [08:56:18] Oh, nice catch [08:56:47] I didn't know that task was there, I got pinged on a similar one for performance [08:56:57] if I knew, I would have let you on your own [08:57:15] Why? It is good to have other inputs! [08:57:34] well, we duplicated efforts, not that it is bad per se [08:57:44] I didn't catch that thing you did :) [08:58:07] I got pinged on the performance task, that is why I focused on contention [09:17:50] I am in an author's block, snapshotting is 99% there but there are things failing still, need to think more [09:18:09] what is failing? [09:18:39] the prepare process [09:19:09] but I don't know if it is the state or the code yet [09:19:20] Yeah, I was going to ask if it is code or mariabackup itself [09:19:42] no, I don't think it is the tool, it could be the db or a bug [09:20:02] I will do some manual tests now [09:20:34] One of those things that never arise until properly in production, glad we deployed :) [09:21:18] the funny thing is it worked while not deployed, so either it never worked (well) or it is a new thing [09:34:48] wasn't the only-postprocess an option already? [09:34:52] Aaah but not in config, no? [09:35:17] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/498029/ [09:35:40] yeah, I am asking about that one [09:36:10] but those are not the core issue [09:36:25] Line 256 would allow it already but not in config, no? [09:37:10] it is actually compulsory to only postprocess [09:37:29] the option was on backup_mariadb [09:37:40] but it was not an option on daily_snapshot [09:37:58] ah ok ok [09:38:20] only_postprocess on daily_snapshot == skip transfer.py [09:38:40] so do the same thing with an existing snapshot [09:38:46] transfer.py is the part we actually know it works, there are some issues with the postprocessing [09:38:46] run the prepare, stats and everything? [09:38:51] yeah [09:39:00] gotcha [09:39:08] compress is what takes more for enwiki, for example [09:39:23] the files end up on dbstore/es [09:40:20] but they bail out, need to configure the logging, which exists is not configured properly [09:40:43] so it writes e.g. to /var/log/mariadb-backups [09:41:15] that'd be nice [09:41:17] logging of cumin is bad because of the <===> which cannot be disbled yet [09:41:34] so we get [blob] [blob] [09:43:41] you can see it with 'journalctl | grep mariadb-snapshots' [09:44:13] I think I am going to take a break, have a coffee and think a bit about my strategy [09:44:22] * marostegui checking that command on dbstore1001 [09:44:31] jynus: enjoy the pincho de tortilla [10:39:20] 10DBA: Purge and monitor old metadata for the mariadb backups database - https://phabricator.wikimedia.org/T205627 (10jcrespo) [14:46:31] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2052 - https://phabricator.wikimedia.org/T218776 (10Papaul) a:05Papaul→03Marostegui Disk replacement complete [14:47:00] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2052 - https://phabricator.wikimedia.org/T218776 (10Marostegui) Thanks ` physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SAS, 600 GB, Rebuilding) ` [16:04:44] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2052 - https://phabricator.wikimedia.org/T218776 (10Marostegui) 05Open→03Resolved This is now good! ` logicaldrive 1 (3.3 TB, RAID 1+0, OK) physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, OK) physicaldrive 1I:1:2 (port 1I:box... [16:05:20] 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Marostegui) [16:35:37] 10DBA: Purchase and setup remaining hosts for database backups - https://phabricator.wikimedia.org/T213406 (10Papaul) [16:45:31] 10DBA: Purchase and setup remaining hosts for database backups - https://phabricator.wikimedia.org/T213406 (10jcrespo) [16:45:34] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/deploy dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10jcrespo) [16:45:55] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/deploy codfw dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10jcrespo) [16:46:32] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/deploy codfw dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10jcrespo) Allow me to edit the title to not confuse it with the same task that will be filed for eqiad :-D [16:51:52] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/deploy codfw dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10Papaul) [18:50:55] 10DBA, 10MediaWiki-General-or-Unknown, 10MW-1.33-notes (1.33.0-wmf.3; 2018-11-06), 10Patch-For-Review, and 2 others: [Bug] Update old nonuniformly distributed page_random values - https://phabricator.wikimedia.org/T208909 (10MBinder_WMF) [19:47:57] 10DBA, 10Performance-Team, 10Reading-Infrastructure-Team-Backlog, 10WikimediaEditorTasks, and 2 others: Performance review of Extension:WikimediaEditorTasks - https://phabricator.wikimedia.org/T218087 (10aaron) >>! In T218087#5042864, @jcrespo wrote: > 66 millon rows is a small table, the size itself does... [20:10:44] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/deploy codfw dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10Papaul) [20:35:39] 10DBA, 10Growth-Team, 10StructuredDiscussions, 10Wikimedia-Extension-setup, and 2 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792 (10GTirloni) [20:45:11] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/deploy codfw dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10Papaul) ` papaul@asw-a-codfw# run show interfaces xe-4/0/18 descriptions Interface Admin Link Description xe-4/0/18 up... [20:47:38] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/deploy codfw dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10Papaul) [20:51:19] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/deploy codfw dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10Papaul) @Marostegui @jcrespo all is set at my end RAID 0 for the 2 SSD's and RAID 6 for the 8 other disks don't know who to assign the t... [20:52:36] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/deploy codfw dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10Papaul) Also please don't forget to merge the DHCP and DNS changes. [22:14:51] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/deploy codfw dedicated backup recovery/provisioning hosts - https://phabricator.wikimedia.org/T218336 (10jcrespo) a:05Papaul→03None