[03:37:11] 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items, 10Growth-Team, and 2 others: ClearUserWatchlistJob/WatchedItemStore::removeWatchBatchForUser bad database peformance on enwiki and others, causing database lag - https://phabricator.wikimedia.org/T270481 (10MusikAnimal) >>! In T270481#6705544, @Marostegu... [05:41:29] 10DBA: Ensure InnoDB is compressed on the new clouddb hosts - https://phabricator.wikimedia.org/T270473 (10Marostegui) [05:41:47] 10DBA: Ensure InnoDB is compressed on the new clouddb hosts - https://phabricator.wikimedia.org/T270473 (10Marostegui) 05Open→03Resolved All done! [05:53:00] 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items, 10Growth-Team, and 2 others: ClearUserWatchlistJob/WatchedItemStore::removeWatchBatchForUser bad database peformance on enwiki and others, causing database lag - https://phabricator.wikimedia.org/T270481 (10Marostegui) Thank you - let's chat there about... [05:58:18] 10DBA, 10Patch-For-Review: Productionize x2 databases - https://phabricator.wikimedia.org/T269324 (10Marostegui) Excellent - thank you Chris :-) [08:02:28] 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items, 10Growth-Team, and 2 others: ClearUserWatchlistJob/WatchedItemStore::removeWatchBatchForUser bad database peformance on enwiki and others, causing database lag - https://phabricator.wikimedia.org/T270481 (10Marostegui) Thanks @MusikAnimal for providing v... [09:12:07] 10DBA, 10Data-Persistence-Backup: Drop table profiling from WMF wiki mariadb servers - https://phabricator.wikimedia.org/T266125 (10Marostegui) [09:16:20] 10DBA, 10Data-Persistence-Backup: Drop table profiling from WMF wiki mariadb servers - https://phabricator.wikimedia.org/T266125 (10Marostegui) [10:00:39] 10DBA, 10CheckUser: Monitor the growth of CheckUser tables at enwiki and few other very large wikis - https://phabricator.wikimedia.org/T267275 (10Marostegui) [10:05:41] 10DBA, 10CheckUser: Monitor the growth of CheckUser tables at enwiki and few other very large wikis - https://phabricator.wikimedia.org/T267275 (10Marostegui) So `enwiki` seems a bit more stable in the last week, but it's grown around 1.3G in 49 days. That means at that rate we are looking at around 9.1G per y... [10:06:44] 10DBA, 10CheckUser: Monitor the growth of CheckUser tables at enwiki and few other very large wikis - https://phabricator.wikimedia.org/T267275 (10Marostegui) Waiting for the s4 backup to be finished to complete the stats [10:09:08] 10DBA, 10CheckUser: Monitor the growth of CheckUser tables at enwiki and few other very large wikis - https://phabricator.wikimedia.org/T267275 (10Marostegui) To confirm T253802#6185761 means that we won't be storing things past 90 days, right? @Urbanecm do you happen to know if those entries would get purged... [10:24:07] 10DBA, 10CheckUser: Monitor the growth of CheckUser tables at enwiki and few other very large wikis - https://phabricator.wikimedia.org/T267275 (10Urbanecm) >>! In T267275#6707804, @Marostegui wrote: > To confirm T253802#6185761 means that we won't be storing things past 90 days, right? @Urbanecm do you happen... [10:49:18] 10DBA, 10CheckUser: Monitor the growth of CheckUser tables at enwiki and few other very large wikis - https://phabricator.wikimedia.org/T267275 (10Marostegui) Excellent, thank you for the fast response [11:02:04] 10DBA, 10CheckUser: Monitor the growth of CheckUser tables at enwiki and few other very large wikis - https://phabricator.wikimedia.org/T267275 (10Marostegui) [11:02:29] 10DBA, 10CheckUser: Monitor the growth of CheckUser tables at enwiki and few other very large wikis - https://phabricator.wikimedia.org/T267275 (10Marostegui) 05Open→03Resolved Closing this as everything looks stable [11:03:33] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for skrwiktionary - https://phabricator.wikimedia.org/T268458 (10Marostegui) @nskaggs this wiki needs the views too - pinging in case it was missed as there was a bunch of new wiki related tickets lately [11:03:37] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for wawikisource - https://phabricator.wikimedia.org/T269432 (10Marostegui) @nskaggs this wiki needs the views too - pinging in case it was missed as there was a bunch of new wiki related tickets lately [11:05:32] 10DBA, 10Patch-For-Review: Productionize x2 databases - https://phabricator.wikimedia.org/T269324 (10Marostegui) [11:56:00] 10DBA, 10Operations: Refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [11:56:03] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-11-29) rack/setup/install db11[51-76] - https://phabricator.wikimedia.org/T267043 (10Marostegui) [11:58:30] 10DBA, 10Patch-For-Review: Productionize x2 databases - https://phabricator.wikimedia.org/T269324 (10Marostegui) [12:54:36] 10DBA, 10Patch-For-Review: Productionize x2 databases - https://phabricator.wikimedia.org/T269324 (10Marostegui) [14:01:20] elukey: do you happen to know if we need to replicate the new x2 section to dbstore hosts? T212129 essentially [14:01:21] T212129: Move MainStash out of Redis to a simpler multi-dc aware solution - https://phabricator.wikimedia.org/T212129 [14:01:34] elukey: It is not urgent, just let me know once discussed with your team [14:01:38] and we can place it there [14:02:02] It doesn't have any data yet [14:04:41] marostegui: from my point of view, I'd say no, but thanks for asking, I'll ask :) [14:04:59] good, let me know what you all decide in the end [14:05:13] in theory the content of the main-stash is more caching-related, so I don't see people asking for it (last famous words) [14:05:28] also really happy that this is happening, getting rid of redis will be massive [14:05:48] (there is the lock manager use case to solve I know but removing main stash is a big win) [14:06:26] indeed [14:06:36] But asking just in case your team can see something needed there [14:06:43] But yeah, I agree it doesn't really make much sense [18:58:42] 10DBA, 10Community-Tech, 10Expiring-Watchlist-Items, 10Growth-Team, and 3 others: ClearUserWatchlistJob/WatchedItemStore::removeWatchBatchForUser bad database peformance on enwiki and others, causing database lag - https://phabricator.wikimedia.org/T270481 (10MusikAnimal) I don't think there is a need to O... [19:41:07] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1101 - https://phabricator.wikimedia.org/T270571 (10Marostegui) 05Open→03Resolved Not sure who changed the disk, but thank you either John or Chris! ` 19:06:25 <+icinga-wm> RECOVERY - MegaRAID on db1101 is OK: OK: optimal, 1 logical, 10 physical, Write...