[01:31:05] Hi, bd808: are you here? [01:31:50] oder Steinsplitter? [01:33:26] grumble? [04:01:50] 10DBA: Drop `external_user` from all databases - https://phabricator.wikimedia.org/T184247#3877254 (10demon) p:05Triage>03Normal [06:18:13] 10DBA, 10Operations, 10ops-codfw: db2054: Disk with predictive failure - https://phabricator.wikimedia.org/T183887#3877406 (10Marostegui) 05Open>03Resolved All good now - thank you! ``` root@db2054:~# hpssacli controller all show config Smart Array P420i in Slot 0 (Embedded) (sn: 0014380337FE1C0)... [07:05:06] 10DBA, 10Data-Services, 10Goal, 10Patch-For-Review, 10cloud-services-team (FY2017-18): Migrate all users to new Wiki Replica cluster and decommission old hardware - https://phabricator.wikimedia.org/T142807#3877442 (10Marostegui) Hi! What's the status of labsdb1001 and 1003 regarding its decommissioning... [07:17:36] 10DBA: Drop `external_user` from all databases - https://phabricator.wikimedia.org/T184247#3877445 (10Marostegui) [08:41:00] 10Blocked-on-schema-change, 10DBA, 10Data-Services, 10Dumps-Generation, and 2 others: Schema change for refactored comment storage - https://phabricator.wikimedia.org/T174569#3877489 (10Marostegui) s7 is all done but the master. I will alter s7 master on Monday. Once s7 is done, we only have s5 and s8 pe... [10:15:54] 10DBA: Decommission db1039 - https://phabricator.wikimedia.org/T184262#3877626 (10Marostegui) p:05Triage>03Normal [10:16:13] 10DBA: Decommission db1039 - https://phabricator.wikimedia.org/T184262#3877642 (10Marostegui) I will start with it on Monday [11:07:06] 10DBA: Drop `external_user` from all databases - https://phabricator.wikimedia.org/T184247#3877254 (10Marostegui) Just confirming, they are indeed empty. [12:22:20] 10DBA, 10GlobalRename, 10MediaWiki-extensions-CentralAuth: Renaming is stuck for account Mega809 on eswp - https://phabricator.wikimedia.org/T184263#3877937 (10Aklapper) [14:59:14] did you see 10.1.30 ? [15:30:30] yeah, I saw its release notes a week ago or si [15:30:32] so [15:36:27] 10DBA, 10Goal: Check data consistency across production shards - https://phabricator.wikimedia.org/T183735#3878467 (10Marostegui) [15:36:30] 10DBA, 10Operations, 10Goal, 10Patch-For-Review: Decommission old coredb machines (<=db1050) - https://phabricator.wikimedia.org/T134476#3878468 (10Marostegui) [15:36:32] 10DBA, 10Patch-For-Review: Checksum data on s7 - https://phabricator.wikimedia.org/T163190#3878465 (10Marostegui) 05Open>03Resolved I have fixed all the drifts reported by pt-table-checksum as well as all the ones found by compare.py. It is hard to say this is 100% fixed, but we are in a lot better state t... [16:08:11] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2055 - https://phabricator.wikimedia.org/T184285#3878520 (10Marostegui) a:03Papaul Can we get a new disk for this host? Thanks! [16:16:29] I am going to reboot labsdb1010 [16:16:56] then I will upgrade mysql on labsdb1004 - I cannot reboot the machine because it also has postgresql data [16:17:24] ok :) [16:17:28] I am going to call it a day I think [16:17:56] have a nice weekend! [16:18:02] you too! [16:21:55] enjoy your weekend :) [16:22:34] you too as well! :) [17:57:17] 10DBA, 10Data-Services, 10Goal, 10Patch-For-Review, 10cloud-services-team (FY2017-18): Migrate all users to new Wiki Replica cluster and decommission old hardware - https://phabricator.wikimedia.org/T142807#3878758 (10chasemp) >>! In T142807#3877442, @Marostegui wrote: > Hi! > > What's the status of lab... [17:59:19] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2055 - https://phabricator.wikimedia.org/T184285#3878760 (10Papaul) Dear Mr Papaul Tshibamba, Hewlett Packard Enterprise Reference Number: 5325864400 STATUS: Customer Self Repair Part has been shipped Part/s shipped: 653952-001 Part description: SPS-DRV... [17:59:32] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2055 - https://phabricator.wikimedia.org/T184285#3878761 (10Papaul) p:05High>03Normal [19:43:48] 10DBA, 10Data-Services, 10Goal, 10Patch-For-Review, 10cloud-services-team (FY2017-18): Migrate all users to new Wiki Replica cluster and decommission old hardware - https://phabricator.wikimedia.org/T142807#3878968 (10bd808) >>! In T142807#3877442, @Marostegui wrote: > Hi! > > What's the status of labsd... [19:43:53] 10DBA, 10Data-Services: Create backups of user tables from decommissioned database servers - https://phabricator.wikimedia.org/T183758#3862884 (10bd808) A full archive seems unmanageable from an access control point of view. Releasing any of the data held in a non `*_p` database would require a data audit. I s... [19:54:41] 10DBA, 10Data-Services: Create backups of user tables from decommissioned database servers - https://phabricator.wikimedia.org/T183758#3878999 (10Marostegui) >>! In T183758#3878969, @bd808 wrote: > A full archive seems unmanageable from an access control point of view. Releasing any of the data held in a non `... [20:10:07] 10DBA, 10Data-Services: Create backups of user tables from decommissioned database servers - https://phabricator.wikimedia.org/T183758#3879024 (10bd808) >>! In T183758#3878999, @Marostegui wrote: > Maybe we should start by getting a list users/tools databases present on labsdb1003 obviously excluding core wiki... [20:28:18] 10DBA, 10Data-Services: Create backups of user tables from decommissioned database servers - https://phabricator.wikimedia.org/T183758#3879042 (10Marostegui) Thanks! I am a bit confused because `_p` core databases ie: `enwiki_p` are views: ``` mysql:root@localhost [enwiki_p]> show create table revision\G *****... [21:04:45] 10DBA, 10Data-Services: Create backups of user tables from decommissioned database servers - https://phabricator.wikimedia.org/T183758#3879126 (10bd808) >>! In T183758#3879042, @Marostegui wrote: > My point is, there might be normal user databases without `_p` on its name that are being used too? Yes, but at... [21:08:38] 10DBA, 10Data-Services: Create backups of user tables from decommissioned database servers - https://phabricator.wikimedia.org/T183758#3879127 (10Marostegui) Ah! Gotcha! Missed that point - thanks for clarifying that. We can try to backup the non_p databases you listed then. We need to be aware that labsdb1...