[05:13:59] 10DBA, 10Operations, 10ops-codfw: db2070: Disk about to fail - https://phabricator.wikimedia.org/T167623#3339052 (10Marostegui) [05:14:10] 10DBA, 10Operations, 10ops-codfw: db2070: Disk about to fail - https://phabricator.wikimedia.org/T167623#3339066 (10Marostegui) p:05Triage>03Normal [05:34:04] 10DBA, 10Operations, 10Wikimedia-Site-requests: Renaming Neoalpha: supervision needed - https://phabricator.wikimedia.org/T167597#3339069 (10Marostegui) p:05Triage>03Normal Hi Please ping me or @jcrespo on IRC before doing this so we can monitor the DBs. Also please make sure this is not happening at t... [05:37:57] !log Deploy alter table s4 - labsdb1003 - T166206 [05:37:57] marostegui: Not expecting to hear !log here [05:37:58] T166206: Convert unique keys into primary keys for some wiki tables on s4 - https://phabricator.wikimedia.org/T166206 [05:38:07] gah [05:38:47] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Convert unique keys into primary keys for some wiki tables on s4 - https://phabricator.wikimedia.org/T166206#3339073 (10Marostegui) #cloud-services-team I have started the alter table on labsdb1003 [08:16:55] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: x1 master db1031: Faulty BBU - https://phabricator.wikimedia.org/T166108#3339308 (10Marostegui) p:05Triage>03Normal [08:30:25] 10DBA, 10Patch-For-Review: Migrate parsercache hosts to file per table - https://phabricator.wikimedia.org/T167567#3339355 (10Marostegui) Until we decide how and when we migrate these hosts to file per table (it needs to be done soon) I have left a screen running on pc1004-1006 and pc2004-2006 purging logs eve... [08:41:28] 10DBA, 10Labs, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3339390 (10Marostegui) The import finished during the weekend on the labs hosts and I have configured replication on db1095 for s5, and it is now flowing.... [08:42:05] 10DBA, 10Labs, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3339393 (10Marostegui) [08:43:35] 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1089: update RAID controller firwmare - https://phabricator.wikimedia.org/T166935#3339407 (10Marostegui) As per my chat with @Cmjohnson on Friday this will be done on today (Monday) [09:30:27] 10DBA: DROP OAI-related tables - https://phabricator.wikimedia.org/T139342#3339625 (10Marostegui) [10:29:11] 10DBA, 10Operations, 10Patch-For-Review: Better mysql monitoring for number of connections and processlist strange patterns - https://phabricator.wikimedia.org/T112473#3339848 (10Marostegui) p:05High>03Normal [10:35:28] 10DBA, 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3339865 (10elukey) @Marostegui we decided not to proceed with the BBU replacement, the risk it too high with a little gain. We are ok for the moment to use WriteThrough... [10:38:06] 10DBA, 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3339869 (10Marostegui) >>! In T166141#3339865, @elukey wrote: > @Marostegui we decided not to proceed with the BBU replacement, the risk it too high with a little gain.... [10:47:13] 10DBA, 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3339893 (10elukey) 05Open>03Resolved [10:48:03] 10DBA, 10Analytics-Kanban, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3286435 (10elukey) @Cmjohnson sorry for the extra pings, we don't need anymore the BBU replacement. Thanks a lot anyway! [11:05:26] 10DBA: DROP OAI-related tables - https://phabricator.wikimedia.org/T139342#3339983 (10Marostegui) [11:11:40] 10DBA: DROP OAI-related tables - https://phabricator.wikimedia.org/T139342#3339997 (10Marostegui) a:03Marostegui [11:46:28] 10DBA, 10Operations, 10Wikimedia-Site-requests: Renaming Neoalpha: supervision needed - https://phabricator.wikimedia.org/T167597#3340077 (10revi) 05Open>03Resolved `There are no renames in progress for 95016maphack. They may have already finished.` `(change visibility) 20:33, 12 June 2017 -revi (talk |... [11:47:17] 10DBA, 10Operations, 10Wikimedia-Site-requests: Renaming Neoalpha: supervision needed - https://phabricator.wikimedia.org/T167597#3340093 (10Marostegui) Thanks for the heads up before running it :-) [12:58:19] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2070 - https://phabricator.wikimedia.org/T167667#3340322 (10Marostegui) Please @Papaul change the disk when you can. Thanks! [12:59:16] 10DBA, 10Operations, 10ops-codfw: db2070: Disk about to fail - https://phabricator.wikimedia.org/T167623#3340339 (10Marostegui) 05Open>03declined And the disk finally failed: T167667 So let's close this. [13:01:00] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2070 - https://phabricator.wikimedia.org/T167667#3340352 (10Marostegui) p:05Triage>03Normal [16:58:20] 10DBA, 10Scoring-platform-team, 10artificial-intelligence: [Discuss] Hosting the monthly draft quality dataset on labsDB - https://phabricator.wikimedia.org/T167697#3341466 (10Sumit) [16:58:56] 10DBA, 10Scoring-platform-team, 10draftquality-modeling, 10artificial-intelligence: [Discuss] Hosting the monthly draft quality dataset on labsDB - https://phabricator.wikimedia.org/T167697#3341451 (10Sumit) [18:29:05] 10DBA, 10Labs: Prepare and check storage layer for atjwiki - https://phabricator.wikimedia.org/T167715#3341927 (10Reedy) [18:34:21] 10DBA, 10Scoring-platform-team, 10draftquality-modeling, 10artificial-intelligence: [Discuss] Hosting the monthly draft quality dataset on labsDB - https://phabricator.wikimedia.org/T167697#3341982 (10Aklapper) @Sumit: As this task was tagged with #DBA, could you elaborate [[ https://phabricator.wikimedia.... [19:52:59] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1053 - https://phabricator.wikimedia.org/T151465#3342520 (10RobH)