[08:00:57] 10DBA, 10Operations, 10ops-codfw: db2091 rebooted unexpectedly - https://phabricator.wikimedia.org/T224393 (10jcrespo) Power drain and firmware upgrade, please (T216240), at least. [09:11:04] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2035 - https://phabricator.wikimedia.org/T224456 (10Volans) p:05Triage→03Normal [09:16:43] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2035 - https://phabricator.wikimedia.org/T224456 (10jcrespo) a:03Papaul Please change it with a spare when possible. [09:41:46] FYI, installing PHP updates on dbmonitor, tendril will be unavailable for a few seconds [10:01:16] 10DBA, 10Wikimedia-Site-requests: Global rename of Fiona B. → Fiona*: supervision needed - https://phabricator.wikimedia.org/T224348 (10jcrespo) @Anomie Based on your comments on T222224#5182855, I am going to suggest to possibly wait for this big rename on dewiki it it can, if you disagree (e.g. because actor... [11:09:23] 10DBA, 10Wikimedia-Site-requests: Global rename of Fiona B. → Fiona*: supervision needed - https://phabricator.wikimedia.org/T224348 (101997kB) [14:12:25] 10DBA, 10Goal, 10Patch-For-Review: Implement database binary backups into the production infrastructure - https://phabricator.wikimedia.org/T206203 (10jcrespo) I am manually preparing x1 and s7 snapshots, which failed again. Maybe s1 on codfw, too? Giving a general check. [15:43:04] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2035 - https://phabricator.wikimedia.org/T224456 (10Papaul) a:05Papaul→03jcrespo disk replacement complete [15:49:39] 10DBA, 10MediaWiki-Database, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), 10Core Platform Team Kanban (Waiting for Review), 10MW-1.34-notes (1.34.0-wmf.7; 2019-05-28): Slow query ApiQueryRevisions on enwiki - https://phabricator.wikimedia.org/T224017 (10Maintenance_bot) [17:55:27] 10DBA, 10Operations, 10ops-codfw: db2091 rebooted unexpectedly - https://phabricator.wikimedia.org/T224393 (10Papaul) 05Open→03Resolved Power drain and firmware upgrade. Before Firmware Version 2.40.40.40 IP Address(es) 10.193.2.127 iDRAC MAC Address 84:7B:EB:F6:70:58 DNS Domain Name Lifecycle Contr... [18:08:13] 10DBA, 10Operations: db2091 rebooted unexpectedly - https://phabricator.wikimedia.org/T224393 (10Marostegui) 05Resolved→03Open a:05Papaul→03Marostegui Thanks, I will take it from here I am reopening because we still have to do stuff with it (bring mysql up, check data etc) Thanks @Papaul [18:23:18] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install dbproxy200[1-4] - https://phabricator.wikimedia.org/T223492 (10Papaul) [18:24:20] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2035 - https://phabricator.wikimedia.org/T224456 (10Marostegui) 05Open→03Resolved The rebuilt finished, but it is reporting predictive failure. Let's not change it again until it has fully failed (as this host will be decommissioned soonish). Let's kee... [18:24:39] 10DBA, 10Operations: Predictive failures on disk S.M.A.R.T. status - https://phabricator.wikimedia.org/T208323 (10Marostegui) [19:00:48] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240 (10Marostegui) [19:12:00] 10DBA, 10Operations: db2091 rebooted unexpectedly - https://phabricator.wikimedia.org/T224393 (10Marostegui) I am waiting for replication to catch up to start checking data consistency. [19:20:27] 10DBA, 10Operations: Decommission db1061-db1073 - https://phabricator.wikimedia.org/T217396 (10Marostegui) [19:37:14] 10DBA, 10Operations, 10Patch-For-Review: correctable memory errors db1068 (commons primary master database) - https://phabricator.wikimedia.org/T213664 (10Marostegui) For the record, the master failover for this host will be scheduled for the 19th June. [19:45:16] 10DBA: Decommission old coredb machines (<=db2042) - https://phabricator.wikimedia.org/T221533 (10Dzahn) fyi db2035 is shown again as having unhealthy disks (https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=db2035&service=Device+not+healthy+-SMART-) [20:01:00] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install dbproxy200[1-4] - https://phabricator.wikimedia.org/T223492 (10Papaul) [20:23:17] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install dbproxy200[1-4] - https://phabricator.wikimedia.org/T223492 (10Dzahn)