[01:47:45] 10DBA, 10Operations: db2088 rebooted itself and came back sick - https://phabricator.wikimedia.org/T202822 (10Andrew) [01:50:05] 10DBA, 10Operations: db2088 rebooted itself and came back sick - https://phabricator.wikimedia.org/T202822 (10Andrew) I don't see anything in the syslog to warn about the coming crash... it just stops dead at 00:50:01 [05:51:54] 10DBA, 10Operations, 10ops-codfw: db2058: Disk #11 predictive failure - https://phabricator.wikimedia.org/T202798 (10Marostegui) 05Open>03Invalid The disk finally failed, so let's follow up there - T202824 [05:52:29] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2058 - https://phabricator.wikimedia.org/T202824 (10Marostegui) p:05Triage>03Normal a:03Papaul Let's get the disk replaced! Thanks [05:54:30] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2088 rebooted itself and came back sick - https://phabricator.wikimedia.org/T202822 (10Marostegui) p:05Triage>03Normal a:03Papaul Thanks a lot for triaging this @Andrew. HW logs look empty unfortunately but this crash looks really similar to the... [13:09:13] 10DBA, 10Cloud-Services: Prepare and check storage layer for fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T202820 (10Urbanecm)