[00:09:34] <wikibugs_>	 10DBA, 10Operations: db1098 crashed and got rebooted - https://phabricator.wikimedia.org/T193331#4166266 (10Volans) I've downtimed db1098 on Icinga until Wed mid EU day and disabled notifications.
[00:15:49] <wikibugs_>	 10DBA, 10Operations: db1098 crashed and got rebooted - https://phabricator.wikimedia.org/T193331#4166270 (10Marostegui) This is the same error as db2081 earlier today: T193325 ```  The Intel Management Engine has recovered the ability to utilize the PECI over DMI facility.  If the PWR2262 "internal system erro...
[00:19:12] <wikibugs_>	 10DBA, 10Operations: db1098 crashed and got rebooted - https://phabricator.wikimedia.org/T193331#4166274 (10Marostegui) T175973#3615656 db1100 suffered it too which is the same batch as db1098
[00:19:29] <wikibugs_>	 10DBA, 10Operations: db1098 crashed and got rebooted - https://phabricator.wikimedia.org/T193331#4166276 (10jcrespo) ```      2018-04-28T23:28:04-0500 LOG007  The previous log entry was repeated 1 times.          2018-04-29T00:13:43-0500 SYS1003  System CPU Resetting.          2018-04-29T00:13:42-0500 SYS1000...
[06:45:15] <wikibugs_>	 10DBA, 10Operations: db1098 crashed and got rebooted - https://phabricator.wikimedia.org/T193331#4166362 (10Marostegui) a:03Cmjohnson @Cmjohnson can we do the same thing we did to db1100? (which had never had another crash ever since):  - Check if there are BIOS/firmware updates available - Power drain the h...
[06:45:39] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad: db1098 crashed and got rebooted - https://phabricator.wikimedia.org/T193331#4166365 (10Marostegui)
[07:01:38] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1098 crashed and got rebooted - https://phabricator.wikimedia.org/T193331#4166371 (10Marostegui) I have started MySQL on db1098 to:  - Make sure nothing is corrupted and replication can flow - Avoid leaving the host to fall behind replication for 2 da...
[11:35:59] <wikibugs_>	 10DBA, 10Operations, 10ops-eqiad, 10Patch-For-Review: db1098 crashed and got rebooted - https://phabricator.wikimedia.org/T193331#4166479 (10Marostegui) As a side note. Either db1098 (T193331), db2081 (T193325) and db1100 (T175973) (they were all coming from the same batch of purchases (T162159 and T162233...
[11:36:07] <wikibugs_>	 10DBA, 10Operations, 10ops-codfw: db2081 crashed/rebooted, probably due to hardware failure - https://phabricator.wikimedia.org/T193325#4166483 (10Marostegui) As a side note. Either db1098 (T193331), db2081 (T193325) and db1100 (T175973) (they were all coming from the same batch of purchases (T162159 and T16...
[18:26:39] <wikibugs_>	 10DBA, 10Patch-For-Review: Test MySQL 8.0 with production data and evaluate its fit for WMF databases - https://phabricator.wikimedia.org/T193226#4166726 (10jcrespo) pymysql version 0.8 or later is required as with the default collation, a newly added one, an exception is rised on connection. It may require a...