[09:08:36] <wikibugs>	 10DBA, 10Operations: db1082 crashed - https://phabricator.wikimedia.org/T258336 (10RhinosF1) In UK time (UTC+1) today, it went down for 13 mins: > 09:55:58 <icinga-wm> PROBLEM - Host db1082 is DOWN: PING CRITICAL - Packet loss = 100% > 10:03:58 <icinga-wm> PROBLEM - MariaDB Replica IO: s5 on db1124 is CRITICAL...
[09:21:28] <wikibugs>	 10DBA, 10Operations: db1080-95 batch possibly suffering BBU issues - https://phabricator.wikimedia.org/T258386 (10Kormat)
[09:21:31] <wikibugs>	 10DBA, 10Operations: db1082 crashed - https://phabricator.wikimedia.org/T258336 (10Kormat) 05Resolved→03Open Re-opening to track the latest crash.
[09:21:34] <wikibugs>	 10DBA, 10Operations: Refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Kormat)
[09:23:53] <wikibugs>	 10DBA, 10Operations: db1082 crashed - https://phabricator.wikimedia.org/T258336 (10Kormat) ` /system1/log1/record18   Targets   Properties     number=18     severity=Caution     date=07/25/2020     time=08:53     description=Smart Storage Battery has exceeded the maximum amount of devices supported (Battery 1,...
[09:24:18] <wikibugs>	 10DBA, 10Operations: db1082 crashed - https://phabricator.wikimedia.org/T258336 (10Volans) RElated hw logs for this new crash looks the same of last week: ` </>hpiLO-> show /system1/log1/record19  status=0 status_tag=COMMAND COMPLETED Sat Jul 25 09:22:56 2020    /system1/log1/record19   Targets   Properties...
[09:29:09] <wikibugs>	 10DBA, 10Operations: db1082 crashed - https://phabricator.wikimedia.org/T258336 (10Volans) For reference mysqld start logs: ` Jul 25 09:17:37 db1082 mysqld[3356]: 2020-07-25  9:17:37 139825736157440 [Note] /opt/wmf-mariadb101/bin/mysqld (mysqld 10.1.44-MariaDB) starting as process 3356 ... Jul 25 09:17:37 db10...
[12:34:42] <wikibugs>	 10DBA, 10Operations: db1082 crashed - https://phabricator.wikimedia.org/T258336 (10Marostegui) Interestingly the OS still sees the BBU (which is something we've seen sometimes too). `    Battery/Capacitor Count: 1 `  This means that the above crash can still happen again. It would be better if the BBU would be...
[12:34:53] <wikibugs>	 10DBA, 10Operations: Refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) p:05Medium→03High