[06:50:28] 10DBA, 10Operations: db2091 mysql service stopped running - https://phabricator.wikimedia.org/T224393 (10jijiki) [06:56:23] 10DBA, 10Operations: db2091 mysql service stopped running - https://phabricator.wikimedia.org/T224393 (10jijiki) p:05Triage→03High [07:11:43] 10DBA, 10Operations: db2091 mysql service stopped running - https://phabricator.wikimedia.org/T224393 (10Marostegui) p:05High→03Normal Decreasing to normal, codfw isn't in use at the moment. Thanks for creating the task and working out the depool patch! [08:11:53] 10DBA, 10Operations, 10Patch-For-Review: db2091 rebooted unexpectedly - https://phabricator.wikimedia.org/T224393 (10Volans) [08:31:18] good morning [08:32:54] regarding wb_terms new schema changes and migration plan (context https://phabricator.wikimedia.org/T221764) .. I got informed by addshoreVacation that replication from production to labs (for tools) for the new tables might require some manual change so that those tables are included in replication. [08:34:35] is there a documentation or a place where I can look to know what that change may look like? is this something we have to do (wb_terms team in wmde)? [08:34:35] I have a phab task on our board for this https://phabricator.wikimedia.org/T224403, but it includes as much information as I have on this topic ^^ [08:34:35] appreciate any insights on this :) [08:50:49] alaa_wmde: there is some info here, but I don't know how much they are up-to-date honestly, so wait for a more official answer: https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Wiki_Replicas#Step_1:_sanitization [08:52:01] thank you volans will do some reading starting from there then until I get official answer :) [08:52:44] yw :) [09:17:05] 10DBA, 10Operations, 10Patch-For-Review: db2091 rebooted unexpectedly - https://phabricator.wikimedia.org/T224393 (10jijiki) `-------------------------------------------------------------------------------- SeqNumber = 138 Message ID = LOG007 Category = Audit AgentID = DE Severity... [09:30:53] 10DBA, 10Operations, 10Patch-For-Review: db2091 rebooted unexpectedly - https://phabricator.wikimedia.org/T224393 (10Volans) Related documentation for the most useful messages: - PWR2262 https://www.dell.com/support/manuals/it/it/itbsdt1/dell-opnmang-sw-v8.2/eemi_13g_v1.3-v2/pwr-event-messages?guid=guid-5bc... [09:33:08] 10DBA, 10Operations, 10Patch-For-Review: db2091 rebooted unexpectedly - https://phabricator.wikimedia.org/T224393 (10Volans) Forgot to mention, nothing in syslog or journalctl for MySQL on s2/s4 units. [11:10:55] 10DBA, 10MediaWiki-Database, 10Core Platform Team (Multi-DC (TEC1)), 10Core Platform Team Backlog (Watching / External), and 4 others: FY18/19 TEC1.6 Q4: Replace the usage of GTID_WAIT with pt-heartbeat in MW - https://phabricator.wikimedia.org/T221159 (10mobrovac) [11:24:26] 10DBA, 10MediaWiki-Database, 10Core Platform Team (Multi-DC (TEC1)), 10Core Platform Team Backlog (Watching / External), and 4 others: FY18/19 TEC1.6 Q4: Replace the usage of GTID_WAIT with pt-heartbeat in MW - https://phabricator.wikimedia.org/T221159 (10mobrovac) [19:33:29] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2091 rebooted unexpectedly - https://phabricator.wikimedia.org/T224393 (10Marostegui) @papaul should we contact the vendor with these logs? [20:11:01] 10DBA, 10Operations, 10ops-codfw: db2091 rebooted unexpectedly - https://phabricator.wikimedia.org/T224393 (10Maintenance_bot) [20:38:30] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2091 rebooted unexpectedly - https://phabricator.wikimedia.org/T224393 (10Marostegui) a:03Papaul Also, upgrade firmware and BIOS I guess? [21:10:52] 10DBA, 10Operations, 10ops-codfw: db2091 rebooted unexpectedly - https://phabricator.wikimedia.org/T224393 (10Maintenance_bot)