[06:14:13] 10DBA: MariaDB killed by systemd with ABRT6 - https://phabricator.wikimedia.org/T209754 (10colewhite) [06:18:42] 10DBA: MariaDB killed by systemd with ABRT6 - https://phabricator.wikimedia.org/T209754 (10colewhite) [06:25:40] <_joe_> O [06:25:55] 10DBA, 10Operations: MariaDB killed by systemd with ABRT6 - https://phabricator.wikimedia.org/T209754 (10Joe) [06:25:57] <_joe_> shdubsh: ping? [06:26:02] hi [06:28:44] 10DBA, 10Operations: MariaDB killed by systemd with ABRT6 - https://phabricator.wikimedia.org/T209754 (10colewhite) The server was depooled: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/474417/ [06:34:03] 10DBA, 10Operations: MariaDB killed by systemd with ABRT6 - https://phabricator.wikimedia.org/T209754 (10Marostegui) Thank you for letting us know Thanks also @Joe for calling me up. We will take it from here :-) [06:37:37] 10DBA, 10Operations: db1078 (s3 candidate master) crashed - https://phabricator.wikimedia.org/T209754 (10Marostegui) p:05Triage>03High [06:42:08] 10DBA, 10Operations: db1078 (s3 candidate master) crashed - https://phabricator.wikimedia.org/T209754 (10Marostegui) MySQL got corrupted - this host needs to be rebuilt. [06:49:00] 10DBA, 10Operations, 10Patch-For-Review: db1078 (s3 candidate master) crashed - https://phabricator.wikimedia.org/T209754 (10Marostegui) I haven't found anything on HW logs that might indicate a HW malfunction [06:55:00] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) >>! In T85757#4752915, @Marostegui wrote: >>>! In T85757#4750588, @Banyek wrote: >>>>! In T85757#4750443, @Marostegui wr... [06:59:19] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) >>! In T85757#4752916, @Marostegui wrote: > Did you just re-started replication on db2095:3316 by doing a skip transacti... [07:03:58] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Marostegui) >>! In T85757#4755381, @Banyek wrote: >>>! In T85757#4752916, @Marostegui wrote: >> Did you just re-started replicat... [07:12:27] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) np, I had a bad feeling about that, that's why I checked if something happened. I had to log more indeed, it was a mista... [07:15:53] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Marostegui) No, replication didn't break again after you added the columns back and the alert recovered - they never got dropped... [07:18:12] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Banyek) oh, good. [07:25:05] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change, 10User-Banyek: Dropping user.user_options on wmf databases - https://phabricator.wikimedia.org/T85757 (10Marostegui) >>! In T85757#4755400, @Banyek wrote: > oh, good. Not too good, that host has now differences with the rest of the... [11:46:21] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10Community-consensus-needed, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Zoranzoki21) [11:46:32] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Zoranzoki21) [11:53:46] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10MarcoAurelio) [11:53:48] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10MarcoAurelio) [13:32:12] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Zoranzoki21) a:05Zoranzoki21>03None [14:34:32] 10DBA, 10Epic, 10Tracking: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui) [14:34:34] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Marostegui) 05Open>03stalled I assume this is not ready to go until T209251 is completed, right? [14:55:59] 10DBA, 10MediaWiki-extensions-FlaggedRevs, 10Wikimedia-Site-requests, 10User-Zoranzoki21: Drop FlaggedRevs tables in database for srwikinews - https://phabricator.wikimedia.org/T209761 (10Zoranzoki21) >>! In T209761#4755695, @Marostegui wrote: > I assume this is not ready to go until T209251 is completed,...