[06:16:56] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1100 - https://phabricator.wikimedia.org/T241506 (10Marostegui) Thanks! [06:18:11] 10DBA: Remove partitions from revision table - https://phabricator.wikimedia.org/T239453 (10Marostegui) [07:39:52] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [08:19:25] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [09:49:17] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [12:28:12] 10DBA, 10Operations: backup2001 crashed 2019-12-08 - https://phabricator.wikimedia.org/T240177 (10jcrespo) backup2001 is now down and ready to be done maintenance (no need to ask again). @papaul please, when done, just boot it back up and ping here. Thanks. [12:39:03] https://www.reddit.com/r/mariadb/comments/em36sq/we_are_the_mariadb_engineering_team_ask_the/ [12:39:53] another one? [12:40:01] this is from the corporation [12:40:16] happens in 4 hours [12:56:08] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [13:53:06] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) s3 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [] db1124 [] db1123 [] db1112 [] db1095 [] db1078 [... [13:54:57] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [14:18:25] 10DBA, 10Operations, 10ops-eqiad: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui) [14:19:43] 10DBA, 10Operations, 10ops-codfw: (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Marostegui) @Papaul - reminder the RAID10 is done with 256K (the reminder is just because we recently found old some servers with RAID stripe being set... [14:20:39] 10DBA, 10Operations, 10ops-codfw: (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Marostegui) [15:08:34] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [15:11:42] 10DBA, 10Operations: backup2001 crashed 2019-12-08 - https://phabricator.wikimedia.org/T240177 (10Papaul) 05Open→03Resolved Before ` BIOS Version 1.3.7 iDRAC Firmware Version 3.34.34.34 ` After ` BIOS Version 2.4.8 iDRAC Firmware Version 4.00.00.00 FW upgrade complete. We can resolve this task and re-open... [18:51:12] 10DBA, 10TechCom, 10TechCom-RFC (TechCom-Approved): MediaWiki database policy and/or guidelines (2019) - https://phabricator.wikimedia.org/T220056 (10Krinkle) 05Open→03Resolved Now published at ., and incorporated at 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1100 - https://phabricator.wikimedia.org/T241506 (10Jclark-ctr) @Marostegui Drive has arrives Please PM me on IRC so we can get this swapped