[06:25:03] 10DBA, 10Operations: backup2001 crashed 2019-12-08 - https://phabricator.wikimedia.org/T240177 (10Marostegui) Thanks for clarifying Papaul. Jaime is off and will be back online the 9th of January [06:34:56] 10DBA: Remove partitions from revision table - https://phabricator.wikimedia.org/T239453 (10Marostegui) [06:58:23] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [07:57:44] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) @Andrew I have executed this alter on `labtestwiki` on clouddb2001-dev [07:58:27] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [08:05:30] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [08:12:40] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [08:18:42] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [08:52:45] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) s1 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [] db1139 [] db1134 [] db1124 [] db1119 [] db1118 [... [17:30:06] 10DBA, 10Operations: backup2001 crashed 2019-12-08 - https://phabricator.wikimedia.org/T240177 (10Papaul) @Marostegui thanks will wait tomorrow the 9th so he can take the server down for the FW upgrade. [17:57:48] 10DBA, 10Phabricator, 10Release-Engineering-Team-TODO, 10Documentation, and 2 others: Prepare a disaster recovery plan for failing over Phabricator - https://phabricator.wikimedia.org/T190572 (10mmodell) I believe we can close this as resolved? [18:30:53] marostegui: you online? [18:31:09] what's up [18:43:02] 10DBA: Remove partitions from revision table - https://phabricator.wikimedia.org/T239453 (10Marostegui) [21:23:01] 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1100 - https://phabricator.wikimedia.org/T241506 (10Jclark-ctr) Drive was ordered should arrive shortly will update when it arrives [21:23:47] 10DBA, 10TechCom-RFC (TechCom-Approved): MediaWiki database policy and/or guidelines (2019) - https://phabricator.wikimedia.org/T220056 (10daniel) [21:24:09] 10DBA, 10TechCom-RFC (TechCom-Approved): MediaWiki database policy and/or guidelines (2019) - https://phabricator.wikimedia.org/T220056 (10kchapman) TechCom approved this RFC on 18 Dec 2019 [21:27:17] 10DBA, 10TechCom-RFC (TechCom-Approved): MediaWiki database policy and/or guidelines (2019) - https://phabricator.wikimedia.org/T220056 (10Krinkle) a:03Krinkle [21:41:04] 10DBA, 10Operations, 10ops-codfw: (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Papaul) [22:13:26] 10DBA, 10TechCom, 10TechCom-RFC (TechCom-Approved): MediaWiki database policy and/or guidelines (2019) - https://phabricator.wikimedia.org/T220056 (10daniel) Putting on the TechCom board for implementation. That is, putting the document in the correct place on the wiki. [22:22:43] 10DBA, 10Operations, 10ops-codfw: (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Papaul)