[06:47:20] <wikibugs>	 10DBA, 10Operations, 10ops-eqiad: Degraded RAID on db1100 - https://phabricator.wikimedia.org/T241506 (10Marostegui) Any ETA on when the new disk will be ordered? I wouldn't like to leave the primary database master for s5 with a broken disk for long. If another disk on the same span fails, the master will g...
[06:48:05] <wikibugs>	 10DBA: Remove partitions from revision table - https://phabricator.wikimedia.org/T239453 (10Marostegui)
[06:56:18] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Upgrade BIOS and firmware on db2084 - https://phabricator.wikimedia.org/T241103 (10Marostegui) 05Open→03Resolved a:05Marostegui→03Papaul >>! In T241103#5774374, @jcrespo wrote: > I have started mysql instances back again, and replication, as on codfw there is low loa...
[06:56:20] <wikibugs>	 10DBA, 10Operations: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240 (10Marostegui)
[06:56:47] <wikibugs>	 10DBA, 10Operations: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240 (10Marostegui)
[07:10:42] <wikibugs>	 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui)
[07:25:00] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui)
[07:31:55] <wikibugs>	 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui)
[07:34:32] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) s8 eqiad progress  [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1005 [] db1126 [] db1124 [] db1116 [] db1109 [] db1104 [...
[08:21:11] <wikibugs>	 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Marostegui) a:03Marostegui
[08:23:50] <wikibugs>	 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui)
[09:35:53] <wikibugs>	 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui)
[13:56:46] <wikibugs>	 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui)
[14:21:14] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui)
[14:35:47] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Upgrade BIOS and firmware on db2076 - https://phabricator.wikimedia.org/T241647 (10Marostegui) MySQL stopped and host powered off. Ready for @Papaul to act on it. Thank you!
[14:44:18] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui)
[15:01:35] <wikibugs>	 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) s7 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1003 [] db1136 [] db1125 [] db1116 [] db1101 [] db1098 []...
[15:07:31] <wikibugs>	 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui)
[16:19:47] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Upgrade BIOS and firmware on db2076 - https://phabricator.wikimedia.org/T241647 (10Papaul) a:05Papaul→03Marostegui Before  BIOS Version  2.4.3 Firmware Version  2.40.40.40  After  BIOS Version  2.11.0 Firmware Version  2.70.70.70  @Marostegui complete
[17:21:18] <wikibugs>	 10DBA, 10Operations, 10ops-codfw: Upgrade BIOS and firmware on db2076 - https://phabricator.wikimedia.org/T241647 (10Marostegui) 05Open→03Resolved a:05Marostegui→03Papaul Thank you Papaul, I have started MySQL. I will repool the host once it has caught up! Thanks again
[17:21:20] <wikibugs>	 10DBA, 10Operations: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240 (10Marostegui)
[17:21:31] <wikibugs>	 10DBA, 10Operations: Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 - https://phabricator.wikimedia.org/T216240 (10Marostegui)
[17:22:59] <wikibugs>	 10DBA: Remove partitions from revision table - https://phabricator.wikimedia.org/T239453 (10Marostegui)
[20:19:05] <wikibugs>	 10DBA, 10cloud-services-team (Kanban): DB planning: include a writeable (?) misc DB cluster in codfw for WMCS - https://phabricator.wikimedia.org/T218570 (10Marostegui) Resuming the conversation here from {T242036}  >>! In T242036#5783634, @bd808 wrote: > The discussion has been silent because I was under the...
[23:40:20] <wikibugs>	 10DBA, 10Operations: backup2001 crashed 2019-12-08 - https://phabricator.wikimedia.org/T240177 (10Papaul) @jcrespo the last time the system crashed we just upgrade the IDRAC and not the BIOS. The system BIOS version is at 1.37 right now or the new BISO version for the server is 2.4.8 and according the the Dell...