[05:15:03] 10DBA, 10Operations, 10ops-codfw: (No Need By Date Provided) codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Papaul) [06:19:52] 10DBA: Remove partitions from revision table - https://phabricator.wikimedia.org/T239453 (10Marostegui) [06:26:48] 10DBA, 10Operations, 10ops-codfw: (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Marostegui) [06:27:07] 10DBA, 10Operations, 10ops-eqiad: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui) [06:49:08] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: (Needed By 31st January) codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Marostegui) These hosts have been assigned to the partman recipe and set as initial spare for the installation: https://gerrit.wiki... [06:51:07] 10DBA, 10Operations, 10ops-eqiad: (Needed by 31st January) eqiad: rack/setup/install es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10Marostegui) These hosts have been assigned to the partman recipe and set as initial spare for the installation: https://github.com/wikimedia/puppet/blob/... [07:08:59] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [07:33:20] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [07:49:10] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) s2 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [] db1129 [] db1125 [] db1122 [] db1105 [] db1103 [... [08:17:45] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [08:18:07] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) @Andrew can you check if this schema change applies also to `labtestwiki` and if so, run it? Thanks! [08:26:37] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [11:28:37] 10DBA, 10Operations: backup2001 crashed 2019-12-08 - https://phabricator.wikimedia.org/T240177 (10Marostegui) Looks like this host crashed again? ` [11:23:13] <+icinga-wm> PROBLEM - Host backup2001 is DOWN: PING CRITICAL - Packet loss = 100% [11:23:39] <+icinga-wm> RECOVERY - Host backup2001 is UP: PING OK -... [11:29:52] 10DBA, 10Operations: backup2001 crashed 2019-12-08 - https://phabricator.wikimedia.org/T240177 (10MoritzMuehlenhoff) There's nothing which indicates a cause of crash in SEL, syslog or kernel logs. [11:31:18] 10DBA, 10Operations: backup2001 crashed 2019-12-08 - https://phabricator.wikimedia.org/T240177 (10MoritzMuehlenhoff) I think this bug is just another case of T238305 [11:39:51] 10DBA, 10Operations: backup2001 crashed 2019-12-08 - https://phabricator.wikimedia.org/T240177 (10jcrespo) a:05jcrespo→03Papaul @Papaul could you proceed with T240177#5727654 as this is the 3rd crash, and the second since firmware upgrade. [11:42:53] all db backups are going to succeed, but s3 now takes 4+ hours on codfw [11:43:06] may have to tune that [13:49:12] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [14:01:37] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [14:09:22] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) s4 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1004 [] db1138 [] db1125 [] db1121 [] db1103 [] db1102 [... [15:02:52] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [15:07:26] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [17:18:32] 10DBA, 10Operations, 10ops-codfw: Upgrade BIOS and firmware on db2084 - https://phabricator.wikimedia.org/T241103 (10Papaul) a:05Papaul→03jcrespo Before BIOS Version 2.4.3 Firmware Version 2.40.40.40 After BIOS Version 2.11.0 Firmware Version 2.70.70.70 Upgrade complete I tried to start MYSQL wit... [17:54:10] 10DBA, 10Operations, 10ops-codfw: Upgrade BIOS and firmware on db2084 - https://phabricator.wikimedia.org/T241103 (10jcrespo) a:05jcrespo→03Marostegui I have started mysql instances back again, and replication, as on codfw there is low load. I saw to minor issues @Marostegui , not important in this case... [22:46:32] 10DBA, 10Operations: Decommission db2043-db2070 - https://phabricator.wikimedia.org/T228258 (10Papaul)