[06:17:14] 10DBA: Remove partitions from revision table - https://phabricator.wikimedia.org/T239453 (10Marostegui) a:03Marostegui [06:17:44] 10Blocked-on-schema-change, 10DBA: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10Marostegui) a:03Marostegui [06:34:08] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [06:43:10] 10Blocked-on-schema-change, 10DBA: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10Marostegui) x1 progress: [] dbstore1005 [] db2131 [] db2115 [] db2101 [] db2096 [] db1140 [] db1137 [] db1127 [] db1120 [06:48:20] 10Blocked-on-schema-change, 10DBA: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10Marostegui) db2131 flowdb.flow_wiki_ref has been altered and this is how it looks like now: ` CREATE TABLE `flow_wiki_ref` ( `ref_src_object_id` binary(11) NOT NULL, `ref_src_object_type` va... [07:19:15] 10Blocked-on-schema-change, 10DBA: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10Marostegui) Wikitech has been altered, it was empty anyways: ` root@db1133.eqiad.wmnet[labswiki]> select count(*) from flow_wiki_ref; +----------+ | count(*) | +----------+ | 0 | +--------... [07:19:24] 10Blocked-on-schema-change, 10DBA: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10Marostegui) [07:28:03] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) @Andrew @Bstorm who in WMCS would be responsible for restarting mysql on these hosts? ` labservices1001 labservices1002 labtestserv... [07:28:56] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [07:47:32] 10Blocked-on-schema-change, 10DBA: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10Marostegui) As I was expecting, as soon as we got a write to that table (it doesn't have many, so it took a while), replication broke on db2131 x1 (we use ROW): ` Last_SQL_Error: C... [07:49:58] 10Blocked-on-schema-change, 10DBA: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10Marostegui) [08:11:05] 10Blocked-on-schema-change, 10DBA: Extend flow_wiki_ref.ref_src_wiki - https://phabricator.wikimedia.org/T241387 (10Marostegui) [08:28:06] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [08:38:35] 10DBA, 10Operations, 10Puppet, 10User-jbond: DB: perform rolling restart of mariadb deamons to pick up CA changes - https://phabricator.wikimedia.org/T239791 (10Marostegui) [08:59:31] 10DBA, 10Operations, 10ops-codfw: Upgrade BIOS and firmware on db2076 - https://phabricator.wikimedia.org/T241647 (10Marostegui) This host is depooled and ready to get, downtimed MySQL stopped and powered off for @Papaul to proceed. [13:18:02] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) a:03Marostegui [13:26:23] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) I have altered s5 codfw wikis and this is how the table looks like now: ` mysql.py -hdb2123 dewiki -e "show create table abuse_filter_log\G" *... [13:27:22] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) @Daimona ^ looks good to you? [13:27:39] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [13:49:08] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Daimona) >>! In T234052#5770727, @Marostegui wrote: > @Daimona ^ looks good to you? Yes, thanks! [13:57:36] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) >>! In T234052#5770727, @Marostegui wrote: > @Daimona ^ looks good to you? Thanks for confirming! [13:57:58] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) s5 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [] dbstore1003 [] db1130 [] db1124 [] db1113 [] db1110 [] db1102 []... [14:23:14] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [14:34:46] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [14:36:49] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) s6 eqiad progress [] labsdb1012 [] labsdb1011 [] labsdb1010 [] labsdb1009 [x] dbstore1005 [] db1139 [] db1131 [] db1125 [] db1113 [] db1098 [... [14:59:23] 10Blocked-on-schema-change, 10DBA, 10AbuseFilter: Add abuse_filter_log.afl_filter_id and afl_global columns - https://phabricator.wikimedia.org/T234052 (10Marostegui) [17:15:55] 10DBA, 10Operations, 10ops-codfw: codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10Papaul) [23:36:49] 10DBA, 10Operations, 10ops-eqiad: (No Need By Date) eqiad: rack/setup/install es102[0-5].eqiad.wmnet - https://phabricator.wikimedia.org/T241359 (10wiki_willy) [23:41:57] 10DBA, 10Operations, 10ops-codfw: (No Need By Date Provided) codfw: rack/setup/install es202[0-5].codfw.wmnet - https://phabricator.wikimedia.org/T241336 (10wiki_willy)