[04:45:09] 10DBA, 10Patch-For-Review, 10User-Kormat, 10cloud-services-team (Kanban): Upgrade m5 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T260324 (10Marostegui) >>! In T260324#6416028, @Andrew wrote: > Sorry I missed this before! I can definitely be available at 14:00 UTC on Monday. It might be... [05:32:37] 10DBA, 10Patch-For-Review, 10User-Kormat, 10cloud-services-team (Kanban): Upgrade m5 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T260324 (10Andrew) ok! [07:02:28] 10DBA: Compare a few tables per section before the switchover - https://phabricator.wikimedia.org/T260042 (10Marostegui) [07:03:35] 10DBA: Compare a few tables per section before the switchover - https://phabricator.wikimedia.org/T260042 (10Marostegui) [07:04:47] 10DBA: Compare a few tables per section before the switchover - https://phabricator.wikimedia.org/T260042 (10Marostegui) I am doing a few more checks with some more tbales, so the tables are warmed up even more. [08:24:43] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Kormat) [08:27:58] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Kormat) [08:28:35] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Kormat) Confirmed that GTID is disabled on all codfw masters: ` sudo cumin 'A:db-role-master and A:codfw' 'mysql -e "show slave status\G" | grep Using_Gtid' 15 hosts will be targeted:... [08:43:13] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) 05Open→03Resolved I have reviewed the weights again so this is all done. As part of T260042 I am warming up all the hosts in codfw, including the masters, parsercache,... [08:57:59] I am cleaning up the masters table on zarcillo as it doesn't look super updated [09:26:16] 10DBA, 10observability, 10Patch-For-Review: check_mariadb_dump failing on alert[12]* hosts - https://phabricator.wikimedia.org/T260686 (10jcrespo) 05Open→03Resolved [10:36:01] 10DBA: Failover DB masters in row D - https://phabricator.wikimedia.org/T186188 (10Marostegui) [10:36:44] 10DBA: Failover DB masters in row D - https://phabricator.wikimedia.org/T186188 (10Marostegui) 05Stalled→03Open This needs review: we need to check how many masters we have per row and distribute them equally once eqiad is stand by [11:08:19] 10DBA: Failover DB masters in row D - https://phabricator.wikimedia.org/T186188 (10Marostegui) Current distribution: ` root@db1115.eqiad.wmnet[zarcillo]> select instance,rack,section from masters inner join servers on masters.instance=servers.hostname where masters.dc='eqiad' order by rack; +----------+------+... [11:37:25] 10DBA: Switchover s8 primary database master db1109 -> db1104 - Date TBD - https://phabricator.wikimedia.org/T239238 (10Marostegui) 05Stalled→03Open [11:37:31] 10Blocked-on-schema-change, 10DBA, 10Wikidata: Schema change on production for increase the size of wbt_text_in_lang.wbxl_language - https://phabricator.wikimedia.org/T237120 (10Marostegui) [11:37:33] 10DBA: Failover DB masters in row D - https://phabricator.wikimedia.org/T186188 (10Marostegui) [18:32:42] 10DBA: Compare a few tables per section before the switchover - https://phabricator.wikimedia.org/T260042 (10Ladsgroup) Do you want to run the schema drift thingy for codfw too?