[04:38:05] 10DBA: Compare a few tables per section before the switchover - https://phabricator.wikimedia.org/T260042 (10Marostegui) [06:16:24] 10DBA, 10Patch-For-Review, 10User-Kormat, 10cloud-services-team (Kanban): Upgrade m5 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T260324 (10Marostegui) @andrew does the above works for you? [07:23:27] 10DBA: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) @jcrespo @kormat thoughts on codfw -> eqiad replication? Initially I didn't include them as we are not switching those services, but maybe for consistency we should [08:13:30] 10DBA: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) [08:22:34] 10DBA: Compare a few tables per section before the switchover - https://phabricator.wikimedia.org/T260042 (10Marostegui) [08:48:34] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Kormat) [08:56:22] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Kormat) [09:08:34] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Kormat) [09:10:19] 10DBA: Compare a few tables per section before the switchover - https://phabricator.wikimedia.org/T260042 (10Marostegui) [09:21:30] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Kormat) [10:31:18] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Kormat) [10:31:26] 10DBA: Compare a few tables per section before the switchover - https://phabricator.wikimedia.org/T260042 (10Marostegui) [10:31:53] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Kormat) [10:45:36] 10DBA: Compare a few tables per section before the switchover - https://phabricator.wikimedia.org/T260042 (10Marostegui) [10:51:55] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Kormat) [10:53:45] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) I am re-doing a compare on all those sections where cross replication is enabled to be double sure (and to keep warming up tables), and so far so good. [10:56:34] The calendar alert has fired, no more maintenance to be done on eqiad core hosts :) [10:58:07] moratorium in effect? :) [10:58:22] haha [10:58:38] we should do something with MOTD on hosts or something ;) [10:58:52] Same goes for codfw of course! No more schema changes, reboots, upgrades and such [11:06:45] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) Checked all codfw hosts with notifications disabled. Only found db2135 (m5) [11:06:54] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) [11:13:21] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) [11:18:54] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) [11:22:48] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) [11:45:26] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) [11:51:24] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) [11:58:12] 10DBA: Compare a few tables per section before the switchover - https://phabricator.wikimedia.org/T260042 (10Marostegui) [12:02:29] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) [12:08:37] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) [12:24:59] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Kormat) [12:30:50] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) [12:43:18] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10jcrespo) [12:44:10] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) [12:44:29] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) Weights are tackled, but I will give them another review tomorrow morning. [13:02:06] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Kormat) [13:07:08] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10Marostegui) p:05Triage→03Medium [13:14:55] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Kormat) [13:19:19] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Kormat) [13:23:02] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Kormat) >>! In T243373#6414969, @Marostegui wrote: > @jcrespo @kormat thoughts on codfw -> eqiad replication on misc sections? Initially I didn't include them as we are not switching... [13:26:38] CRITICAL slave_sql_state Slave_SQL_Running: No, Errno: 1062, Errmsg: Could not execute Write_rows_v1 event on table heartbeat.heartbeat: Duplicate entry '180359271' for key 'PRIMARY', Error_code: 1062: handler error HA_ERR_FOUND_DUPP_KEY: the event's master log db1074-bin.005718, end_log_pos 302614331 [13:27:00] jynus: which node? [13:27:11] db1125 [13:27:19] thanks [13:27:46] marostegui: i think we have more labsdb breakages due to the codfw ->eqiad replication ^ [13:27:48] I fixed labs for that earlier [13:27:49] yeah [13:28:04] I will fix those [13:28:30] ah that is s2 [13:30:04] done [13:42:10] btw, created this https://wikitech.wikimedia.org/wiki/Switch_Datacenter/planned_db_maintenance#2020_Switch_Datacenter [13:48:48] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) I have gone thru logs from past switchover and we indeed left mX hosts aside from reenabling replication codfw -> eqiad, so let's do that [13:49:29] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) [14:00:03] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Kormat) GTID disabled on all codfw masters except for s1. [14:33:34] 10DBA, 10Patch-For-Review, 10User-Kormat, 10cloud-services-team (Kanban): Upgrade m5 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T260324 (10Andrew) Sorry I missed this before! I can definitely be available at 14:00 UTC on Monday. It might be nice to have a mediawiki expert around as we... [14:40:36] 10DBA, 10Patch-For-Review, 10User-Kormat, 10cloud-services-team (Kanban): Upgrade m5 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T260324 (10Marostegui) [16:44:42] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10wiki_willy) a:03Jclark-ctr [16:45:32] 10DBA, 10Operations, 10ops-eqiad: db1139 memory errors on boot 2020-08-27 - https://phabricator.wikimedia.org/T261405 (10wiki_willy) Hi @Jclark-ctr - can you take a look at this one? It was just purchased last year, so it's still under warranty. Thanks, Willy [20:17:03] 10DBA, 10Data-Services: Prepare and check storage layer for jawikivoyage - https://phabricator.wikimedia.org/T260482 (10Nintendofan885) @Marostegui the wiki was created [21:49:44] 10DBA, 10MediaWiki-extensions-OAuthRateLimiter, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway), and 3 others: Review request for a new database table for OAuthRateLimiter - https://phabricator.wikimedia.org/T258711 (10Pchelolo) Not that the security reviews are done, patches are merged and w... [21:54:35] 10DBA, 10MediaWiki-extensions-OAuthRateLimiter, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway), and 3 others: Review request for a new database table for OAuthRateLimiter - https://phabricator.wikimedia.org/T258711 (10Reedy) >>! In T258711#6417866, @Pchelolo wrote: > Not that the security re... [21:55:37] 10DBA, 10MediaWiki-extensions-OAuthRateLimiter, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway), and 3 others: Review request for a new database table for OAuthRateLimiter - https://phabricator.wikimedia.org/T258711 (10Pchelolo) > Where does it want to go? With the other OAuth tables on meta... [22:01:24] 10DBA, 10MediaWiki-extensions-OAuthRateLimiter, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway), and 3 others: Review request for a new database table for OAuthRateLimiter - https://phabricator.wikimedia.org/T258711 (10Reedy) `sql.php --wiki=metawiki /path/to/schema.sql` As the extension pat...