[04:53:40] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover - https://phabricator.wikimedia.org/T243373 (10Marostegui) [05:02:08] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover and some other checks - https://phabricator.wikimedia.org/T243373 (10Marostegui) [05:03:00] 10DBA, 10User-Kormat: Enable DB replication codfw -> eqiad before the switchover and some other checks - https://phabricator.wikimedia.org/T243373 (10Marostegui) All codfw have been checked for GTID enabled, they all had it (not codfw masters) [05:03:30] bstorm elukey: labsdb1012 still has puppet disabled for T260843, is that intended? [05:03:30] T260843: Set up roles for new wiki replicas layout - https://phabricator.wikimedia.org/T260843 [05:13:26] 10DBA: Compare a few tables per section before the switchover - https://phabricator.wikimedia.org/T260042 (10Marostegui) Excellent, thank you @Ladsgroup! [05:22:06] marostegui: no. I thought I had enabled all of them. Sorry! [05:22:18] bstorm: I can do it , no worries [05:22:28] Thank you! [05:22:34] bstorm: have a good night! [06:00:49] thanks! [08:07:06] 10DBA, 10Cloud-Services, 10Platform Team Initiatives (API Gateway): Prepare and check storage layer for api.wikimedia.org - https://phabricator.wikimedia.org/T246946 (10Marostegui) @hnowlan can you confirm this wiki has been created? We received an alert of a new wiki database called: `apiportalwiki`, we nee... [08:08:04] 10DBA, 10Data-Services: Prepare and check storage layer for jawikivoyage - https://phabricator.wikimedia.org/T260482 (10Marostegui) Database has been sanitized. Triggers have been tested with my user and they work fine. `jawikivoyage_p` has been created on all hosts. Grants have been added to labsdb1009, 1010,... [10:14:25] 10DBA, 10Cloud-Services, 10Platform Team Initiatives (API Gateway): Prepare and check storage layer for api.wikimedia.org - https://phabricator.wikimedia.org/T246946 (10hnowlan) @Marostegui confirming that it was created on the 27th. [10:54:44] these are the server backup configuration: https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/profile/templates/mariadb/backup_config/ [10:54:48] FYI [10:54:49] 10DBA, 10Cloud-Services, 10Platform Team Initiatives (API Gateway): Prepare and check storage layer for api.wikimedia.org - https://phabricator.wikimedia.org/T246946 (10Marostegui) 05Stalled→03Open Thanks - I will work on its sanitization [10:55:47] jynus: thanks :) [11:00:14] 10DBA, 10Cloud-Services, 10Platform Team Initiatives (API Gateway): Prepare and check storage layer for api.wikimedia.org - https://phabricator.wikimedia.org/T246946 (10Marostegui) a:03Marostegui [11:02:48] I have updated https://wikitech.wikimedia.org/wiki/MariaDB/Backups#Backups_quick_cheatsheet to reflect it [11:10:51] bstorm: puppet was also disabled on labsdb1011, I have enabled it back [11:45:32] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Tue, Sept 8 PDU Upgrade 12pm-4pm UTC- Racks D3 and D4 - https://phabricator.wikimedia.org/T261452 (10Marostegui) [11:45:43] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Wed, Sept 9 PDU Upgrade 12pm-4pm UTC- Racks D5 and D6 - https://phabricator.wikimedia.org/T261453 (10Marostegui) [11:45:54] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Thur, Sept 10 PDU Upgrade 12pm-4pm UTC- Racks D7 and D8 - https://phabricator.wikimedia.org/T261454 (10Marostegui) [11:46:00] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Mon, Sept 14 PDU Upgrade 12pm-4pm UTC- Racks C2 and C3 - https://phabricator.wikimedia.org/T261455 (10Marostegui) [11:46:08] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Tue, Sept 15 PDU Upgrade 12pm-4pm UTC- Racks C4 and C5 - https://phabricator.wikimedia.org/T261456 (10Marostegui) [11:46:14] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Wed, Sept 16 PDU Upgrade 12pm-4pm UTC- Racks C6 and C7 - https://phabricator.wikimedia.org/T261457 (10Marostegui) [11:46:24] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Mon, Sept 21 PDU Upgrade 12pm-4pm UTC- Racks D1 and D2 - https://phabricator.wikimedia.org/T261459 (10Marostegui) [11:55:35] o/ [11:55:45] hello :) [11:59:08] hey sobanski, welcome aboard! (Riccardo here) [11:59:43] Hey Riccardo, good to be here [12:02:05] jynus: oh that's perfect re: cheatsheet <3 [12:03:15] hello, sobanski, Jaime here [12:04:00] sobanski: hey. kormat here. ;) [12:04:37] sobanski: hi Lukasz, welcome! [12:06:03] sobanski: Manuel here [12:11:27] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Mon, Sept 21 PDU Upgrade 12pm-4pm UTC- Racks D1 and D2 - https://phabricator.wikimedia.org/T261459 (10Marostegui) dbproxy1016 needs to be failover. I will take care of that now Please take extra care of db1125. Thanks! [12:13:07] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Wed, Sept 9 PDU Upgrade 12pm-4pm UTC- Racks D5 and D6 - https://phabricator.wikimedia.org/T261453 (10Marostegui) Please take extra care of db1122 as it is an eqiad master and lots of slaves hang from it. We might stop mysql there just in case prior the maintenance. [12:15:43] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Thur, Sept 10 PDU Upgrade 12pm-4pm UTC- Racks D7 and D8 - https://phabricator.wikimedia.org/T261454 (10Marostegui) Please take extra care with db1123, db1093 and db1109, they are an eqiad masters and lots of slaves hang from them. We might stop mysql just in case. [12:15:57] 10DBA: Failover DB masters in row D - https://phabricator.wikimedia.org/T186188 (10Marostegui) We definitely need to move one out of D8, having 3 there is too much (T261454#6422998) [12:18:19] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Mon, Sept 14 PDU Upgrade 12pm-4pm UTC- Racks C2 and C3 - https://phabricator.wikimedia.org/T261455 (10Marostegui) Please take extra care with db1087, db1100 and db1109, they are an eqiad masters and lots of slaves hang from them. We might stop mysql just in case. [12:20:00] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: Tue, Sept 15 PDU Upgrade 12pm-4pm UTC- Racks C4 and C5 - https://phabricator.wikimedia.org/T261456 (10Marostegui) Be careful with dbproxy1018 and dbproxy1019, and labsdb1010 as they serve cloud infra and that service will not be switched to codfw. [12:54:55] 10DBA: Compare a few tables per section before the switchover - https://phabricator.wikimedia.org/T260042 (10Marostegui) [12:55:06] 10DBA: Compare a few tables per section before the switchover - https://phabricator.wikimedia.org/T260042 (10Marostegui) 05Open→03Resolved This is done [12:55:10] 10DBA, 10Patch-For-Review, 10Sustainability (Incident Followup), 10User-Banyek: Automatically compare a few tables per section between hosts and DC - https://phabricator.wikimedia.org/T207253 (10Marostegui) [12:57:06] 10DBA, 10Cloud-VPS, 10cloud-services-team (Kanban): Drop openstack databases from m5-master - https://phabricator.wikimedia.org/T261152 (10Marostegui) @Andrew will you be dropping these before the m5 failover? (T260324) [13:09:53] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: Mon, Sept 21 PDU Upgrade 12pm-4pm UTC- Racks D1 and D2 - https://phabricator.wikimedia.org/T261459 (10Marostegui) dbproxy1016 is no longer active, its service has been failed over. [13:46:26] 10DBA, 10Cloud-VPS, 10cloud-services-team (Kanban): Drop openstack databases from m5-master - https://phabricator.wikimedia.org/T261152 (10Andrew) I've dropped all these databases. Is there anything left to do here? [13:48:27] 10DBA, 10Cloud-VPS, 10cloud-services-team (Kanban): Drop openstack databases from m5-master - https://phabricator.wikimedia.org/T261152 (10Marostegui) Let's clean up the grants too, I can do that [13:48:53] 10DBA, 10Cloud-VPS, 10cloud-services-team (Kanban): Drop openstack databases from m5-master - https://phabricator.wikimedia.org/T261152 (10Marostegui) [13:49:05] 10DBA, 10Cloud-VPS, 10cloud-services-team (Kanban): Drop openstack databases from m5-master - https://phabricator.wikimedia.org/T261152 (10Andrew) thanks! [13:53:15] 10DBA, 10Cloud-VPS, 10cloud-services-team (Kanban): Drop openstack databases from m5-master - https://phabricator.wikimedia.org/T261152 (10Marostegui) All these must go - can you confirm @Andrew ? ` +-----------+--------------------------------+ | user | host | +-----------+--... [13:58:30] 10DBA, 10Cloud-Services, 10Platform Team Initiatives (API Gateway): Prepare and check storage layer for api.wikimedia.org - https://phabricator.wikimedia.org/T246946 (10Marostegui) Database has been sanitized. `apiportalwiki_p` has been created on all hosts. Grants have been added to labsdb1009, 1010, 1011 a... [14:03:08] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for jawikivoyage - https://phabricator.wikimedia.org/T260482 (10Marostegui) check private data finished clean. This is ready for #cloud-services-team for the views creation on the 4 hosts. [14:03:28] 10DBA, 10Cloud-Services, 10Platform Team Initiatives (API Gateway): Prepare and check storage layer for api.wikimedia.org - https://phabricator.wikimedia.org/T246946 (10hnowlan) This wiki is waiting on apache configuration so users can't be created yet - I'll update when that is set up. [14:04:41] 10DBA, 10Cloud-Services, 10Platform Team Initiatives (API Gateway): Prepare and check storage layer for api.wikimedia.org - https://phabricator.wikimedia.org/T246946 (10Marostegui) 05Open→03Stalled Thanks - I will wait for it to confirm our triggers are working fine. [14:23:19] 10DBA, 10Cloud-VPS, 10cloud-services-team (Kanban): Drop openstack databases from m5-master - https://phabricator.wikimedia.org/T261152 (10Andrew) lgtm [14:57:35] 10DBA, 10Cloud-VPS, 10cloud-services-team (Kanban): Drop openstack databases from m5-master - https://phabricator.wikimedia.org/T261152 (10Marostegui) a:05Andrew→03Marostegui [15:01:12] 10DBA, 10Operations, 10Patch-For-Review, 10User-Kormat, 10User-jbond: Refactor mariadb puppet code - https://phabricator.wikimedia.org/T256972 (10Kormat) [15:45:51] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 2020-08-31) rack/setup/install es20[26-34].codfw.wmnet - https://phabricator.wikimedia.org/T260373 (10Papaul) [17:14:17] 10DBA, 10Operations, 10netops, 10ops-eqiad, 10User-Kormat: Upgrade eqiad rack D4 to 10G switch - https://phabricator.wikimedia.org/T196487 (10RobH) [17:41:03] kormat: FYI on the cumin hosts apt says that wmfmariadbpy-common was automatically installed by is no longer required [17:43:39] volans: right, yes. it can be removed, i'll do it tomorrow if i don't forget again :) [17:43:56] ack, thx :) [17:44:01] no hurry ofc [21:07:53] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 2020-08-31) rack/setup/install es20[26-34].codfw.wmnet - https://phabricator.wikimedia.org/T260373 (10Papaul) [21:08:36] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 2020-08-31) rack/setup/install es20[26-34].codfw.wmnet - https://phabricator.wikimedia.org/T260373 (10Papaul)