[05:20:55] 10DBA, 10Cloud-Services: Prepare and check storage layer for fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T202820 (10Marostegui) Let us know when this wiki is created so we can sanitize it and give #cloud-services-team the green light to create the views. [05:21:07] 10DBA, 10Cloud-Services: Prepare and check storage layer for fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T202820 (10Marostegui) p:05Triage>03Normal [06:23:55] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) [06:24:05] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_moved_to_title/rc_moved_to_ns on wmf databases - https://phabricator.wikimedia.org/T51191 (10Marostegui) [06:24:12] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review, 10Schema-change: Dropping rc_cur_time on wmf databases - https://phabricator.wikimedia.org/T67448 (10Marostegui) [06:36:41] 10DBA: Check consistency on x1 - https://phabricator.wikimedia.org/T202519 (10Marostegui) `echo_notification` fixed and fully checked Now checking the last table: `echo_target_page` [06:51:57] 10DBA, 10Core-Platform-Team, 10Patch-For-Review, 10Schema-change: Fix WMF schemas to not break when comment store goes WRITE_NEW - https://phabricator.wikimedia.org/T187089 (10Marostegui) [06:54:56] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737 (10Marostegui) [07:23:19] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2088 rebooted itself and came back sick - https://phabricator.wikimedia.org/T202822 (10jcrespo) Thanks @Andrew for taking the time, I owe you a drink of your preference next time we meet. [08:36:29] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Marostegui) @Bstorm s8 changes have been replicated to labs - whenever you can, please re-run the views there. Thanks! [11:49:51] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10aborrero) [11:50:30] 10DBA, 10Schema-change: Drop externallinks.el_from_namespace on wmf databases - https://phabricator.wikimedia.org/T114117 (10Bstorm) Done [11:56:49] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10Marostegui) [12:02:46] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10Marostegui) I have created the empty databases: ``` nova_api_eqiad1 nova_eqiad1 ``` Also applied the GRANTS ``` GRANT USAGE ON *.* TO 'nova'@'208.80.154.23' IDENTIFIED BY... [12:34:30] re [12:34:53] 10DBA, 10cloud-services-team: cloudvps: dedicated openstack database - https://phabricator.wikimedia.org/T202889 (10aborrero) [12:36:28] 10DBA: Check consistency on x1 - https://phabricator.wikimedia.org/T202519 (10Marostegui) 05Open>03Resolved `echo_target_page` checked. All checked and whatever had differences have been fixed! [12:53:11] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10aborrero) The two new DB will imported tomorrow. [12:54:22] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10jcrespo) [12:55:31] 10DBA, 10Cloud-Services, 10Patch-For-Review: cloudvps: eqiad1: move nova db to m5-master - https://phabricator.wikimedia.org/T202549 (10jcrespo) "Review m5 database backups" Usually means giving extra grants to the backup user, review the list and check those get backed up for the first time. [13:02:01] hi, banyek [13:02:31] this is jaime [13:02:53] hi! [13:36:49] 10Blocked-on-schema-change, 10Wikibase-Quality, 10Wikibase-Quality-Constraints, 10Wikidata, 10Schema-change: Deploy schema change for adding numeric primary key to wbqc_constraints table - https://phabricator.wikimedia.org/T189101 (10Lucas_Werkmeister_WMDE) `constraint_guid` should be kept. The SQL patch... [13:46:45] marostegui, jynus: I've added you to the CR for the mysql module in spicerack. Some caveats: it's mostly the old code from switcdc, and there is a big TODO note on replacing it with something that speaks native mysql. [13:48:06] let me know if I want me to wait a review or could go ahead and integrate any suggestions you might have in a later commit [13:49:19] we have something that speaks mysql now [13:49:30] and has already been used on switchovers... [13:50:47] can be installed as a debian package? [13:51:20] it's easy to integrate? we don't have much time left and I went for the quick way [13:53:12] no, there is not a debian package because things are not finished yet [13:55:25] 10Blocked-on-schema-change, 10Wikibase-Quality, 10Wikibase-Quality-Constraints, 10Wikidata, 10Schema-change: Deploy schema change for adding numeric primary key to wbqc_constraints table - https://phabricator.wikimedia.org/T189101 (10jcrespo) Please if you could give this a lot of priority, because if we... [14:16:31] 10Blocked-on-schema-change, 10Wikibase-Quality, 10Wikibase-Quality-Constraints, 10Wikidata, and 2 others: Deploy schema change for adding numeric primary key to wbqc_constraints table - https://phabricator.wikimedia.org/T189101 (10WMDE-leszek) p:05Normal>03High [14:18:03] 10Blocked-on-schema-change, 10Wikibase-Quality, 10Wikibase-Quality-Constraints, 10Wikidata, and 2 others: Deploy schema change for adding numeric primary key to wbqc_constraints table - https://phabricator.wikimedia.org/T189101 (10WMDE-leszek) [15:04:43] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2058 - https://phabricator.wikimedia.org/T202824 (10Papaul) a:05Papaul>03Marostegui Disk replacement complete [15:05:43] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2058 - https://phabricator.wikimedia.org/T202824 (10Marostegui) Thanks! ``` physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SAS, 600 GB, Rebuilding) ``` [15:28:31] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2088 rebooted itself and came back sick - https://phabricator.wikimedia.org/T202822 (10Papaul) a:05Papaul>03Marostegui BIOS upgrade from version 2.4.3 to 2.8.0 IDRAC upgrade from version 2.40 to 2.60 [15:32:14] jynus, marostegui: so, what should I do for the above? wait for a review, go ahead, do you have alternative proposals... [15:35:32] deploy, but I would like to test it [15:37:42] those unit tests you added are useless [15:38:18] of course! dry-runs and tests will be extensive, I've done so preliminary test and all seems to work fine like last time, but I'll be much more relieved if you will check the queries too [15:38:50] yeah, fully agree, I didn't spent much time on them given I know this module will be thrown away with a mysql-native one, if you're asking why it doesn't check the queries themselves [15:39:07] I am not complaining [15:39:24] just that having unit tests like this are not really worth [15:40:00] is worth to test that calls inside the library works, that there are no typos and such [15:40:06] sure [15:40:18] they will be needed anyway, in addition to tests on teh actual executed queries [15:42:18] I am not sure this will work for the general case- with potential multi-instance masters and so [15:42:31] but if you are confident it works as is, deploy [15:42:55] I still want to schedule proper production tests [15:46:23] we will throw most of that away when we stop using puppet anyway [15:47:17] of course for the tests, for now will just be merged, once we'll have it installed in prod I'll ping you so we can plan a test [15:48:30] I see the functionality, but I don't see where that is called? [15:48:48] it's not yet, that is only the library [15:48:51] ok [15:49:02] I will add some change to the tendril stuff [15:49:25] oh, no, I said that change will not be needed anymore [15:49:33] the zarcillo one? [15:49:36] so ping me for the call [15:49:45] and to schedule tests, deploy as is [15:49:52] sure, ack, thanks! [15:49:59] 10DBA, 10Operations, 10ops-codfw, 10Patch-For-Review: db2088 rebooted itself and came back sick - https://phabricator.wikimedia.org/T202822 (10Marostegui) Thanks Papaul. I have upgraded kernel, mysql and started it. Once it has caught up I will do a data check before repooling it. [15:49:59] for the equivalent to the main() [16:18:13] 10DBA, 10Wikimedia-Site-requests: advisorswiki is not in any s?.dblist - https://phabricator.wikimedia.org/T202904 (10Anomie) [16:20:57] 10DBA, 10Wikimedia-Site-requests, 10Patch-For-Review: advisorswiki is not in any s?.dblist - https://phabricator.wikimedia.org/T202904 (10Marostegui) Was it created 15th of April? If so, it has missed schema changes for sure. I will need to check which ones [16:26:37] 10DBA, 10Wikimedia-Site-requests, 10Patch-For-Review: advisorswiki is not in any s?.dblist - https://phabricator.wikimedia.org/T202904 (10Anomie) The 18th of April at the latest, it looks like. [16:28:12] 10DBA, 10Wikimedia-Site-requests, 10Patch-For-Review: advisorswiki is not in any s?.dblist - https://phabricator.wikimedia.org/T202904 (10Marostegui) I will check all the schema changes we have done since 18th April then. [20:13:19] 10DBA, 10Wikimedia-Site-requests, 10Patch-For-Review: advisorswiki is not in any s?.dblist - https://phabricator.wikimedia.org/T202904 (10Peachey88) I don't see a task for creating this wiki, might want to check if nothing else has been missed off the list as well. [20:25:58] 10DBA, 10Wikimedia-Site-requests, 10Patch-For-Review: advisorswiki is not in any s?.dblist - https://phabricator.wikimedia.org/T202904 (10Anomie) >>! In T202904#4535114, @Marostegui wrote: > I will check all the schema changes we have done since 18th April then. It looks like you might want to check as ear...