[05:02:15] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) [05:04:10] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) [05:09:29] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2067 - https://phabricator.wikimedia.org/T235695 (10Marostegui) a:03Papaul @Papaul can we replace the failed disk with a brand new one? This host is scheduled for decommission, but we have to buy its replacement first (T234608) [05:09:39] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2067 - https://phabricator.wikimedia.org/T235695 (10Marostegui) p:05Triage→03Normal [05:25:47] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) [05:30:58] 10Blocked-on-schema-change, 10DBA, 10Core Platform Team: Schema change for refactored actor and comment storage - https://phabricator.wikimedia.org/T233135 (10Marostegui) [05:31:00] 10Blocked-on-schema-change, 10DBA: Schema change to rename user_newtalk indexes - https://phabricator.wikimedia.org/T234066 (10Marostegui) [06:03:32] 10DBA, 10conftool: specify group (api/vslow/etc) weights in terms of 0..100 instead of 0..1 - https://phabricator.wikimedia.org/T231018 (10Marostegui) [07:48:14] 10DBA, 10Machine vision, 10Product-Infrastructure-Team-Backlog: DBA review for the MachineVision extension - https://phabricator.wikimedia.org/T227355 (10jcrespo) Sadly I am not in charge of databases anymore, @Marostegui will have to do the work. [07:49:00] jynus: I was actually going to ask you if you could continue on that ticket as you've done most of the checks there [07:49:23] well, not after I commented I am not in charge [07:49:28] ok [07:50:57] I am not sure if I will get to that before my holidays and I will be gone when they plan to release that [07:51:01] But I'll see what I can do [07:51:45] sure, comment to that or ask your manager [07:51:55] *about that [07:51:55] ok [07:55:19] 10DBA, 10Machine vision, 10Product-Infrastructure-Team-Backlog: DBA review for the MachineVision extension - https://phabricator.wikimedia.org/T227355 (10Marostegui) I am going on holidays in a few days, and I will be gone till the 11th of Novemeber, so I am not sure I will be able to tackle this before I le... [10:33:10] 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for mnwwiki - https://phabricator.wikimedia.org/T235743 (10jhsoby) [10:47:26] 10DBA, 10Data-Services, 10Operations, 10cloud-services-team (Kanban): Prepare and check storage layer for mnwwiki - https://phabricator.wikimedia.org/T235743 (10Marostegui) Let us know when the database is created so we can sanitize its tables and hand over to WMCS for views creation. [11:15:30] https://www.reddit.com/r/mariadb/comments/dj2p3g/we_are_the_mariadb_engineering_team_ask_the/ [12:09:55] interesting [12:10:14] I might register in reddit just for that [12:48:34] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Marostegui) [12:48:55] 10DBA: Recompress special slaves across eqiad and codfw - https://phabricator.wikimedia.org/T235599 (10Marostegui) [12:56:41] 10DBA: Compress new Wikibase tables - https://phabricator.wikimedia.org/T232446 (10Reedy) [13:05:29] marostegui: don't start trolling! Although mentioning bugs and open source commitment may be interesting [13:05:32] :-P [13:05:42] hahaha [13:18:55] 10DBA, 10Operations, 10serviceops, 10Goal, 10Patch-For-Review: Strengthen backup infrastructure and support - https://phabricator.wikimedia.org/T229209 (10jcrespo) The copy finished correctly and actually found a bug on transfer.py: ` ERROR: Original checksum c89dcd766fa3072718753b9ab0bdfb7d baculasd... [14:29:49] jynus: https://gerrit.wikimedia.org/r/#/c/operations/dns/+/543863 [14:30:03] just taking my mind off things doing some simple busy work [14:30:15] hope I don't step on your toes ;-) [14:30:21] not at all [14:30:30] I was going to ask you if it was just that [14:30:52] plus profile::backup::host ferm rules [14:30:52] because it seemed too simple enough to just be a dns update [14:30:55] coming right up [14:31:11] should allow backup1001 to connect to all hosts on the bacula-fd port [14:31:16] so I thought about the plan, I started and completed alredy the transference [14:31:47] but we didn't have into account the database merge [14:32:58] merge? [14:33:09] should we import "copy" the old catalog first? [14:33:23] before creating new backups? [14:34:26] import here means mysqldump | mysql && upgrade-to-version-9 [14:35:06] so we can later reconnect archive locally and production remotelly? [14:43:42] sure I guess. Why though ? [14:43:57] I mean are we also changing the backend mysql server? [14:44:21] it's not necessary for testing (I think), that's my point. [14:44:47] no, of course [14:44:52] testing is the next phase [14:45:07] I am thinking a step we didn't think about when migrating for real [14:45:24] "migrating" the database, even if it will just be used the same one [14:46:02] moving to bacula9 or using bacula and creating a snapshot befor upgrading is the same [14:46:19] just I wanted it as an explicity step [14:46:52] or I can use bacula9 to test the upgrade first, too [14:50:24] ok, makes sense to do that step as well in order to test the upgrade [14:50:27] fine by me [14:50:41] I just had written "start with an empty database" [14:50:46] and then I realized that was wrong [14:51:00] we have to start with a migration of the old one [14:51:15] we move the pysical files location or not [14:51:27] correct? [14:56:04] in the end we will require to use either the old database upgraded, or a new database that is a copy of the old one (again upgraded). So yeah you are correct [14:56:41] I had planned it wrong and we didn't discuss a lot about the db, that is why I brought out [14:57:01] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2067 - https://phabricator.wikimedia.org/T235695 (10Papaul) a:05Papaul→03Marostegui Disk replaced [14:57:05] my next step is to setup a test, even if manually [14:57:27] and maybe test the database upgrade with production data [14:57:49] 10DBA, 10Operations, 10ops-codfw: Degraded RAID on db2067 - https://phabricator.wikimedia.org/T235695 (10Marostegui) Thanks! ` physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 600 GB, Rebuilding) ` [14:58:17] akosiaris: then we could maybe set a day for a tentative migration, unless you see more blockers (and assuming everything goes as intended) [14:58:48] there is still the firewall issues (which I am hopefully fixing right now) [14:58:56] I can't think though of anything else [14:59:01] yes, also more changes on puppet [14:59:10] * akosiaris meeting in 1 min [14:59:11] which I will prepare [14:59:21] tentative next week or in 2 weeks? [15:14:53] when you come back, aside from the migration schedule, I would like to propose you a wild monitoring idea