[05:27:55] 10DBA, 10Data-Services: Prepare and check storage layer for jawikivoyage - https://phabricator.wikimedia.org/T260482 (10Marostegui) p:05Triage→03Medium Just a reminder, new wikis go to s5 now instead of s3. Let us know when the DB is created so we can sanitize it. [05:29:25] 10Blocked-on-schema-change, 10DBA: Extend sites.site_global_key on WMF production - https://phabricator.wikimedia.org/T260476 (10Marostegui) p:05Triage→03Medium [05:31:15] 10DBA, 10Patch-For-Review, 10User-Urbanecm, 10cloud-services-team (Kanban): Establish process of determining shard for new wikis - https://phabricator.wikimedia.org/T259438 (10Marostegui) @Urbanecm all done I think? [05:32:57] 10DBA: Upgrade m5 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T260324 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db1099.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202008170532_marostegui_19563.log`. [05:49:56] 10DBA: All sorts of random drifts in wikis in s3 - https://phabricator.wikimedia.org/T260111 (10Marostegui) Thanks Amir for reporting this. This will be "fun" to fix... At least those are pretty small wikis, so for some of those we might be even able to do it directly on the master with replication enabled. [06:05:04] 10DBA: Upgrade m5 to Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T260324 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1099.eqiad.wmnet'] ` and were **ALL** successful. [06:28:00] 10DBA, 10cloud-services-team (Kanban): Parametrize wmf-pt-kill so it can connect to different sockets - https://phabricator.wikimedia.org/T260511 (10Marostegui) p:05Triage→03Medium [11:21:02] 10DBA, 10Operations, 10Patch-For-Review, 10User-Kormat, 10User-jbond: Standardize/centralize mapping from section to mariadb port/socket and prom-mysql-exporter port - https://phabricator.wikimedia.org/T257033 (10jbond) > m5 has index 15 guessing this should be "m5 has index 5"? or after reading ". So... [11:24:04] 10DBA, 10Patch-For-Review, 10User-Urbanecm, 10cloud-services-team (Kanban): Establish process of determining shard for new wikis - https://phabricator.wikimedia.org/T259438 (10Urbanecm) 05Open→03Resolved Yup, closing! [11:24:40] 10DBA, 10Patch-For-Review, 10User-Urbanecm, 10cloud-services-team (Kanban): Establish process of determining shard for new wikis - https://phabricator.wikimedia.org/T259438 (10Marostegui) Thanks for all your great work! [12:49:40] 10DBA, 10Operations, 10Patch-For-Review, 10User-Kormat, 10User-jbond: Standardize/centralize mapping from section to mariadb port/socket and prom-mysql-exporter port - https://phabricator.wikimedia.org/T257033 (10Kormat) >>! In T257033#6388369, @jbond wrote: >> m5 has index 15 > guessing this should be "... [12:55:40] 10DBA, 10Operations, 10Patch-For-Review, 10User-Kormat, 10User-jbond: Standardize/centralize mapping from section to mariadb port/socket and prom-mysql-exporter port - https://phabricator.wikimedia.org/T257033 (10jbond) >>! In T257033#6388644, @Kormat wrote: >>>! In T257033#6388369, @jbond wrote: >>> m5... [13:02:53] marostegui: from what I understood from s3 drifts, it's mostly schema changes that fall into cracks for one or two particular wikis. Like dropping the ct_tag column in advisorywiki [13:03:09] yeah [13:03:15] not very surprising, s3 is a mess [13:05:26] but it has been accumulating for years and years, I think once we address it, it'll be pretty healthy again [13:05:48] hopefully [13:57:51] 10DBA: Prepare and check storage layer for thankyouwiki - https://phabricator.wikimedia.org/T260551 (10Urbanecm) [13:58:57] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for lldwiki - https://phabricator.wikimedia.org/T259436 (10Urbanecm) @Marostegui The database was just created, and there should be also some users :). [14:00:42] some maintenance going on on s1? https://grafana.wikimedia.org/d/000000278/mysql-aggregated?panelId=11&fullscreen&orgId=1&from=1597651232530&to=1597672832530&var-site=eqiad&var-group=core&var-shard=All&var-role=All [14:00:55] yes, MCR [14:00:59] ok [14:01:04] it went down anyway now [14:01:12] it will come back :) [14:01:36] running on s1 and s8 (one host each) [14:01:52] 10DBA: Prepare and check storage layer for thankyouwiki - https://phabricator.wikimedia.org/T260551 (10Marostegui) p:05Triage→03Medium a:03Marostegui [14:05:20] 10DBA: Prepare and check storage layer for thankyouwiki - https://phabricator.wikimedia.org/T260551 (10Marostegui) Sanitized correctly on db1124:3315 and db2094:3315. I created a user after the sanitization and the triggers worked fine. Running check_private_data now, just to be fully sure. [14:08:54] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for lldwiki - https://phabricator.wikimedia.org/T259436 (10Marostegui) Sanitized correctly on db1124:3315 and db2094:3315. I created a user after the sanitization and the triggers worked fine. Running check_private_data n... [14:09:59] 10DBA: Prepare and check storage layer for thankyouwiki - https://phabricator.wikimedia.org/T260551 (10Marostegui) Sanitized correctly on db1124:3315 and db2094:3315. @Urbanecm can you create another user so I can check if the triggers are running fine? [14:22:52] jynus: re: https://gerrit.wikimedia.org/r/c/operations/software/wmfmariadbpy/+/620291, i have a patch to make the unittests work on py3.5; what's the best way to share this with you? [14:23:52] just upload it [14:24:32] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for lldwiki - https://phabricator.wikimedia.org/T259436 (10Nintendofan885) [14:24:34] 10DBA, 10Patch-For-Review, 10User-Urbanecm: Move muswiki and mhwiktionary (closed wikis) from s3 to s5 - https://phabricator.wikimedia.org/T259004 (10Nintendofan885) [14:24:43] jynus: done [14:25:53] but that is not the same test [14:26:12] it is an array of lines [14:26:19] I could have done that myself [14:26:39] it is functionally the same [14:27:11] but it is post-csv library [14:27:28] anyway, merge or review, as you prefer [14:27:43] I just need the backup change [14:28:19] but correct the commit msg if you are going to merge as is [14:29:59] +1 from me. [14:30:03] commit msg updated [14:30:29] let me change one thing, putting the testing cases a bit cleaner [14:31:04] so what I was trying it to reimplement the mock's open into an iterator [14:31:11] but I didn't see an elegant way [14:34:39] that's what i did..? [14:35:08] but still reading a single string [14:35:33] i could make it a single string and then just do `csvdata.split("\n")` in the iterator, but 🤷 [14:36:33] BTW, should I merge this without a deploymnt strategy? [14:36:48] this will depend on having a section_ports.csv [14:36:51] file [14:37:02] should that go in the package or puppet? [14:38:13] good question. it feels more like a puppety thing, to me, but i don't have strong feelings on it [14:38:26] on the patch I added a list just in case [14:38:36] but I was expecting, as we discussed at some point [14:38:39] to put it on hiera [14:38:47] and generate it from there to file [14:39:03] right, yeah [14:39:04] so it can be both on puppet and on scripting level for now [14:39:20] i think let's merge the CR as it is now, [14:39:22] let me get it at list +1ed [14:39:34] and then have a proposal for puppet before merging [14:39:35] and i can work on the deployment when it comes to the next wmfmariadbpy release [14:39:37] and I will ask you [14:39:59] let me do at least a proposal or a draft [14:40:10] in case I may be missing something obvious [14:40:11] go for it if you want. [14:40:54] do you have a timeline in mind for deploying wmfbackups? [14:41:09] as soon as possible [14:41:17] because I have a backlog of changes I need to do [14:41:26] non related to refactoring [14:41:34] but if we need to create a repo and more [14:41:42] that may not be in our hads [14:41:44] *hands [14:41:51] to do it quick [14:42:59] 10DBA: Prepare and check storage layer for thankyouwiki - https://phabricator.wikimedia.org/T260551 (10Urbanecm) >>! In T260551#6389038, @Marostegui wrote: > Sanitized correctly on db1124:3315 and db2094:3315. > @Urbanecm can you create another user so I can check if the triggers are running fine? Sure, there s... [14:43:21] that is why we can wait to deploy- after all this should be non-api dependent [14:43:40] hum [14:43:57] as in, backups depend on it, but nothing else depends on mariadb? [14:45:09] my current plan is to deploy wmfmariadbpy 0.4, which would not include this CR [14:45:18] that's ok [14:45:23] and publish 0.5 with this CR included [14:45:34] as I said, I am not in a hurry [14:45:47] i'm confused [14:45:51] you said "as soon as possible" [14:45:55] I want it ASAP [14:45:59] but I don't require it :-D [14:46:15] .. ok :) [14:53:43] after all, developing this on a different branch would have been justified [14:53:55] we would have waited for the release to merge to head [15:02:43] kormat: https://gerrit.wikimedia.org/r/c/operations/software/wmfmariadbpy/+/620291/10..12 [15:06:49] +1'd [15:07:11] I am working on a puppet patch so it is ready for deployment whenever we want [15:35:18] kormat this is fresh from the world of ideas: https://gerrit.wikimedia.org/r/c/operations/puppet/+/620722 [15:37:48] with just a bit of extra work we could deploy it easily to cumin hosts (without any dependency) [15:38:37] but we need to see how useful this could be to apply to other classes (e.g. multiinstance hosts) [15:41:45] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for thankyouwiki - https://phabricator.wikimedia.org/T260551 (10Urbanecm) Adding the other two projects. [16:54:56] 10DBA, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10Patch-For-Review, 10User-Marostegui: DBA review for Echo push notification subscription tables - https://phabricator.wikimedia.org/T246716 (10Mholloway) Hi @Marostegui, thanks for the comments, and sorry for the silence; we've b... [17:09:06] 10DBA, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10Patch-For-Review, 10User-Marostegui: DBA review for Echo push notification subscription tables - https://phabricator.wikimedia.org/T246716 (10Mholloway) There are a couple of other recent changes that I should mention. One is t... [18:14:40] 10DBA, 10Patch-For-Review, 10User-Urbanecm, 10cloud-services-team (Kanban): Establish process of determining shard for new wikis - https://phabricator.wikimedia.org/T259438 (10Urbanecm) [18:14:43] 10DBA, 10Patch-For-Review, 10User-Urbanecm: Move muswiki and mhwiktionary (closed wikis) from s3 to s5 - https://phabricator.wikimedia.org/T259004 (10Urbanecm) [22:55:53] 10DBA, 10Operations, 10Parsoid, 10serviceops, 10Parsoid-Tests: update mysql GRANTs for testreduce - https://phabricator.wikimedia.org/T260627 (10Dzahn) [22:58:20] 10DBA, 10Operations, 10Parsoid, 10serviceops, 10Parsoid-Tests: update mysql GRANTs for testreduce - https://phabricator.wikimedia.org/T260627 (10Dzahn) ` [scandium:~] $ mysql -h m5-master.eqiad.wmnet -u testreduce -p testreduce Enter password: Reading table information for completion of table and column... [23:08:48] 10DBA, 10Operations, 10Parsoid, 10serviceops, 10Parsoid-Tests: update mysql GRANTs for testreduce - https://phabricator.wikimedia.org/T260627 (10Dzahn) So this is everything in `modules/role/templates/mariadb/grants/production-m5.sql.erb` that refers to testreduce (line 5 to 48). Please make that work t...