[05:31:57] 10DBA, 10Cloud-Services, 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09), 10Platform Team Initiatives (MCR Schema Migration), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) [05:32:29] 10DBA, 10Cloud-Services, 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09), 10Platform Team Initiatives (MCR Schema Migration), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) The schema change on s4 (com... [08:27:54] 10DBA, 10Patch-For-Review: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db1107.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202007230827_m... [08:46:50] 10DBA, 10Patch-For-Review: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1107.eqiad.wmnet'] ` and were **ALL** successful. [09:15:33] 10DBA, 10Cloud-Services, 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09), 10Platform Team Initiatives (MCR Schema Migration), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) s4 eqiad progress [] labsdb... [09:50:58] 10DBA, 10Patch-For-Review: Create more tests for transferpy package - https://phabricator.wikimedia.org/T257600 (10jcrespo) The transferpy unit test file is getting much larger. If there is enough time, it would be nice to split, where possible, the different classes on different files, to keep individual file... [10:02:28] 10DBA, 10Patch-For-Review: Create more tests for transferpy package - https://phabricator.wikimedia.org/T257600 (10Privacybatm) I agree on this, As you suggested we will split it based on class. [10:12:56] jynus: I've got a question about the implentation of transfer.py --type=xtrabackup [10:13:18] sure, ask away [10:13:20] I have copied the data from db1117 using transfer.py --no-checksum --no-encrypt --type=xtrabackup db1117.eqiad.wmnet:/run/mysqld/mysqld.m2.sock db1107.eqiad.wmnet:/srv/sqldata [10:14:09] I am checking xtrabackup_slave_info after the copy [10:14:30] after or before prepare? [10:14:34] after [10:14:57] what's the issue? [10:16:00] db1117 is 10.4 so it has to be prepared on a 10.4 host, but I am guessing that is not the issue [10:16:38] jynus: no no, no issue, I was wondering if that file ever contained the master's ip [10:16:48] In previous versions or something [10:16:49] not really [10:17:20] the plan was to store that with a higher level tool so autoprovisioning is easy [10:17:26] however [10:17:42] it may be possible to get from the binlog/gtid indirectly [10:17:53] the work on more automation for provisioning is stopped [10:18:08] yeah, I just cat xtrabackup_slave_info | mysql [10:18:12] and then set upthe master ip and all that [10:18:22] but I was doubting if xtrabackup ever contained the master IP [10:18:28] Even in previous versions [10:18:42] if the question is if it was there, I think not, at least not for mariabackup [10:18:59] for xtrabackup I cannot say [10:18:59] yeah, that question is whether you remember it being there years ago even [10:19:08] maybe when gtid was not in use [10:19:11] cannot say [10:19:18] but it has not changed for 10.4 [10:19:37] but when we started using xtrabackup seriously it was already not there [10:19:53] it may have been for xtrabackup and non-gtid [10:20:03] in fact you can try yourself [10:20:05] Sure, it was more curiosity on whether I was recalling that correctly or not [10:20:08] setup a database without gtid [10:20:17] and I think it output automatically bilog coords [10:20:24] I think it did that for labsdb [10:20:29] yeah, it does for sure [10:20:35] but I was wondering about the master's ip and all that info [10:20:38] but I am not sure if it printed the ip [10:20:46] I don't think it does, as it doesn't know the ip [10:20:50] at most it could print the dns [10:20:59] as we use dns for master change [10:21:31] question, is this something that bother you for automation, or it is just you wondering? [10:21:35] no no [10:21:37] just wondering [10:21:44] as I was inspecting the file to start replication [10:21:47] because if it is the first this is something that will be workarounded [10:21:56] no, just curiosity [10:22:01] either by inspecting coords [10:22:14] or by storeing the current master at the time of the backup, but on higher level tools [10:22:16] it is ok, no need to research further, it was just curiosity [10:22:25] ah [10:22:32] I think I know what your confussion is [10:22:40] for some reason I thought xtrabackup in the past did print master's ip [10:22:53] we run backups with master coord print [10:23:10] we do not run it with current coord print [10:23:19] as many of those hosts don't have binlog [10:23:29] the format of current host may be slightly different [10:23:32] yeah, you mean that xtrabackup_slave_info are the coordinates from the master? [10:23:35] not sure if you get what I meant [10:23:48] let me find the actual name of the option [10:23:54] sure [10:24:22] basically cat xtrabackup_slave_info | mysql and then set up master_host, master_port, master_user, master_password is enough for our use case [10:24:33] as gtid is global then yeah [10:25:27] so there is galera-info [10:25:32] slave-info [10:26:03] and there is a third one [10:26:14] binlog-info [10:26:45] maybe you can test binlog-info on a test host with binlogs and maybe that was what you remembered? [10:27:12] either that or non-gtid instance may be what you find different? [10:27:33] yeah, I think with non gtid it was different [10:27:38] I will test at some point yeah [10:27:41] or could be even the master being printed by mysqldump? [10:27:45] or mydumper? [10:27:50] nah, no need [10:27:52] they are all similary options [10:27:54] it is really just two commands [10:27:58] cat file | mysql [10:28:01] and then set up the change_master [10:28:31] well, yeah, but the point is in the future to just have to do ./provision dbXXXX [10:28:43] yeah, in that case we do need to get the master's ip [10:28:47] don't worry, that will arrive [10:28:50] port and user, pass [10:28:57] it is not designed to be as complicated as now [10:29:02] yep [10:29:03] 0:-) [10:29:17] remember transfer.py is just a low level tool [10:29:37] in fact, maybe for a clone, you could use remote_mariadb_backup.py [10:29:45] which does the prepare automatically for you [10:30:14] yeah [10:30:21] the prepare is super fast for m2 anyways [10:30:46] I hapened to have by the way a recent snapshot of m2 [10:31:04] due to otrs ticket [10:31:30] but it is true we don't generally generate snapshots of m* hosts [10:31:34] maybe we should... [10:32:12] yeah, it wouldn't hurt I think [10:32:15] do we have space? [10:33:58] well, I was thinking of reconfiguring some things [10:34:03] plus the new host will help [10:34:33] yeah, it is low priority (if we wanted to do it) [10:34:50] I think the only blocker is otrs being quite large [10:34:58] they others are doable [10:35:37] but I realized that gaps can be frequent on dumps, as they are manually setup [10:53:49] 10DBA: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10Marostegui) [10:56:43] 10DBA, 10Patch-For-Review: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10Marostegui) [11:09:55] 10DBA, 10Patch-For-Review: Solve transferpy concurrency issue with auto port detection and checksum file names - https://phabricator.wikimedia.org/T256450 (10jcrespo) Almost ready to be closed- no more known concurrency issues. [11:10:44] I didn't setup db2102 prometheus exporter well after wiping it, will have a look at it later [11:11:34] marostegui: could be that you got confused the xtrabackup file with the metadata one from mydumper? [11:11:55] metadata does store the source db dns on the backup file [11:12:14] SHOW SLAVE STATUS:\nHost: db2135.codfw.wmnet [11:12:28] jynus: ah maybe yeah [11:12:48] mydumpet does show the ip [11:12:50] no? [11:12:56] the dns [11:13:22] we use dns for setting up replication [11:13:50] maybe you had a custom xtrabackup install at a previous life (it could be)? [11:14:02] I am not familiar with mariabackup enterprise [11:14:20] no, I think it is the mydumper confusion, as I have been looking more at mydumper than xtrabackup lately [11:15:07] it wouldn't be too complicated to add it to mariabackup [11:15:21] we already have our own package to be able to do it [11:15:25] we could ask for it as a feature request :) [11:15:32] that, too [11:16:11] let me double check there is not already an option [12:49:38] 10DBA: Upgrade m2 to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T257540 (10Marostegui) The only thing pending is the master switchover. Going to wait a few days to make sure db1107 works ok and then I will go ahead and get a day scheduled [13:09:04] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) Wow, this is a big mess. I have compared a few wikis and the issues are indeed the PK being the UNIQUE and the UNIQUE being the PK on most wikis except wikidatawiki and the new on... [13:18:19] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) [13:21:20] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) [13:58:53] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) [14:14:13] 10DBA, 10Platform Team Initiatives (API Gateway): Review request for a new database table for OAuthRateLimiter - https://phabricator.wikimedia.org/T258711 (10Pchelolo) [14:16:00] jynus: would you mind taking care of that ^ I am already reviewing lots of new table schemas, and new features - I don't want to delay the above because I still have to be checking the other things. Do you think you'd have time for that? [14:17:27] cannot kormat take care and I or you overview? [14:17:59] kormat: do you feel comfortable reviewing a new table that'd go into production? [14:18:23] i have no idea on what basis to evaluate it [14:19:05] maybe I talk to him later and he prepares it but check with me before answering? [14:19:40] if you can do that, that'd be great. I just don't want to delay things even more than I am doing already [14:21:56] kormat, for now, whenevery you have time, 1) acknowledge the ask 2) request for privacy information of all rows (so they can be shown publicly on labs) and 3) mention that it has a foreign key and we have non currently on production, and that is an unusual feature [14:22:12] then promise a deeper look after they get back to us [14:22:21] <3 [14:22:26] thanks [14:22:40] kormat: we can have later a small discussion on things to review [14:22:50] probably tomorrow [14:23:02] s/tomorrow/not today/ [14:27:17] "privacy information of all rows" - do you mean all columns? [14:27:26] (otherwise i don't understand) [14:37:35] both :) [14:37:51] they need to tell us if there will be any private data in the table, ever :) [14:37:54] (IIRC) [14:38:20] because if so we need to prevent it from replicating in labs the private data [15:18:05] 10DBA, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway): Review request for a new database table for OAuthRateLimiter - https://phabricator.wikimedia.org/T258711 (10Kormat) Hey. There's a couple of comments/questions to start with: - Will this table contain private/sensitive information? If so w... [15:23:31] sorry, I was on a meeting, yeah, I meant columns/all data [15:23:45] as we can filter per table or per column [15:24:22] your answer was enough for now, let's see what they say [15:24:37] in reality we only care about the privacy party [15:25:03] the design issues probably there will be people with more experience than us [15:25:13] at platform team [15:25:38] but sometimes there are weird things like the fk that we can point out [15:25:55] we can talk tomorrow about typical design issues [15:36:33] 10DBA, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway): Review request for a new database table for OAuthRateLimiter - https://phabricator.wikimedia.org/T258711 (10Pchelolo) > Will this table contain private/sensitive information? If so we'll need to filter it so it doesn't end up publicly quer... [15:55:16] 10DBA, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway): Review request for a new database table for OAuthRateLimiter - https://phabricator.wikimedia.org/T258711 (10jcrespo) Following up as I think Kormat may be done for the day, so you don't have to wait an extra day: > so this one should be f... [16:03:36] 10DBA, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway): Review request for a new database table for OAuthRateLimiter - https://phabricator.wikimedia.org/T258711 (10Pchelolo) Thank you a lot @jcrespo. We will remove the FK to follow the current pattern, as for abstract schema - I'll consult with... [16:07:20] 10DBA, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway): Review request for a new database table for OAuthRateLimiter - https://phabricator.wikimedia.org/T258711 (10Reedy) >>! In T258711#6330075, @jcrespo wrote: > One last thing, I am not sure if new table creations should follow the new system,... [16:21:10] 10DBA, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway): Review request for a new database table for OAuthRateLimiter - https://phabricator.wikimedia.org/T258711 (10Ladsgroup) I'm doing the migration to abstract schema (so far 6 tables have been moved) and wrote [[https://www.mediawiki.org/wiki/... [17:37:21] 10DBA, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway): Review request for a new database table for OAuthRateLimiter - https://phabricator.wikimedia.org/T258711 (10Pchelolo) The extension now uses abstract schema - thus the foreign keys and abstract schemas concert were addressed. As for filt... [17:44:58] 10DBA, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway): Review request for a new database table for OAuthRateLimiter - https://phabricator.wikimedia.org/T258711 (10jcrespo) > os would you ping us when it's done? ^This, as I mentioned before. Likely to be done by tomorrow. [19:19:45] 10DBA, 10MediaWiki-extensions-OAuthRateLimiter, 10Patch-For-Review, 10Platform Team Initiatives (API Gateway): Review request for a new database table for OAuthRateLimiter - https://phabricator.wikimedia.org/T258711 (10Pchelolo) [20:17:41] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 2020-09-30) rack/setup/install dbprov2003.codfw.wmnet - https://phabricator.wikimedia.org/T258749 (10RobH) [20:18:49] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 2020-09-30) rack/setup/install dbprov2003.codfw.wmnet - https://phabricator.wikimedia.org/T258749 (10RobH) [20:20:26] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: () rack/setup/install dbprov1003.eqiad.wmnet - https://phabricator.wikimedia.org/T258750 (10RobH) [20:20:36] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: () rack/setup/install dbprov1003.eqiad.wmnet - https://phabricator.wikimedia.org/T258750 (10RobH) [20:21:16] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (2020-09-30) rack/setup/install dbprov1003.eqiad.wmnet - https://phabricator.wikimedia.org/T258750 (10RobH) [20:22:46] 10DBA, 10DC-Ops, 10Operations, 10ops-codfw: (Need By: 2020-09-14) rack/setup/install dbprov2003.codfw.wmnet - https://phabricator.wikimedia.org/T258749 (10RobH) [20:22:52] 10DBA, 10DC-Ops, 10Operations, 10ops-eqiad: (2020-09-14) rack/setup/install dbprov1003.eqiad.wmnet - https://phabricator.wikimedia.org/T258750 (10RobH)