[02:54:27] 10DBA, 10Performance-Team: Database for XHGui profiles - https://phabricator.wikimedia.org/T254795 (10dpifke) For testing, I've created a new MariaDB host in the beta cluster: deployment-mdb01.deployment-prep.eqiad.wmflabs. On there, I manually created the xhgui user and database, and verified I can connect t... [04:44:15] 10DBA, 10Performance-Team: Database for XHGui profiles - https://phabricator.wikimedia.org/T254795 (10Marostegui) Excellent - let's resume this once you are back. Enjoy your days off! [04:55:34] 10Blocked-on-schema-change, 10DBA: CentralNotice: Update DB schema on Meta for new features - https://phabricator.wikimedia.org/T254371 (10Marostegui) 05Stalled→03Open Thank you! [04:56:52] 10Blocked-on-schema-change, 10DBA: CentralNotice: Update DB schema on Meta for new features - https://phabricator.wikimedia.org/T254371 (10Marostegui) [05:06:57] 10Blocked-on-schema-change, 10DBA: CentralNotice: Update DB schema on Meta for new features - https://phabricator.wikimedia.org/T254371 (10Marostegui) @AndyRussG the first task description does not match the second edit, so I am unsure about which alters to apply. The first one pointed to: 1- https://gerrit.w... [05:09:17] 10DBA, 10Growth-Team, 10MediaWiki-Recent-changes, 10Schema-change: recentchanges table indexes: tmp1, tmp2 and tmp3 - https://phabricator.wikimedia.org/T206103 (10Marostegui) [07:04:43] 10DBA: Upgrade x1 databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254871 (10Marostegui) [07:11:55] 10DBA, 10Growth-Team, 10MediaWiki-Recent-changes, 10Schema-change: recentchanges table indexes: tmp1, tmp2 and tmp3 - https://phabricator.wikimedia.org/T206103 (10Marostegui) I am altering s3 directly on the master, with a big sleep between alters. This will take around 30 hours to complete. [08:03:08] 10DBA: Upgrade x1 databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254871 (10jcrespo) >>! In T254871#6205682, @Marostegui wrote: > @jcrespo would you be okay if I upgrade db1095 and db2101 to Buster? Those are backup sources. If you upgrade them, snapshots may not work, as dbprov hosts... [08:18:06] 10DBA: Upgrade x1 databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254871 (10jcrespo) > They also host s2 and s3 (db1095). I can migrate x1 to db1140, which is already on buster, solving the extra sections issue. But not sure how to go about dbprov hosts. [08:51:01] marostegui, jynus: I'd like to reimage cumin2001 to Buster next Monday, are there any DB maintenance or backup jobs which would interfere with that? [08:52:27] backups for codfw are scheduled there, but whatever is ongoing will just fail and run again, just ping whenever you start so I am aware [08:52:45] moritzm: from my side there's nothing [08:53:27] jynus: ok, will do [08:53:39] marostegui: ack, thanks, will send an announcemnt to ops@ later the day [08:54:11] any DB dependency that might need rebuilding for buster? custom deb packages, scripts that do not support Py 3.7, etc... [08:54:30] we fixed/built those in advance [08:55:07] we are actually waiting for the upgrade to happen so python has good support of mysql8 [08:57:33] as long as cumin api is the same, the other things have been already tested [08:58:56] nothing major changed, release notes are here anyway: https://doc.wikimedia.org/cumin/master/release.html#v4-0-0rc1-2020-06-09 [08:59:51] is it possible to support both a buster and a stretch cumin version with the same client code? [09:00:46] AFAIK there is no API breakage between the two versions [09:01:01] ok [09:01:16] the puppetdb backend behaviour is slightly different in some special cases to support native types [09:01:58] in any case, there is no big logic on cumin, everthing there is just client calls and crons [09:02:13] if for any reason somethin breaks we could move the logic elsewhere [09:02:26] 10DBA: Upgrade x1 databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254871 (10Marostegui) I don't really know how to proceed :-( I was looking for ideas, should maybe upgrade more hosts in s2 and s3 so it it "worth" upgrading db1095 and dbprov? [09:02:29] you run your script in a venv or using system libraries? [09:02:40] system [09:02:52] ok, then most likely nothing to touch [09:10:33] marostegui: ok to use db1077 for a test myloader? [09:10:50] jynus: I think kormat is using db1077 at the moment [09:10:55] ok [09:11:03] will wait/ask him [09:11:03] it's currently reimaging [09:11:26] are you using it for a task, if yes, I will search another? [09:11:49] jynus: i've been using it to test reuse-parts, [09:11:53] but this run should be the last one [09:12:12] ok, I just need it 20 minutes for a test load [09:13:30] I could use db2102, but that is in codfw and I have the data on eqiad :-( [09:13:53] jynus: yeah that's fine. this should be done in 10mins [09:14:00] i'll let you know [09:14:04] I can wait, no worries [09:14:06] thanks [09:16:03] 10DBA, 10Patch-For-Review: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db1103.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202006100915_marostegui_8666... [09:20:18] jynus: alright, db1077 is back up, and available for you. [09:21:06] jynus: can you stop the event_scheduler for your test and leave it stopped? [09:21:12] once you are done, let me know, so I can resume my tests [09:21:23] ok [09:22:30] db1077 set global event_scheduler=0; / Variable_name: event_scheduler / Value: OFF [09:22:48] will test import and report here when finished [09:23:20] thanks :* [09:23:34] 10DBA: Make checksum parallel to the data transfer in transferpy package - https://phabricator.wikimedia.org/T254979 (10Privacybatm) [09:24:57] 10DBA: Make checksum parallel to the data transfer in transferpy package - https://phabricator.wikimedia.org/T254979 (10Privacybatm) I would like to calculate the checksum for the actual tarred file. We can do this parallel to transfer like this: At sender: `tar cf - | tee >(echo $(md5sum) > /tmp/tr... [09:26:06] marostegui: reimaging db1077 with reuse-parts is successful (just requires https://gerrit.wikimedia.org/r/c/operations/puppet/+/604315 to fix the recipe) [09:26:26] kormat: <3 <3 [09:26:35] should I start using it once that is merged? [09:26:47] i think so, yeah [09:26:49] I won't use it with dbstore though (lots of data to recover in case it fails), so I would start with other hosts [09:27:01] SGTM [09:30:01] there's one thing i want to do before i'm finally done with this, which is to add useful ui feedback to the user if something fails [09:30:08] which is tricky to do, but.. [09:30:38] 10DBA: Make checksum parallel to the data transfer in transferpy package - https://phabricator.wikimedia.org/T254979 (10jcrespo) I think it is a good starting point- I suggest you do some benchmarking (doesn't need to be implemented on code yet) of how much expensive this strategy would be compared to the curren... [09:31:43] ^ marostegui this may be interesting for you to have a look [09:32:19] wilco! [09:32:38] 10DBA, 10Patch-For-Review: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1103.eqiad.wmnet'] ` and were **ALL** successful. [09:33:08] 10-4 :-D [09:33:12] XDDD [09:40:29] 10DBA: Make checksum parallel to the data transfer in transferpy package - https://phabricator.wikimedia.org/T254979 (10Privacybatm) >>! In T254979#6210033, @jcrespo wrote: > I think it is a good starting point- I suggest you do some benchmarking (doesn't need to be implemented on code yet) of how much expensive... [09:46:40] 10DBA, 10Wiktionary: Cannot connect to database on nl.wiktionary - https://phabricator.wikimedia.org/T254987 (10AlexisJazz) [09:47:10] 10DBA, 10Wiktionary: Cannot connect to database on nl.wiktionary - https://phabricator.wikimedia.org/T254987 (10Marostegui) 05Open→03Resolved a:03Marostegui This was fixed already - and several users confirmed fixed. Thanks for reporting! [09:48:23] 10DBA: Upgrade x1 databases to Buster and Mariadb 10.4 - https://phabricator.wikimedia.org/T254871 (10Marostegui) [09:58:48] db1077 test done and data removed [09:58:58] won't use it anymore [09:59:13] but I am about to load data into m2, heads up for the log and possible lag [10:00:01] ok, I will take db1077 back then [10:02:48] 10DBA, 10Patch-For-Review: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10Marostegui) [10:05:59] load ran, created 1 minute of lag or so on the passive hosts [10:34:51] 10DBA, 10Patch-For-Review: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10Marostegui) [10:39:33] 10DBA, 10Gerrit: Get a writable reviewdb clone to test Gerrit upgrade with - https://phabricator.wikimedia.org/T254516 (10jcrespo) 05Open→03Resolved A reviewdb dump from `2020-06-09 00:00:01` has been loaded into m2, under the reviewdb-test database name. A separate pair of user & password has been created... [10:42:51] 10DBA, 10Patch-For-Review: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db1127.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202006101042_marostegui_1562... [10:45:06] 10DBA, 10Gerrit: Get a writable reviewdb clone to test Gerrit upgrade with - https://phabricator.wikimedia.org/T254516 (10Marostegui) Thank you for handling this. Is it worth adding that user to the .sql files we have in puppet to track users? [10:45:55] 10DBA, 10Gerrit: Get a writable reviewdb clone to test Gerrit upgrade with - https://phabricator.wikimedia.org/T254516 (10jcrespo) > Is it worth adding that user to the .sql files we have in puppet to track users? I intended not to do that as I was told this would only need a week. [10:48:01] 10DBA, 10Gerrit: Get a writable reviewdb clone to test Gerrit upgrade with - https://phabricator.wikimedia.org/T254516 (10Marostegui) Sounds good! [11:00:04] kormat: root@db1127:/srv/sqldata# du -sh . [11:00:04] 228G . [11:00:05] :) [11:00:08] it went well! [11:00:11] congratulations [11:00:20] \m/ [11:01:14] 10DBA, 10Patch-For-Review: Relocate "old" s4 hosts - https://phabricator.wikimedia.org/T253217 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1127.eqiad.wmnet'] ` and were **ALL** successful. [11:07:16] 10DBA: Make checksum parallel to the data transfer in transferpy package - https://phabricator.wikimedia.org/T254979 (10jcrespo) > It will be great if I can get a pair of virtual machines. I will ask what I can get you. > If netcat results in error, any way the transfer should also fail right? (I don't know... [11:07:55] 10DBA, 10Gerrit: Get a writable reviewdb clone to test Gerrit upgrade with - https://phabricator.wikimedia.org/T254516 (10Dzahn) This also needs a puppet change to allow changing the database name in Hiera and the password needs to be added to the private repo. Otherwise @QChris would have to have puppet disa... [11:11:43] 10DBA, 10Patch-For-Review: Package transferpy framework under wmfmariadbpy - https://phabricator.wikimedia.org/T253736 (10jcrespo) Cumin 4 for buster is packaged in an rc status: https://phabricator.wikimedia.org/T245114#6207237 This may help with dependencies? I was told api should be compatible with cumin... [11:13:38] 10Blocked-on-schema-change, 10DBA, 10MachineVision, 10SDC-Statements (Machine-vision-depicts), and 2 others: Review & apply schema changes for T250748 - https://phabricator.wikimedia.org/T255003 (10matthiasmullie) [11:22:11] 10Blocked-on-schema-change, 10DBA, 10MachineVision, 10SDC-Statements (Machine-vision-depicts), and 2 others: Review & apply schema changes for T250748 - https://phabricator.wikimedia.org/T255003 (10Marostegui) p:05Triage→03Medium That UPDATE seems too big to actually do it at once (2M rows or so), plea... [11:24:16] 10Blocked-on-schema-change, 10DBA, 10MachineVision, 10SDC-Statements (Machine-vision-depicts), and 2 others: Review & apply schema changes for T250748 - https://phabricator.wikimedia.org/T255003 (10Marostegui) [11:24:56] 10Blocked-on-schema-change, 10DBA, 10MachineVision, 10SDC-Statements (Machine-vision-depicts), and 2 others: Review & apply schema changes for T250748 - https://phabricator.wikimedia.org/T255003 (10Marostegui) [11:40:23] 10Blocked-on-schema-change, 10DBA, 10MachineVision, 10SDC-Statements (Machine-vision-depicts), and 2 others: Review & apply schema changes for T250748 - https://phabricator.wikimedia.org/T255003 (10Marostegui) This is how the tables looks like in `testcommonswiki`, please confirm if this is what you expect... [11:49:29] 10Blocked-on-schema-change, 10DBA, 10MachineVision, 10SDC-Statements (Machine-vision-depicts), and 2 others: Review & apply schema changes for T250748 - https://phabricator.wikimedia.org/T255003 (10matthiasmullie) That looks good! [11:50:19] 10Blocked-on-schema-change, 10DBA, 10MachineVision, 10SDC-Statements (Machine-vision-depicts), and 2 others: Review & apply schema changes for T250748 - https://phabricator.wikimedia.org/T255003 (10Marostegui) [12:16:06] 10Blocked-on-schema-change, 10DBA, 10MachineVision, 10SDC-Statements (Machine-vision-depicts), and 2 others: Review & apply schema changes for T250748 - https://phabricator.wikimedia.org/T255003 (10Marostegui) I have altered codfw already, so we can do some tests there. This query looks good: ` SELECT mvi_... [12:20:38] 10Blocked-on-schema-change, 10DBA, 10MachineVision, 10SDC-Statements (Machine-vision-depicts), and 2 others: Review & apply schema changes for T250748 - https://phabricator.wikimedia.org/T255003 (10Marostegui) [12:31:19] 10DBA: Make checksum parallel to the data transfer in transferpy package - https://phabricator.wikimedia.org/T254979 (10Privacybatm) Oh okay, how about giving the user a choice? - Checksum parallel to transfer (document the issues we find at testing) - Checksum after the transfer (document the delay issues) [12:38:07] 10DBA, 10Patch-For-Review: Productionize db114[1-9] - https://phabricator.wikimedia.org/T252512 (10Marostegui) @jcrespo can you take care of productionizing db1145? It will be the one to replace db1102 (whenever we feel comfortable having a backupsource running 10.4). For now I would like to have it installed,... [12:39:30] 10DBA, 10Patch-For-Review: Productionize db114[1-9] - https://phabricator.wikimedia.org/T252512 (10Marostegui) Apart from the above, I would also feel fine if you want to install it with Stretch+mariadb 10.1 so we can remove db1102 entirely and I can relocate it somewhere else (T253217) [13:17:58] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) @Reedy did you hear back from Legal about this? [13:18:57] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Reedy) >>! In T253802#6210946, @Huji wrote: > @Reedy did you hear back from Legal ab... [13:26:53] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) >>! In T253802#6210953, @Reedy wrote: >>>! In T253802#6210946, @Huji wrote: >>... [13:31:13] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Reedy) >>! In T253802#6210974, @Huji wrote: >>>! In T253802#6210953, @Reedy wrote: >... [13:33:52] 10DBA, 10CheckUser, 10Trust-and-Safety, 10WMF-Legal, and 2 others: Set wgCheckUserLogLogins to true on WMF wikis to log successful and unsuccessful login attempts in CheckUser - https://phabricator.wikimedia.org/T253802 (10Huji) Understandable. Just wanted to make sure we have reasonable expectations. This... [13:48:17] 10Blocked-on-schema-change, 10DBA: CentralNotice: Update DB schema on Meta for new features - https://phabricator.wikimedia.org/T254371 (10AndyRussG) >>! In T254371#6208638, @Marostegui wrote: > So the schema changes to `cn_notice_log` aren't there on the second edit, is that intended? Ooops! No, I'll update... [13:52:33] 10Blocked-on-schema-change, 10DBA: CentralNotice: Update DB schema on Meta for new features - https://phabricator.wikimedia.org/T254371 (10AndyRussG) [13:54:06] 10Blocked-on-schema-change, 10DBA: CentralNotice: Update DB schema on Meta for new features - https://phabricator.wikimedia.org/T254371 (10Marostegui) >>! In T254371#6211038, @AndyRussG wrote: >>>! In T254371#6208638, @Marostegui wrote: >> So the schema changes to `cn_notice_log` aren't there on the second edi... [17:20:28] 10DBA, 10Patch-For-Review: Package transferpy framework under wmfmariadbpy - https://phabricator.wikimedia.org/T253736 (10Privacybatm) Thank you for the information, and yes, it was helpful :-)