[05:27:28] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db2106.codfw.wmnet'] ` The log can be found in `/var/log/wmf... [05:52:23] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db2106.codfw.wmnet'] ` and were **ALL** successful. [06:32:07] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db2117.codfw.wmnet'] ` The log can be found in `/var/log/wmf... [06:56:43] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db2117.codfw.wmnet'] ` and were **ALL** successful. [07:35:35] do you have time for a 5 minute unschedled meeting? [07:35:49] me? [07:35:56] both or any [07:35:59] I do [07:36:30] I want to tell you a think I discovered with analytics, but it will take me much less time on voice [07:36:35] *thing [07:36:43] probably less than 5 minutes [07:36:55] sure, send me the meeting invite [07:39:02] let me prepare my notes, one second [07:39:09] sure [07:42:27] i'm around [07:42:40] if you haven't started already, i'll take an invite [07:43:15] we haven't yet [07:43:31] (damn) [07:44:47] ok, I think I found the ticket [07:46:51] Amir1: I didn't realise revision was _so_ big on wikidata -rw-rw---- 1 mysql mysql 239G Jul 29 07:46 revision.ibd [07:47:13] I sent you the invites [08:07:04] apparently mysql is not the only service that may struggle with ipv6 T253986 [08:07:04] T253986: update bacula-sd config so that it listens on IPv6 - https://phabricator.wikimedia.org/T253986 [08:07:26] uf bacula... [08:43:44] 10DBA, 10Quibble: Optimize MySQL settings for MediaWiki CI / Quibble - https://phabricator.wikimedia.org/T218196 (10hashar) >>! In T218196#6314391, @Marostegui wrote: > @hashar is this still needed? I completely forgot about this task, but yes it would be nice to have better MySQL settings in the test/CI cont... [08:57:32] https://wikitech.wikimedia.org/wiki/MariaDB#IPv6_and_MariaDB [08:58:10] jynus: thank you :))) [08:59:13] jynus: if you write "as of writing", please add a date or something to that so a reader has an idea on how outdated it is [08:59:41] ok [08:59:45] good catch [09:26:14] 10DBA, 10Patch-For-Review, 10User-Urbanecm: Move muswiki and mhwiktionary (closed wikis) from s3 to s5 - https://phabricator.wikimedia.org/T259004 (10Marostegui) Wow, thanks for the tests @Urbanecm, very complete and very detailed info. This is very helpful. I think the next step would be to try to change d... [10:36:57] 10DBA, 10Operations, 10User-Kormat: Package wmfmariadbpy as a .deb - https://phabricator.wikimedia.org/T259021 (10Kormat) There are a number of scripts (i.e. executable python scripts) in this repo, but i'm not sure which ones are actively used or not: I know these are used: ` switchover.py replication_tree... [10:37:13] jynus: i'm hoping you can help answer this ^ [10:41:32] 10DBA, 10Operations, 10User-Kormat: Package wmfmariadbpy as a .deb - https://phabricator.wikimedia.org/T259021 (10Marostegui) So: these are definitely used: ` compare.py mysql.py backup_mariadb.py osc_host.py ` [10:58:52] 10DBA, 10Operations, 10User-Kormat: Package wmfmariadbpy as a .deb - https://phabricator.wikimedia.org/T259021 (10jcrespo) My intention is to put backup_mariadb.py and its dependencies (remote execution, etc.) on a separate package (that is why I showed you the https://gerrit.wikimedia.org/r/c/operations/sof... [11:00:30] ^sorry if the answer is not an easy yes or no 0:-( [11:17:18] 10DBA, 10Operations, 10User-Kormat: Package wmfmariadbpy as a .deb - https://phabricator.wikimedia.org/T259021 (10jcrespo) Let me go a bit overboard an propose you the following to be added to a potential wmfmariadbpy package- to be installed on cumin hosts: * Libraries: WMFMariaDB, WMFReplication (I believ... [11:19:01] 10DBA, 10Operations, 10User-Kormat: Package wmfmariadbpy as a .deb - https://phabricator.wikimedia.org/T259021 (10jcrespo) BTW, this is duplicate of 3yo T165358 :-). [11:19:33] 10DBA: Set up and package wmfmariadbpy helper scripts so they can easily be deployed to all database server and client hosts - https://phabricator.wikimedia.org/T165358 (10jcrespo) [11:19:35] 10DBA, 10Operations, 10User-Kormat: Package wmfmariadbpy as a .deb - https://phabricator.wikimedia.org/T259021 (10jcrespo) [11:22:29] 10DBA, 10Operations, 10User-Kormat: Package wmfmariadbpy as a .deb - https://phabricator.wikimedia.org/T259021 (10Marostegui) >>! In T259021#6344214, @jcrespo wrote: > My intention is to put backup_mariadb.py and its dependencies (remote execution, etc.) on a separate package (that is why I showed you the ht... [11:24:45] So looks like the web role for labsdb is no longer able to serve things with just one host [11:24:58] I had to finally decrease labsdb1009 load to 1 and set labsdb1010 to 1 as well [11:25:02] otherwise labsdb1009 gets lag :( [11:25:03] but wasn't that also true of analytics? [11:25:19] yeah, but web used to be able to be on its own [11:25:31] so technically, we no longer have redundancy? [11:25:34] analytics had a lot more heavy queries [11:25:42] web used to be fine [11:26:06] I am glad we are advancing on the new host procurement [11:32:43] I am going to make T259007 open following transparency good practices [11:32:55] I don't see open private info there except for pastes [11:33:06] any issues? [11:42:36] marostegui: hey Manuel, I'm sorry, but I don't understand what you mean by https://phabricator.wikimedia.org/T259004#6344026. What exactly do you want me to do? [11:51:48] Urbanecm: Not sure if you can do that, it would be mostly me and/or Amir1 I think [11:52:13] Basically just use mwdebug2001 with the change you made to test "live" in codfw that the site doesn't break [11:52:22] it didn't break on Beta, but I want to make sure it doesn't break in live [11:53:16] marostegui: I didn't touch db-labs.php at all - the flag used to make a wiki read only is touched in InitialiseSettings.php [11:53:41] (and actually, in production, it's even easier to test that than in beta - just install the WikimediaDebug extension, and check "set wiki to read only") [11:53:43] Urbanecm: ah, ok that one yeah, basically the same thing you did but on mwdebug200 and browse codfw hosts [11:53:49] gotcha [11:54:36] it's possible to easily turn the flag on via X-Wikimedia-Debug header through https://usercontent.irccloud-cdn.com/file/Jluwn7VG/image.png [11:54:52] Urbanecm: yeah, but I want to test just for an specific wiki, like you did [11:55:25] i see [11:55:54] just to be fully sure we won't break anything [11:56:11] I understand. I can do that as well :-) - thanks for explaining [11:57:51] ah if you can do that, that's awesome then! [11:57:55] much appreciated [11:58:40] marostegui: through I can't see last writes in production in the same way as I did for beta, because I don't have shell to db hosts. Is there a way to see last write from mysql console? [11:58:55] you could just query the logging table [11:59:01] which is where I saw the writes mostly [11:59:10] okay, that works too - thanks [11:59:15] no, thank you! [12:32:18] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for avkwiki - https://phabricator.wikimedia.org/T258077 (10Urbanecm) The database was just created (on s5, as requested). [12:32:43] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for avkwiki - https://phabricator.wikimedia.org/T258077 (10Marostegui) a:03Marostegui Thanks - I will sanitize it [14:00:01] 10DBA, 10wikitech.wikimedia.org: Move database for wikitech (labswiki) to a main cluster section - https://phabricator.wikimedia.org/T167973 (10Marostegui) @Andrew I have double checked with Chris Danis and looks like we can set wikitech on RO via `dbctl` as any other section. Do you think 1h RO time for wikit... [15:35:22] do we know why the WMCS replicas are so backed up? s1 is lagged by ~43 hours https://replag.toolforge.org/ [15:36:51] musikanimal: I think there is some transient maintenance on production that is affecting wmcs [15:37:29] musikanimal, look at: https://dbtree.wikimedia.org/ [15:37:55] ah I see. Okay so it's expected, then [15:38:12] sorry for the impact, some of those are very difficult to avoid [15:38:54] as cloud wikireplcias both suffer from production AND cloud maintenances [15:44:15] musikanimal: yep, there is maintenance on-going on their master [15:44:28] musikanimal: I expect it to last at around 48h more [15:45:50] ok thanks, good to have a timeline [15:47:04] musikanimal: also expect lag on s4 (commonswiki) as their master is now executing the MCR schema change on the revision table, which takes around 12-14h [15:48:12] 10DBA, 10Patch-For-Review, 10User-Urbanecm: Move muswiki and mhwiktionary (closed wikis) from s3 to s5 - https://phabricator.wikimedia.org/T259004 (10Urbanecm) I have repeated my beta tests from T259004#6341365, and I confirm production doesn't do any database writes in cases I tested for beta. Furthermore,... [15:52:19] 10DBA, 10Patch-For-Review, 10User-Urbanecm: Move muswiki and mhwiktionary (closed wikis) from s3 to s5 - https://phabricator.wikimedia.org/T259004 (10Urbanecm) @Marostegui I believe all necessary tests were done. If there is anything I forgot to test and should test, let me know. [16:18:53] 10DBA, 10wikitech.wikimedia.org: Move database for wikitech (labswiki) to a main cluster section - https://phabricator.wikimedia.org/T167973 (10bd808) >>! In T167973#6344856, @Marostegui wrote: > That looks independent from the database itself, so that would keep working? We can migrate even manually the entri... [16:19:51] 10DBA, 10wikitech.wikimedia.org: Move database for wikitech (labswiki) to a main cluster section - https://phabricator.wikimedia.org/T167973 (10Andrew) One hour RO sounds just fine. Two would also be fine :) [18:50:13] 10DBA, 10Patch-For-Review, 10User-Urbanecm: Move muswiki and mhwiktionary (closed wikis) from s3 to s5 - https://phabricator.wikimedia.org/T259004 (10BPirkle) See in logstash on 2020-07-29: [{exception_id}] {exception_url} Wikimedia\DependencyStore\DependencyStoreException from line 113 of /srv/mediawiki/ph... [19:43:55] jynus marostegui is there a task for the db maintenance causing the replication lag? Just so we have somewhere to send people for more info [19:50:48] I'm guessing it's https://phabricator.wikimedia.org/T238966 [20:03:24] 10DBA, 10Patch-For-Review, 10User-Urbanecm: Move muswiki and mhwiktionary (closed wikis) from s3 to s5 - https://phabricator.wikimedia.org/T259004 (10Urbanecm) @BPirkle Hello, yes, that is an expected error. We were testing if does wgReadOnly prevent all writes, or just some, to help preparation in moving tw... [20:42:08] 10DBA, 10MediaWiki-Page-derived-data, 10Platform Engineering, 10TechCom-RFC, and 2 others: RFC: Normalize MediaWiki link tables - https://phabricator.wikimedia.org/T222224 (10Krinkle)