[05:13:25] musikanimal: correct, that is one, and another one is https://phabricator.wikimedia.org/T254462 which I expect to be finished with labs later today or tomorrow [05:21:21] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) [05:22:05] 10DBA: page_restrictions indexes have been majestically drifting from code - https://phabricator.wikimedia.org/T256682 (10Marostegui) 05Open→03Resolved a:03Marostegui All done [05:22:10] 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Sustainability (Incident Followup), 10WorkType-NewFunctionality: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459 (10Marostegui) [05:22:22] Amir1: if you want to run your drifts script again, feel free please :) [05:23:52] 10DBA, 10Cloud-Services, 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09), 10Platform Team Initiatives (MCR Schema Migration), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) s4 is complete, the master i... [05:24:19] 10DBA, 10Cloud-Services, 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09), 10Platform Team Initiatives (MCR Schema Migration), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) [05:27:38] 10DBA, 10Cloud-Services, 10MW-1.35-notes (1.35.0-wmf.36; 2020-06-09), 10Platform Team Initiatives (MCR Schema Migration), and 2 others: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966 (10Marostegui) [06:02:43] marostegui: Good morning, sure :D [06:08:07] this wouldn't include the abstract schema tables (currently 7, around 10%), I need to get around to make it use the abstract schema tables too, it's pretty easy actually [06:47:38] The only thing beside MCR so far is: [06:47:43] https://www.irccloud.com/pastebin/rUJh6huy/ [06:56:14] aaah https://phabricator.wikimedia.org/T256685#6325359 [07:10:39] Amir1: only that? niceeeeee [07:14:11] yup ^_^ it's 70% done, I wait for it to finish all but that's likely [07:14:40] 10DBA: es1017 IPMI issues - https://phabricator.wikimedia.org/T259218 (10Marostegui) [07:14:53] 10DBA: es1017 IPMI issues - https://phabricator.wikimedia.org/T259218 (10Marostegui) p:05Triage→03Low [07:18:25] marostegui: how do you feel about renaming `stop_in_sync.py` to `stop_nsync.py`? ;) [07:18:57] that's more confusing no? XD [07:19:35] depends on your feelings about boy bands :) [07:19:41] "bye bye bye" [07:19:41] haha [07:20:14] * godog back to lurking [07:20:29] godog: you have been observed [07:20:49] well played sir kormat [07:22:34] 10DBA, 10wikitech.wikimedia.org: Move database for wikitech (labswiki) to a main cluster section - https://phabricator.wikimedia.org/T167973 (10Marostegui) Excellent - thank you guys. I will get a procedure written and pasted here in a few days! [07:22:45] sir? [07:22:50] Clearly you don't know him [07:24:15] haha! [07:27:03] i concur :) [07:37:25] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db2083.codfw.wmnet'] ` The log can be found in `/var/log/wmf... [07:57:46] 10DBA, 10Epic, 10Patch-For-Review: Upgrade WMF database-and-backup-related hosts to buster - https://phabricator.wikimedia.org/T250666 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db2083.codfw.wmnet'] ` and were **ALL** successful. [08:00:47] https://www.irccloud.com/pastebin/e7UOsQjZ/ [08:01:01] marostegui: beside the MCR stuff, this is all ^ [08:02:16] kormat wants it that way [08:02:57] https://usercontent.irccloud-cdn.com/file/lgtmWKCL/image.png [08:07:28] Amir1: Thanks, I will review those! [08:08:34] these are similar to the ones we had (unique mismatch in *links tables) and in several s3 hosts, I assume half done in some wikis, not done in the rest because that's how s3 works :D [08:08:50] haha [08:08:51] yeah [08:09:12] Amir1: :D [08:35:38] 10DBA: Code and production differs on s3 on imagelinks table - https://phabricator.wikimedia.org/T259232 (10Marostegui) [08:35:50] 10DBA: Code and production differs on s3 on imagelinks table - https://phabricator.wikimedia.org/T259232 (10Marostegui) p:05Triage→03Medium [08:36:29] Amir1: ^ :) [08:37:31] marostegui: thanks. There is a similar thing with pagelinks and templatelinks too [08:37:55] yep, going to go for those too [08:38:02] Cool! [08:55:14] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for avkwiki - https://phabricator.wikimedia.org/T258077 (10Marostegui) a:05Marostegui→03None This has been executed on labsdb1009-labsdb1012: ` set session sql_log_bin=0; create database avkwiki_p; GRANT SELECT, SHOW... [09:00:28] 10DBA: Code and production differs on s3 on pagelinks table - https://phabricator.wikimedia.org/T259238 (10Marostegui) [09:00:39] 10DBA: Code and production differs on s3 on pagelinks table - https://phabricator.wikimedia.org/T259238 (10Marostegui) p:05Triage→03Medium [09:19:04] 10DBA: Code and production differs on s3 on templatelinks table - https://phabricator.wikimedia.org/T259241 (10Marostegui) [09:19:10] 10DBA: Code and production differs on s3 on templatelinks table - https://phabricator.wikimedia.org/T259241 (10Marostegui) p:05Triage→03Medium [09:30:28] Can I get some IP/DNS checking on https://gerrit.wikimedia.org/r/617402 ? [09:31:11] you can. [09:41:42] depends how much you're willing to pay :D [11:02:59] 10DBA, 10Patch-For-Review, 10User-Urbanecm: Move muswiki and mhwiktionary (closed wikis) from s3 to s5 - https://phabricator.wikimedia.org/T259004 (10Marostegui) Thank you again @Urbanecm for all the testing. With all those things tested, I think the procedure could be something like: - Downtime codfw s5 ho... [11:50:19] 10DBA, 10Patch-For-Review, 10User-Urbanecm: Move muswiki and mhwiktionary (closed wikis) from s3 to s5 - https://phabricator.wikimedia.org/T259004 (10Kormat) I'm wondering about removing the databases from s3 to ensure that any testing is hitting the right section. Also, are there MW caches that need to be c... [11:56:13] 10DBA, 10Patch-For-Review, 10User-Urbanecm: Move muswiki and mhwiktionary (closed wikis) from s3 to s5 - https://phabricator.wikimedia.org/T259004 (10Marostegui) >>! In T259004#6348318, @Kormat wrote: > I'm wondering about removing the databases from s3 to ensure that any testing is hitting the right section... [12:10:04] 10DBA, 10Patch-For-Review, 10User-Urbanecm: Move muswiki and mhwiktionary (closed wikis) from s3 to s5 - https://phabricator.wikimedia.org/T259004 (10Urbanecm) The cited procedure seems good to me MediaWiki-side, through the canonical source of dblist files is [muswiki.yaml](https://github.com/wikimedia/oper... [12:11:49] 10DBA, 10Patch-For-Review, 10User-Urbanecm: Move muswiki and mhwiktionary (closed wikis) from s3 to s5 - https://phabricator.wikimedia.org/T259004 (10Marostegui) >>! In T259004#6348342, @Urbanecm wrote: > The cited procedure seems good to me MediaWiki-side, through the canonical source of dblist files is [mu... [12:12:48] 10DBA, 10Patch-For-Review, 10User-Urbanecm: Move muswiki and mhwiktionary (closed wikis) from s3 to s5 - https://phabricator.wikimedia.org/T259004 (10Urbanecm) >>! In T259004#6348347, @Marostegui wrote: >>>! In T259004#6348342, @Urbanecm wrote: >> The cited procedure seems good to me MediaWiki-side, through... [12:13:26] 10DBA, 10Patch-For-Review, 10User-Urbanecm: Move muswiki and mhwiktionary (closed wikis) from s3 to s5 - https://phabricator.wikimedia.org/T259004 (10Marostegui) >>! In T259004#6348350, @Urbanecm wrote: >>>! In T259004#6348347, @Marostegui wrote: >>>>! In T259004#6348342, @Urbanecm wrote: >>> The cited proce... [12:22:37] 10DBA, 10Patch-For-Review, 10User-Urbanecm: Move muswiki and mhwiktionary (closed wikis) from s3 to s5 - https://phabricator.wikimedia.org/T259004 (10Reedy) You can just edit all three manually... and then CI will complain if you've not done it correctly ;P [14:21:04] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for lijwikisource - https://phabricator.wikimedia.org/T258389 (10Andrew) [14:21:06] 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for arywiki - https://phabricator.wikimedia.org/T257725 (10Andrew) [14:21:09] 10DBA, 10Data-Services, 10User-Ladsgroup, 10cloud-services-team (Kanban): Prepare and check storage layer for shnwiktionary - https://phabricator.wikimedia.org/T256010 (10Andrew) [19:11:42] 10DBA, 10Patch-For-Review: transferpy 1.0 release - https://phabricator.wikimedia.org/T257601 (10jcrespo) ` 16:28:13 Warning, treated as error: 16:28:13 /src/transferpy/transfer.py:docstring of transferpy.transfer.to_bool:5:Field list ends without a blank line; unexpected unindent. 16:28:13 ERROR: InvocationEr...