[09:02:27] 10Continuous-Integration-Infrastructure, 10Release Pipeline: Pipeline lib leaves workspace behind on the Jenkins agents - https://phabricator.wikimedia.org/T302477#10737595 (10hashar) [09:02:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Priority Backlog πŸ“₯): Jenkins jobs leaving workdirs, filling disk on integration-agent-docker-1041 - https://phabricator.wikimedia.org/T391060#10737598 (10hashar) β†’14Duplicate dup:03T302477 [09:22:15] 10Continuous-Integration-Config: Several recent slow (>15 minute) helm-lint job runs - https://phabricator.wikimedia.org/T387781#10737662 (10hashar) I thought I had commented on that task, but that might only have happened on IRC. The task is about [[ https://gerrit.wikimedia.org/r/c/operations/deployment-chart... [09:40:08] 10Continuous-Integration-Config, 13Patch-For-Review: Several recent slow (>15 minute) helm-lint job runs - https://phabricator.wikimedia.org/T387781#10737776 (10hashar) a:03hashar [09:59:12] 10Continuous-Integration-Infrastructure, 10Castor, 10Testing Support: Running `cypress` in Wikimedia CI requires unusual env variables - https://phabricator.wikimedia.org/T361624#10737831 (10hashar) β†’14Duplicate dup:03T361520 [10:18:39] (03update) 10cgoubert: mediawiki-cli: Copy and link mwcron scripts [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/161 (https://phabricator.wikimedia.org/T391665) [10:42:14] 10Phabricator, 10Bitu, 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: Phabricator should use IDP for developer account logins - https://phabricator.wikimedia.org/T377061#10738020 (10SLyngshede-WMF) @Aklapper I've merged the Phabricator Dev service configuration for IDP. Can you drop me an... [10:58:42] 10Continuous-Integration-Infrastructure, 06Diffusion-Repository-Administrators, 10Projects-Cleanup, 06translatewiki.net, and 2 others: Archive the DoubleWiki extension - https://phabricator.wikimedia.org/T344544#10738066 (10hashar) [10:59:28] 10Continuous-Integration-Infrastructure, 06Diffusion-Repository-Administrators, 10Projects-Cleanup, 06translatewiki.net, and 2 others: Archive the DoubleWiki extension - https://phabricator.wikimedia.org/T344544#10738078 (10hashar) >>! In T344544#10626700, @Pppery wrote: > Adding #continuous-integration-in... [11:17:48] 10Beta-Cluster-Infrastructure, 06MediaWiki-Platform-Team, 05SUL3: Set up some beta cluster wikis with different registrable domain - https://phabricator.wikimedia.org/T355281#10738093 (10matmarex) (that patch is not related, I tagged the wrong task) [12:10:39] hello! `ssh gerrit.wikimedia.org -p 29418 gerrit show-queue --by-queue --wide` throws me a "Bad server host key: Invalid key length", so I'm guessing I might be missing a permission somewhere? [12:16:13] 10GitLab (Project Migration), 10Wikidata, 10wmde-wikidata-tech, 10[Archived]Wikidata Dev Team (Quality Tools "Sprint"): [QB] Investigate moving Query Builder from Gerrit to GitLab - https://phabricator.wikimedia.org/T350705#10738337 (10karapayneWMDE) 05Openβ†’03Resolved [12:53:41] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-on-K8s, 06serviceops, 13Patch-For-Review: Migrate mwdebug* hosts to PHP8.1 - https://phabricator.wikimedia.org/T391452#10738603 (10jijiki) [13:20:55] 10Phabricator: Please give @milimetric access to bulk edit tasks - https://phabricator.wikimedia.org/T391833 (10Milimetric) 03NEW [13:22:41] 10Phabricator: Please give @milimetric access to bulk edit tasks - https://phabricator.wikimedia.org/T391833#10738788 (10taavi) 05Openβ†’03Resolved a:03taavi Done. [14:02:23] Thanks Taavi! <3 [14:34:22] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Ironblade101] - https://phabricator.wikimedia.org/T391203#10739133 (10Ironblade101) @Aklapper could my GitLab account please be activated? That account is of the same name as this one [15:36:19] 10Phabricator, 13Patch-For-Review, 07PM: Reset status of tickets that languish in "in progress" - https://phabricator.wikimedia.org/T380300#10739592 (10Ottomata) 05Duplicateβ†’03Open Sorry, I closed the wrong task in a wrong tab of my browser. [15:48:38] (03open) 10dancy: build-images.py: Avoid cascading error if docker build fails [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/163 [15:48:40] (03update) 10dancy: build-images.py: Avoid cascading error if docker build fails [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/163 [15:49:28] (03update) 10dancy: build-images.py: Avoid cascading error if docker build fails [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/163 [15:49:32] 10Continuous-Integration-Infrastructure, 06Diffusion-Repository-Administrators, 10Projects-Cleanup, 06translatewiki.net, and 2 others: Archive the DoubleWiki extension - https://phabricator.wikimedia.org/T344544#10739716 (10Pppery) 05Openβ†’03Resolved a:03Pppery I think we're done here then. [16:03:00] (03update) 10dancy: build-images.py: Avoid cascading error if docker build fails [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/163 [16:04:09] 10MediaWiki-Releasing, 10CheckUser, 06Trust and Safety Product Team, 05MW-1.44-release, 13Patch-For-Review: Bundle CheckUser extension with MediaWiki - https://phabricator.wikimedia.org/T317147#10739838 (10MSantos) >>! In T317147#10731706, @Jdforrester-WMF wrote: > I think new additions / removals in the... [16:04:55] 10MediaWiki-Releasing, 10TemplateStyles, 05MW-1.44-release: Bundle Extension:TemplateStyles with MediaWiki core - https://phabricator.wikimedia.org/T327006#10739849 (10MSantos) >>! In T327006#10731862, @Jdforrester-WMF wrote: > I think new additions / removals in the bundle should get sign-off from the MW re... [16:06:17] (03update) 10dancy: ThreadedApp: Use futures instead of plain threads [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/162 [16:07:56] (03merge) 10dancy: ThreadedApp: Use futures instead of plain threads [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/162 [16:32:02] (03update) 10cgoubert: mediawiki-cli: Copy and link mwcron scripts [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/161 (https://phabricator.wikimedia.org/T391665) [16:32:47] (03update) 10cgoubert: mediawiki-cli: Copy and link mwcron scripts [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/161 (https://phabricator.wikimedia.org/T391665) [16:37:17] (03approved) 10dancy: mediawiki-cli: Copy and link mwcron scripts [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/161 (https://phabricator.wikimedia.org/T391665) (owner: 10cgoubert) [16:41:48] (03update) 10cgoubert: mediawiki-cli: Copy and link mwcron scripts [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/161 (https://phabricator.wikimedia.org/T391665) [16:57:25] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Doing 😎): Stand up Zuul 11 experiment environment in zuul3 cloud VPS project - https://phabricator.wikimedia.org/T391374#10740144 (10hashar) [16:57:58] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Doing 😎): Stand up Zuul 11 experiment environment in zuul3 cloud VPS project - https://phabricator.wikimedia.org/T391374#10740151 (10hashar) @dduvall excellent! The event stream permission is probably good enough, it does not... [17:01:38] (03approved) 10dduvall: build-images.py: Avoid cascading error if docker build fails [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/163 (owner: 10dancy) [17:24:43] (03CR) 10Hashar: [C:03+2] opensource: remove OOjs Router [integration/docroot] - 10https://gerrit.wikimedia.org/r/1136006 (https://phabricator.wikimedia.org/T358813) (owner: 10Novem Linguae) [17:25:14] (03Merged) 10jenkins-bot: opensource: remove OOjs Router [integration/docroot] - 10https://gerrit.wikimedia.org/r/1136006 (https://phabricator.wikimedia.org/T358813) (owner: 10Novem Linguae) [17:38:17] (03update) 10dancy: build-images.py: Avoid cascading error if docker build fails [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/163 [17:39:15] (03merge) 10dancy: build-images.py: Avoid cascading error if docker build fails [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/163 [17:41:27] (03update) 10dancy: mediawiki-cli: Copy and link mwcron scripts [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/161 (https://phabricator.wikimedia.org/T391665) (owner: 10cgoubert) [17:57:37] hashar: I'm thinking that to get the call rolling, we can switch the dep recursion more slowly. Two ideas: One, go back to expanding everything and making a big commit that lists everything explicitly and whittle down from there. This has the benefit of removing hidden recursion on day 1, and preventing more in the future, but the downside of being muddy, slow, and low in impact. The other option I'm thinking of is to add a temporary [17:57:37] flag that opts-in or out. So we can flip it for a few repos until we have enough confidence and then roll out more widely. We could start with Math as per the original issue. [17:58:16] The latter might be more practical as it doesn't require up- and down-stream to agree at the same time. Eg Math > Wikibase > other stuff. [17:59:03] Anyway, I'm continuing to make progress on fixing unintended hard dependencies that have slipped in. A few more and Math can be a pilot. [17:59:36] Right now I'm looking at fixing Cite > VisualEditor due to the way it registers QUnit tests [19:44:53] Krinkle: Temporary flag would be great, I think, but more work for you. [19:46:17] Krinkle: the script I wrote can be run as often as needed and it seems the batch of jobs is processed in just 3hours and a half [19:46:46] ack, it won't need to re-run for a while. I've got enough to go on for a while :D [19:46:58] so it should be possible to fix most of them, it is probably not that many failures as a whole [19:50:13] that is how I handled big migrations in the past [19:50:33] run a them all a few times, catch failures and once most of them passed flip [19:50:45] then catch up with the long tail [19:52:08] https://gerrit.wikimedia.org/r/c/integration/config/+/1132644 can probably be enhanced to emit a slightly better output [19:52:25] and we can certainly run it on a daily basis at like 2am UTC (or whenever it is calm) [19:53:38] the ones I have pasted on the task/https://gist.github.com/hashar/c6b06159056cb6d0ea5cc43fafdaae5b are probably gone now [19:53:46] link https://gist.github.com/hashar/c6b06159056cb6d0ea5cc43fafdaae5b [19:57:15] and in your 2nd option, yes you could add a feature flag to disable recursion on a per repo basis, then I imagine it will take a while to proceed :) [20:00:56] there are also some extensions that should probably be archived or are known to be broken [20:01:05] example the PhpTags* ones ( https://www.mediawiki.org/wiki/Extension:PhpTags ) [20:38:01] 10Gerrit, 07Upstream: Gerrit's syntax highlighting for PHP code breaks when encountering an apostrophe in a // comment in a function call - https://phabricator.wikimedia.org/T372404#10741312 (10hashar) I have poked the pull request ( https://github.com/highlightjs/highlight.js/pull/4230#issuecomment-2802927830 ) [20:51:57] 10GitLab (Upstream pit of despair πŸ•³οΈ), 07Upstream: GraphQL error: Field 'troubleshootJobWithAi' doesn't exist on type 'Pipeline' - https://phabricator.wikimedia.org/T388459#10741356 (10bd808) >>! In T388459#10729898, @Reedy wrote: > And again on https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/7... [20:54:48] 10Continuous-Integration-Config: Experiment with running deptrac in CI for detecting undesired dependencies - https://phabricator.wikimedia.org/T383887#10741359 (10hashar) a:05hasharβ†’03None [21:07:19] 10GitLab (Project Migration), 10BDC-Implementation: Move Analytics from Gerrit to Gitlab - https://phabricator.wikimedia.org/T391404#10741444 (10IAckerman-WMF) p:05Triageβ†’03Medium [21:12:31] 10Continuous-Integration-Infrastructure, 10Testing Support, 10ci-test-error (WMF-deployed Build Failure), 10MW-1.44-notes (1.44.0-wmf.23; 2025-04-01), and 2 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10741459 (10Daimona)... [21:13:48] 10GitLab (Misc), 10Release-Engineering-Team (Radar): Investigate opendev.org hosting options and their costs, risks, & benefits - https://phabricator.wikimedia.org/T344372#10741463 (10bd808) 05Openβ†’03Declined [21:26:59] 10GitLab (Misc), 06Release-Engineering-Team, 10wikimedia.biterg.io: How to identify affiliation of indexed GitLab accounts - https://phabricator.wikimedia.org/T306770#10741517 (10bd808) >>! In T306770#7933958, @Aklapper wrote: > Thanks. That means that I could at least run manual checks on https://ldap.toolf... [21:47:23] 10Continuous-Integration-Infrastructure, 10Testing Support, 10ci-test-error (WMF-deployed Build Failure), 10MW-1.44-notes (1.44.0-wmf.23; 2025-04-01), and 2 others: Selenium timeouts can cause the job to remain stuck until the build times out - https://phabricator.wikimedia.org/T389536#10741596 (10bd808) >... [21:54:58] 10GitLab (Misc), 10Release-Engineering-Team (Priority Backlog πŸ“₯): Puppetize Zuul v9 - https://phabricator.wikimedia.org/T350814#10741614 (10thcipriani) 05Openβ†’03Invalid Superceeded by zuul upgrade work happening this quarter #zuul-upgrade [21:55:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 07Zuul: Proof-of-concept: Run MediaWiki tests in latest Zuul - https://phabricator.wikimedia.org/T349214#10741617 (10thcipriani) 05Openβ†’03Declined [21:56:25] 10GitLab (Misc), 10Release-Engineering-Team (Priority Backlog πŸ“₯): Investigate and document "Depends-On" in GitLab - https://phabricator.wikimedia.org/T349872#10741622 (10thcipriani) 05Openβ†’03Declined May investigate this again Somedayβ„’. Declining for now. [23:22:40] (03approved) 10thcipriani: spiderpig: Add means to prepopulate backport change numbers [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/760 (owner: 10dancy) [23:44:17] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Doing 😎): Stand up Zuul 11 experiment environment in zuul3 cloud VPS project - https://phabricator.wikimedia.org/T391374#10741764 (10dduvall) >>! In T391374#10740144, @hashar wrote: > @dduvall excellent! > > The event stream p... [23:52:21] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Doing 😎): Stand up Zuul 11 experiment environment in zuul3 cloud VPS project - https://phabricator.wikimedia.org/T391374#10741795 (10dduvall) The Zuul dashboard is available at https://zuul-dev.wmcloud.org/tenants