[01:11:14] (03update) 10siebrand: Draft: docs: specs for translatewiki.net gerrit onboarding (RFC) [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/738 [01:12:54] (03update) 10siebrand: Draft: docs: specs for translatewiki.net gerrit onboarding (RFC) [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/738 [01:16:47] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for SnowFire - https://phabricator.wikimedia.org/T442255#12475689 (10SnowFire) 05Invalid→03Open @Aklapper : Unfortunately, for whatever reason, my Gitlab account still appears to be blocked. Yes, I did a full log... [02:13:15] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell - https://phabricator.wikimedia.org/T442482 (10Andrew) 03NEW [02:13:29] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12475799 (10Andrew) [02:22:21] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12475803 (10Andrew) Here is the raw output from the cumin run of the audit script on all VMs. Interpretation to follow... ` ===== NODE GROUP ====... [02:23:52] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12475804 (10Andrew) [02:25:37] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12475805 (10Andrew) [02:26:31] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12475806 (10Andrew) [05:09:59] (03approved) 10volker-e: CI: Add a test suite [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/8 (owner: 10jforrester) [05:10:23] (03merge) 10volker-e: CI: Add a test suite [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/8 (owner: 10jforrester) [05:10:26] (03update) 10volker-e: tyops.js: Do not skip lines after a line with a typo [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/9 (owner: 10jforrester) [06:24:59] (03PS1) 10Hashar: Disable banner about emergency maintenance [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1359144 [06:28:09] (03CR) 10Hashar: [C:03+2] Disable banner about emergency maintenance [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1359144 (owner: 10Hashar) [06:28:38] (03Merged) 10jenkins-bot: Disable banner about emergency maintenance [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1359144 (owner: 10Hashar) [06:52:15] 10Phabricator: Intermittent database connection error on Phabricator - https://phabricator.wikimedia.org/T442499 (10Mr.Ibrahem) 03NEW [07:12:47] (03CR) 10Hashar: [C:03+1] castor: skip the load and save when CASTOR_DISABLED=1 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1357927 (https://phabricator.wikimedia.org/T439116) (owner: 10Mhurd) [07:13:16] 10Phabricator: Intermittent database connection error on Phabricator - https://phabricator.wikimedia.org/T442499#12476179 (10SD_hehua) Yes, from yesterday. [07:25:56] 10Phabricator, 06Collaboration-Services: Intermittent database connection error on Phabricator - https://phabricator.wikimedia.org/T442499#12476193 (10Marostegui) [07:27:15] 10Phabricator, 06Collaboration-Services: Intermittent database connection error on Phabricator - https://phabricator.wikimedia.org/T442499#12476198 (10Marostegui) I have been having glitches for a couple of hours now and I just got a 500 when running a cookbook that updates phab: ` Exception raised while execu... [07:30:50] 10Phabricator: Phabricator mysql connection unstable - https://phabricator.wikimedia.org/T442356#12476204 (10ArthurTaylor) [07:30:51] 10Phabricator, 06Collaboration-Services: Intermittent database connection error on Phabricator - https://phabricator.wikimedia.org/T442499#12476207 (10ArthurTaylor) →14Duplicate dup:03T442356 [07:31:14] 10Phabricator, 06Collaboration-Services: Intermittent database connection error on Phabricator - https://phabricator.wikimedia.org/T442499#12476194 (10Marostegui) p:05Triage→03High [07:32:46] 10Phabricator, 06Collaboration-Services: Intermittent database connection error on Phabricator - https://phabricator.wikimedia.org/T442499#12476210 (10ArthurTaylor) 05Duplicate→03Open Sorry - merged this in the wrong direction. I meant to close the other ticket [07:33:30] 10Phabricator, 06Collaboration-Services: Intermittent database connection error on Phabricator - https://phabricator.wikimedia.org/T442499#12476213 (10ArthurTaylor) [07:33:33] 10Phabricator: Phabricator mysql connection unstable - https://phabricator.wikimedia.org/T442356#12476216 (10ArthurTaylor) →14Duplicate dup:03T442499 [07:50:13] (03update) 10addshore: dev: bump default MariaDB image to 11.4 LTS [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/737 (owner: 10siebrand) [07:53:09] 10Phabricator, 06Collaboration-Services: Intermittent database connection error on Phabricator - https://phabricator.wikimedia.org/T442499#12476259 (10Jelto) a:03Jelto [08:04:23] 10Phabricator, 06Collaboration-Services: Intermittent database connection error on Phabricator - https://phabricator.wikimedia.org/T442499#12476295 (10hashar) I have noticed similar //MySQL server has gone away// issues yesterday (Oct 26) around the same hours (UTC morning). I am not sure whether it has been... [08:17:39] 10Phabricator, 06Collaboration-Services: Intermittent database connection error on Phabricator - https://phabricator.wikimedia.org/T442499#12476361 (10Jelto) thank you for reporting this task. It's a traffic related issue. I updated some rate-limiting (internal links https://requestctl.wikimedia.org/commit/0e7... [08:25:12] 10Phabricator, 06Collaboration-Services: Intermittent database connection error on Phabricator - https://phabricator.wikimedia.org/T442499#12476381 (10Marostegui) Thanks Jelto. For what is worth, the database is also not having major issues, it is the same HW we have in production for wikis and it is doing a l... [08:51:29] (03merge) 10addshore: dev: bump default MariaDB image to 11.4 LTS [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/737 (owner: 10siebrand) [09:02:00] codders: that Cypress cache issue ( https://integration.wikimedia.org/ci/job/generic-node26-browser/12/console ) looks quite similar to https://phabricator.wikimedia.org/T361520 [09:02:51] ah yeah - it does. Is there any reason why that might pop up again after the node upgrade? [09:02:55] 09:25:16 npm warn Unknown env config "cypress-cache-folder". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. [09:02:55] 09:25:16 npm warn Unknown project config "CYPRESS_CACHE_FOLDER". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. [09:02:57] I'll comment in that ticket and link there [09:03:01] noticed that in the console output [09:03:33] for the context CI exposes XDG_CACHE_CONFIG=/cache [09:03:42] and whatever is written there get restored for next builds [09:04:05] Cypress mentions: We expected the binary to be installed here: /src/cypress/.cache/13.17.0/Cypress/Cypress [09:04:37] 06Release-Engineering-Team, 10Scap, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12476513 (10Sfaci) [09:04:48] does it matter that `rsync: [sender] change_dir "/wikibase-new-lexeme-special-page/main/generic-node26-browser" (in caches) failed: No such file or directory (2) ` ? [09:05:19] or is that just a function of there never having been a successful build with the new image? [09:06:10] hmm [09:06:51] there is some cache stuff from Sept 2nd ~ 7:46 UTC [09:07:04] ssh integration-castor06.integration.eqiad1.wikimedia.cloud ls -la /srv/castor/wikibase-new-lexeme-special-page/main/generic-node24-browser/Cypress [09:07:05] drwxrwxr-x 3 jenkins-deploy wikidev 4096 Sep 2 07:46 13.15.0 [09:07:05] drwxrwxr-x 3 jenkins-deploy wikidev 4096 Sep 2 07:48 13.17.0 [09:07:21] but node24 [09:07:25] this is node26 [09:07:29] hahahah [09:07:46] so yeah hmm indeed there is no cache [09:08:03] I don't know how that ever got populated [09:08:15] the change_dir error is then because there is no directory [09:08:20] seems a bit chicken and egg - there's no `cypress install` in the npm task and there never was [09:08:35] it is certainly mentioned as an absolute path as / because it iruns in a chroot on the rsync side of thing [09:08:38] (there's just the npm install, the installs cypress) [09:09:31] then I guess Cypress somehow had a post install hook which led it install itself and store itself under /cache? [09:10:14] `09:25:05 npm warn install-scripts cypress@13.17.0 (postinstall: node index.js --exec install)` [09:10:18] I am making assumption, I don't know how Cypress installs / cache stuff itself [09:10:26] is allowScripts now being enforced where it wasn't before? [09:11:44] https://gerrit.wikimedia.org/r/c/wikibase/new-lexeme-special-page/+/1333231 was merged on Sept 2nd at 7:48. It updated Cypress from 13.15.0 to 13.17.0 which reflects in the cache I pasted above. The old node24 job definitely managed to install and store a new version of Cypress [09:12:14] meanwhile we upgraded NodeJS from 24 to 26. No idea whether npm got upgraded as well [09:12:43] I can try adding cypress to the allowScripts and see if that makes a difference. I don't really know anything about the feature [09:12:59] neither do I :/ [09:14:01] yeah - seems to be new in node26, or at least fresh-node24 doesn't know about it whereas fresh-node26 does [09:14:59] and the node version would affect npm? [09:15:16] seems so [09:15:23] if you can +2 https://gerrit.wikimedia.org/r/c/wikibase/new-lexeme-special-page/+/1358166, we can see if it helped [09:16:09] hmm. nope [09:16:10] https://integration.wikimedia.org/ci/job/generic-node26-browser/13/console [09:16:13] I checked the images, we also upgraded npm from 11.19. to 11.20. [09:16:15] I checked the images, we also upgraded npm from 11.19. to 11.20.0 [09:16:28] - image: docker-registry.wikimedia.org/releng/node24-test-browser:24.22.1 [09:16:29] + image: docker-registry.wikimedia.org/releng/node26-test-browser:26.12.1 [09:17:04] adding the allow-script cleared the warning, but doesn't seem to have made a difference to the outcome [09:17:14] 10Gerrit: Improve the Gerrit login page (legacy LDAP layout, styling and rotating Commons background) - https://phabricator.wikimedia.org/T442505 (101Veertje) 03NEW [09:18:54] We expected the binary to be installed here: /cache/Cypress/13.17.0/Cypress/Cypress [09:18:59] (03PS8) 101Veertje: Fix login form: responsive labels and mobile-friendly layout [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 (https://phabricator.wikimedia.org/T442505) [09:19:00] which it looks like where it should be expected [09:19:25] but apparently Cypress does not install / saves data there :\ [09:19:33] it is probably reproducible [09:20:33] (03PS9) 101Veertje: Fix login form: responsive labels and mobile-friendly layout [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 (https://phabricator.wikimedia.org/T442505) [09:22:36] codders: I have sent a dummy change not touching anything https://gerrit.wikimedia.org/r/c/wikibase/new-lexeme-special-page/+/1359206 [09:22:48] but it looks like the issue is Cypress now suddenly refuses to install [09:22:59] or it installs somewhere else [09:23:11] yeah. "something changed" :) [09:24:19] npm 11.20.0 changelog https://github.com/npm/cli/blob/release/v11/CHANGELOG.md#11200-2026-09-21 [09:25:14] the allows script thing is https://github.com/npm/cli/pull/9913 [09:25:21] no clue what it is or whether it is related [09:26:12] weird. In `fresh-node24`, the `allow-scripts` command seems to be completely absent [09:26:16] not sure just that patch would account for it [09:26:21] but fresh-node is also not the releng image [09:26:33] probably not the same versions instead [09:27:10] cause fresh-node has manual pin, I have updated them recenly but the CI images have continued their journey [09:27:33] (hey maybe Fresh should learn about --latest flag to pull/use latest image from the registry) [09:27:48] assuming the CI images are immediately rolled to Jenkins, that would work [09:28:28] 06Release-Engineering-Team (Radar), 07Essential-Work, 10Test Kitchen (Experiment Platform Sprint 31): Determine if we can expand TK experiment deployment windows - https://phabricator.wikimedia.org/T440612#12476597 (10MLechvien-WMF) From Serviceops perspective, we'd request no deploys during the [[ https://w... [09:28:47] codders: I think we can reuse https://phabricator.wikimedia.org/T361520 [09:28:51] that looks like the same issue [09:29:07] and maybe I will role back that repo back to node24 to get you unblocked [09:29:19] I am going to grab a coffee first [09:32:32] 10Phabricator, 06Collaboration-Services, 06Data-Engineering (Q2 FS26/27 October 1st - December 31th)), 13Patch-For-Review: data lake: phabricator events - https://phabricator.wikimedia.org/T434047#12476600 (10Antoine_Quhen) Update: Reviews are done. We’re making minor tweaks before deployment. [09:35:55] back [09:38:40] fun locally with fresh-node26 / npm 11.19.1 I get the same issue [09:43:21] (I also had to pop out to the doctor) [09:43:29] I'll be back online after 11h [09:50:41] :) [09:52:17] (03PS1) 10Samtar: Review access change [labs/xtools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1359215 [09:52:56] (03PS2) 10Samtar: Review access change [labs/xtools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1359215 (https://phabricator.wikimedia.org/T441761) [09:54:10] TheresNoTime: you can further tweak that commit message to replace "Review access change" by something such as "Allow l10n-bot-watcher" :] [09:54:15] 10Gerrit, 06Gerrit-Privilege-Requests, 06Moderator-Tools-Team, 10XTools, 13Patch-For-Review: Give access to l10n-bot to labs/xtools - https://phabricator.wikimedia.org/T441761#12476663 (10TheresNoTime) a:03TheresNoTime [09:54:17] or whatever makes sense :) [09:54:52] will do :) I think I should have just clicked "save" tbh :D [09:55:04] hehe [09:55:15] eventually it will be enforced to have a change created for any repo config change [09:55:16] (03PS3) 10Samtar: Add l10n-bot-watcher to labs/xtools [labs/xtools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1359215 (https://phabricator.wikimedia.org/T441761) [09:55:20] at somepoint in the future [09:55:24] want me to merge that one [09:55:25] ? [09:58:03] hashar: oh if you don't mind, yes please. Otherwise I think someone from my team will do it later [09:58:17] (03CR) 10Hashar: [V:03+2 C:03+2] Add l10n-bot-watcher to labs/xtools [labs/xtools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1359215 (https://phabricator.wikimedia.org/T441761) (owner: 10Samtar) [09:58:24] TheresNoTime: submitted! :) [09:59:02] Thanks! [10:01:22] 10Gerrit, 06Gerrit-Privilege-Requests, 06Moderator-Tools-Team, 10XTools, 13Patch-For-Review: Give access to l10n-bot to labs/xtools - https://phabricator.wikimedia.org/T441761#12476705 (10TheresNoTime) 05Open→03Resolved All done @Nikerabbit - let me know if there's any issues :) [10:36:18] codders: I am giving up. It seems to be the node 24 > 26 upgrade which breaks Cypress somehow :/ [10:38:38] hmpf. that is frustrating. I'll also try and do a deeper dive, but if you're stumped I don't fancy my chances [10:39:27] I guess some dependency is broken with Node 266 [10:39:34] which causes Cypress to silently fail to install [10:39:50] ex https://github.com/cypress-io/cypress/issues/33891 [10:40:16] though it applies to Cypress 15.x and node 24.15.0 [10:40:18] anyway [10:40:45] you can try with the latest Cypress and see how it goes [10:40:49] I am off for lunch prep etc [10:41:33] worse case scenario, we roll back that repo to Node 24 until that is figured out [11:07:23] codders: Might be quickest to just port the Cypress tests back to Selenium so they use a supported technology? I can do that pretty swiftly, I think. [11:08:16] James_F: it isn't just the one repo using cypress, and it's our team's technology of choice. I'd rather spend some time investigating what the actual issue is [11:08:45] codders: Sure. But the “technology of choice” comment is odd. It’s not up to teams to make that call, they’re made Wikimedia-wide. [11:08:47] (I understood that cypress was also a supported, or at least tolerated, technology for tests in our repos) [11:09:48] It was pitched as an experiment to fix the flakiness of Selenium. That test failed; Cypress has been much, much worse than Selenium, and has much less expertise. But let’s focus on getting you fixed! [11:10:35] https://phabricator.wikimedia.org/T374000#10339961 [11:10:58] seems like there's a few repos that didn't get that memo, if it was a memo [11:11:18] The managers responsible for this decision at WMF keep changing, sadly. [11:12:23] codders: Can you point to a gerrit request that exhibits the issue? [11:12:33] https://gerrit.wikimedia.org/r/c/wikibase/new-lexeme-special-page/+/1358166 [11:12:41] Thanks. [11:20:48] also re the decision - we did have a public ADR before adopting cypress: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/WikibaseLexeme/+/refs/heads/master/docs/adr/0004-use-cypress-for-browser-testing.md [11:21:14] Who from Release Engineering, Quality Services, and the other teams that need to support such choices signed that off? [11:21:59] OK, I think I’ve found the issue. [11:22:27] The Cypress code is falling back to an npm implementation of unzip that is broken for large files. [11:22:47] We can easily bodge this by just putting an OS-level unzip into the images, I think. [11:23:12] (This won’t fix `fresh` until we do a release of it with the new image, but we can bump CI immediately and see if that fixes it.) [11:23:41] We’ll also need to purge the castor caches as these partial Cypress unzip downloads will have been saved there, meh. [11:24:07] I can purge those for the new-lexeme-special-page jobs [11:24:17] and thanks for the quick fix! [11:24:30] Thank me if it actually works. :-) This is just speculation right now. [11:30:37] (03PS1) 10Jforrester: Docker: [node26] Add zip/unzip to maybe help Cypress [integration/config] - 10https://gerrit.wikimedia.org/r/1359248 (https://phabricator.wikimedia.org/T361520) [11:30:49] (03CR) 10Jforrester: [C:03+2] Docker: [node26] Add zip/unzip to maybe help Cypress [integration/config] - 10https://gerrit.wikimedia.org/r/1359248 (https://phabricator.wikimedia.org/T361520) (owner: 10Jforrester) [11:32:00] (03Merged) 10jenkins-bot: Docker: [node26] Add zip/unzip to maybe help Cypress [integration/config] - 10https://gerrit.wikimedia.org/r/1359248 (https://phabricator.wikimedia.org/T361520) (owner: 10Jforrester) [11:32:26] !log Docker: [node26] Add zip/unzip to maybe help Cypress, for T361520 [11:32:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:32:28] T361520: "The cypress npm package is installed, but the Cypress binary is missing" error prevents merging changes - https://phabricator.wikimedia.org/T361520 [11:34:13] I don't think the cache needs clearing - there's nothing there for generic-node24-browser, since it's never succeeded [11:34:21] sorry - for generic-node26-browser [11:34:25] Aha, good. [11:42:38] (03update) 10jforrester: tyops.js: Do not skip lines after a line with a typo [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/9 [11:50:27] I always forget how long the docker publish step takes. :-( [11:58:56] (03CR) 10Jforrester: [C:03+2] "Successfully published image docker-registry.discovery.wmnet/releng/ajv:0.4.1-s7" [integration/config] - 10https://gerrit.wikimedia.org/r/1359248 (https://phabricator.wikimedia.org/T361520) (owner: 10Jforrester) [11:59:12] (03PS1) 10Jforrester: jjb: Switch node jobs to images with zip/unzip to maybe help Cypress [integration/config] - 10https://gerrit.wikimedia.org/r/1359253 (https://phabricator.wikimedia.org/T361520) [11:59:46] codders: Tah-dah: https://integration.wikimedia.org/ci/job/generic-node26-browser/15/console [12:00:04] I’ll roll the fix out to all the other CI jobs, but for you this is fixed.\ [12:00:05] amazing! [12:00:12] I'll tidy up that patch [12:00:15] thanks again! [12:00:36] Also I suppose I should write an upstream bug report for extract-zip. [12:04:11] (03CR) 10Jforrester: [C:03+2] jjb: Switch node jobs to images with zip/unzip to maybe help Cypress [integration/config] - 10https://gerrit.wikimedia.org/r/1359253 (https://phabricator.wikimedia.org/T361520) (owner: 10Jforrester) [12:05:55] (03Merged) 10jenkins-bot: jjb: Switch node jobs to images with zip/unzip to maybe help Cypress [integration/config] - 10https://gerrit.wikimedia.org/r/1359253 (https://phabricator.wikimedia.org/T361520) (owner: 10Jforrester) [13:15:11] 10Phabricator: Request to unlink MediaWiki/Wikimedia account from old/broken Phabricator account - https://phabricator.wikimedia.org/T442421#12477693 (10janusthedev) This is primarily an issue for Toolforge, and not so much for Phabricator. [13:16:37] 10Phabricator: Request to unlink MediaWiki/Wikimedia account from old/broken Phabricator account - https://phabricator.wikimedia.org/T442421#12477698 (10janusthedev) {F111009189} [13:23:21] 10Continuous-Integration-Config, 10Cite, 10EntitySchema, 06Growth-Team, and 5 others: Come up with a way for Cypress to work in CI without CYPRESS_CACHE_FOLDER in .npmrc as that's deprecated in npm 11 and will be dropped in npm 13+ - https://phabricator.wikimedia.org/T442537#12477721 (10Jdforrester-WMF) [13:24:23] codders: Heads-up that T442537 isn’t urgent, as the breaking change was landed but then reverted for npm 12 (https://github.com/npm/cli/pull/9276 and then https://github.com/npm/cli/pull/9729). [13:24:24] T442537: Come up with a way for Cypress to work in CI without CYPRESS_CACHE_FOLDER in .npmrc as that's deprecated in npm 11 and will be dropped in npm 13+ - https://phabricator.wikimedia.org/T442537 [13:26:16] thanks for the heads-up! [13:35:36] James_F: codders: i am quite happy to see the issue was somewhere in Cypress and installing zip/unzip fixed it [13:36:00] hashar: TBF it’s not really Cypress’s fault. [13:36:09] of course not [13:36:17] IT IS ALL THE FAULT OF JAVASCRIT ECOSYSTEM [13:36:20] we should RiiR [13:36:22] Indeed. [13:36:30] Rust? Ruby? R? :-) [13:36:33] * hashar waits for LLM access to port everything to rust [13:36:44] hashar: I’ve been doing that for my services! [13:36:53] or maybe x86 assembly... after all the programing language is really just an implementation details [13:37:06] so might as well go straight to CPU/GPU instructions [13:37:18] (03merge) 10addshore: gateway auth for tunnels only, and don't always re tunnel [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/728 [13:37:30] pR0v3 m3 vvR0nG ¿ [13:38:27] on the serious side of thing, if we have some updates, I am more than happy to cut a new release of fresh [13:40:02] though I have a few patches pending, notably one to stop copy pasting between the fresh-node?? scripts: https://gerrit.wikimedia.org/r/c/fresh/+/1347594 :b [13:40:21] hashar: Yeah, who does CR on fresh now? I’ve lost track. It’s now official a RelEng thing? [13:41:00] was Timo idea and thus was de factor placed under the Performance team umbrella [13:41:13] the team got disbanded and every single or their items got transfers to different teams [13:41:21] Fresh got thus official transfered to releng indeed :] [13:42:21] and thus I guess I can bubble that up to my team [13:42:54] bubble bubble [13:43:07] quibble quibble [14:02:37] thcipriani: around at all? :) [14:30:04] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12478167 (10Volans) > I don't know what "php=yes-unknown" means but hopefully apt upgrade will resolve them as well. That's where the php version... [14:31:28] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12478186 (10Volans) [14:33:22] (03approved) 10volker-e: tyops.js: Do not skip lines after a line with a typo [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/9 (owner: 10jforrester) [14:33:24] (03merge) 10volker-e: tyops.js: Do not skip lines after a line with a typo [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/9 (owner: 10jforrester) [14:33:31] (03update) 10volker-e: tyops.js: Report line numbers that start at 1 [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/10 (owner: 10jforrester) [14:50:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:50:41] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown - https://phabricator.wikimedia.org/T442559 (10phaultfinder) 03NEW [14:52:23] addshore: physically, briefly, what's up? [14:55:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [14:59:33] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12478478 (10Zabe) [15:01:41] (03PS1) 10Urbanecm: Zuul: Add Sanskardubey to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1359312 [15:04:36] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12478534 (10thcipriani) p:05Triage→03High [15:05:31] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12478541 (10thcipriani) Thanks @Volans/@Andrew ! Broke down to specific boxes/tasks in the description: lots of folks should be able to help [15:09:10] thcipriani: looking for thoughts (even if initial) on https://phabricator.wikimedia.org/T384781#12445162 [15:09:31] as i'm about to make another release and wonder if its worth noting anything in my changelog noting about the future [15:10:16] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12478584 (10thcipriani) I think https://gitlab.wikimedia.org/cloudvps-repos/deployment-prep/tofu-provisioning should only affect puppet roles/hiera... [15:17:02] addshore: we are a bit slammed at the moment, but i'd be fine just giving it its own top-level group [15:19:07] (pick your name in /repos/) [15:21:06] i'll note that on the task. [15:23:49] 10GitLab, 06Release-Engineering-Team (Radar), 10a Wikimedia CLI: Possibly rename mwcli and mwcli gitlab repo - https://phabricator.wikimedia.org/T384781#12478733 (10brennen) I guess it seems fine to me for this project to have its own container group in `/repos`. That would give it access to the runners, any... [15:31:02] brennen thanks! yeah just needed a vauge node in either the no or yes direction, we can think about actually doing stuff in a week, or a month, or next year :P [15:31:04] ty [15:32:32] 10GitLab, 06Release-Engineering-Team (Doing 😎), 10a Wikimedia CLI: Possibly rename mwcli and mwcli gitlab repo - https://phabricator.wikimedia.org/T384781#12478809 (10brennen) [15:33:15] addshore: sure thing, holler on the task when you land on a good name, i'll create the group. [15:40:24] ty :) [15:40:37] addshore: If you want naming ideas, shout. :-) [15:45:00] James_F: I wouldnt mind [15:45:42] so far I half moved toward "a-wikimeida-cli` xD. the idea being I'l going to continue adding things I would consider good wikimedia services, with easy cli access, but also i cant call it "the" wikimedia cli [15:45:45] WiCLI? [15:45:56] mwrust! [15:45:57] a for addshore too obviosuly [15:46:20] for the record, mwcli is straightforward, easy to remember and has a nice brand! [15:46:26] And wrong. [15:46:34] lets not fall in the pitfall of (formally mwcli) :b [15:47:58] hashar: there are 2 things currently called mwcli [15:48:00] :D [15:48:08] thats one of the drivers for the name change [15:48:15] AHHHHHHHhhhhh [15:48:17] and the other is that it actually is probably only 10% "mediawiki" [15:48:45] so i was basically going to change the binary name from "mw" to something else (right now likely now just "wm") but then needa longer name to go with that [15:49:24] yeah yeah sorry I merely did a drive-by comment [15:49:29] cause I like the name "mwcli" [15:49:46] :D [15:51:06] stuff like this is coming in the next release, and will be documented on mw.org https://usercontent.irccloud-cdn.com/file/rttvtf6p/kubernetes-pod-helloworld.mp4 [15:51:57] more stuff noone asked for https://usercontent.irccloud-cdn.com/file/FGkJaZvC/pageviews.mp4 [15:52:30] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12478904 (10Southparkfan) Thanks Andrew! Let's get the process started. >>! In T442482#12478584, @thcipriani wrote: > I think https://gitlab.wik... [15:58:37] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown (gerrit1003) - https://phabricator.wikimedia.org/T442559#12478944 (10Jelto) [16:03:30] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown (gerrit1003) - https://phabricator.wikimedia.org/T442559#12478958 (10Jelto) [16:03:37] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown (gerrit1003) - https://phabricator.wikimedia.org/T442559#12478965 (10Jelto) [16:03:52] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown (gerrit1003) - https://phabricator.wikimedia.org/T442559#12478972 (10Jelto) [16:04:23] 06Release-Engineering-Team, 06Collaboration-Services: ProbeDown (gerrit1003) - https://phabricator.wikimedia.org/T442559#12478976 (10Jelto) 05Open→03Resolved a:03Jelto Happened because of maintenance for package udpate [16:11:11] James_F: thoughts on `a-wikimedia-cli` then? [16:11:29] or other better ideas, if the binary name on disk was "wm" [16:14:48] 06Release-Engineering-Team, 10Scap, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12479053 (10JVanderhoop-WMF) [16:15:50] 06Release-Engineering-Team (Radar), 07Essential-Work, 10Test Kitchen (Experiment Platform Sprint 31): Determine if we can expand TK experiment deployment windows - https://phabricator.wikimedia.org/T440612#12479072 (10DLynch) Would it make more sense to do your experiment-status changes *immediately-after* r... [16:16:52] addshore: In descending preference from me “WiCLI”,“Addshore’s Wikimedia Advanced Shell Helper / AWASH”, “wmcli” [16:21:15] (03open) 10jforrester: Re-build bookworm and trixie images for libbrotli1 security update [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/132 (https://phabricator.wikimedia.org/T442495) [16:21:21] (03update) 10jforrester: Re-build bookworm and trixie images for libbrotli1 security update [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/132 (https://phabricator.wikimedia.org/T442495) [16:22:10] (03open) 10jforrester: fundraising: Re-build for libbrotli1, and install Node.js 22 from nodejs.org [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/133 (https://phabricator.wikimedia.org/T442495) [16:22:18] (03update) 10jforrester: fundraising: Re-build for libbrotli1, and install Node.js 22 from nodejs.org [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/133 (https://phabricator.wikimedia.org/T442495) [16:23:01] (03update) 10jforrester: Re-build bookworm and trixie images for libbrotli1 security update [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/132 (https://phabricator.wikimedia.org/T442495) [16:23:12] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for SnowFire - https://phabricator.wikimedia.org/T442255#12479169 (10brennen) 05Open→03Resolved a:03brennen > Is this downstream of some Toolforge bot / script still being down from PHP woes? Plausibly. I'v... [16:23:30] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for SnowFire - https://phabricator.wikimedia.org/T442255#12479174 (10brennen) [16:24:17] (03approved) 10brennen: Re-build bookworm and trixie images for libbrotli1 security update [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/132 (https://phabricator.wikimedia.org/T442495) (owner: 10jforrester) [16:26:22] 10Scap (SpiderPig 🕸️): Allow deployment of single releases with Spiderpig deploy-service - https://phabricator.wikimedia.org/T442586 (10Jelto) 03NEW [16:29:46] James_F: shall i publish those? [16:29:52] brennen: Please do. [16:30:02] one moment. [16:30:02] (Can I?) [16:30:24] i've an open terminal sitting right there. :) [16:30:32] Thanks! [16:31:01] (03merge) 10brennen: Re-build bookworm and trixie images for libbrotli1 security update [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/132 (https://phabricator.wikimedia.org/T442495) (owner: 10jforrester) [16:31:46] when there is some room to breathe we'll get this happening via gitlab [16:32:00] Oh, yeah, I’ve seen that as an option and never investigated. [16:32:04] !log Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/132 (T442495) [16:32:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:32:10] T442495: Rebuild dev images for libbrotli1 (CVE-2026-111497) - https://phabricator.wikimedia.org/T442495 [16:33:27] j.eena and i messed with it a while back and it was... well, not intractable, but more tangled than seemed worth the lift at the time. but a lot of things have gotten better since then and dan's done a bunch of work lately that i think will probably help. [16:33:42] * James_F nods. [16:34:07] “Build and deploy this” as a post-merge pipeline step (or button or whatever) would be very helpful. [16:35:03] quite. [16:35:31] Also, some CI to assert that patches actually build correctly would be nice, too. :-D [16:35:53] docker-pkg even has testing features baked in, we just... don't really use them. at least in this repo. [16:36:00] but yeah, agreed. [16:36:08] test.sh in the image root, right? [16:36:14] yeah, something like that [16:36:16] I don’t think I’ve ever seen an image use that. [16:36:26] But each time I build, docker-pkg shames me for not using it. [16:36:29] ^ [16:36:55] there's so much shame to go around that it barely registers, but i do notice it every time and think hmm, probably we should... [16:38:47] (03open) 10jforrester: releases: Bump vue and @vue/compiler-sfc to 3.5.43 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/153 [16:39:29] (03update) 10jforrester: releases: Bump vue and @vue/compiler-sfc to 3.5.43 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/153 [16:39:48] (03merge) 10jforrester: releases: Bump vue and @vue/compiler-sfc to 3.5.43 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/153 [16:53:10] 10Gerrit, 06Collaboration-Services, 06SRE Observability, 13Patch-For-Review: Prometheus scrapping denied by Gerrit - https://phabricator.wikimedia.org/T441498#12479636 (10hnowlan) [17:04:04] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12479834 (10thcipriani) [17:11:27] 10GitLab, 06Release-Engineering-Team (Doing 😎), 10a Wikimedia CLI: Possibly rename mwcli and mwcli gitlab repo - https://phabricator.wikimedia.org/T384781#12479957 (10thcipriani) 👋 Sorry for late reply. In general, moving out of `repos/releng/` makes sense. We'll have to figure out where to move it to retain... [17:12:57] ^ addshore late reply (that says pretty much what scrollback says here): moving it would reflect reality, just gotta make a namespace. [17:13:27] <3 [17:16:47] !log Rearming keyholder on deployment-deploy06.deployment-prep [17:16:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:21:38] (03CR) 10Ahmon Dancy: [C:03+1] castor: skip the load and save when CASTOR_DISABLED=1 [integration/config] - 10https://gerrit.wikimedia.org/r/1357927 (https://phabricator.wikimedia.org/T439116) (owner: 10Mhurd) [17:23:19] hey releng folks, now that things are not quite as off the hook as last week, could I ask what the recommendations are for gpg key algorithm/length for keys for releases.mw.o (i.e. https://www.mediawiki.org/keys/keys.html )? [17:39:06] FIRING: WidespreadInstanceDown: Widespread instances down in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadInstanceDown [17:59:14] (03update) 10jforrester: tyops.js: Report line numbers that start at 1 [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/10 [18:00:55] (03update) 10addshore: Draft: Add interactive documentation review and selective publishing [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/724 [18:01:08] hehe, then i can stop causing all this noise in here :P [18:17:17] (03PS1) 10RLazarus: helm-linter: Update envoy-future to v1.40.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1359350 (https://phabricator.wikimedia.org/T436338) [18:17:18] (03PS1) 10RLazarus: jjb: [helm-lint] Update to helm-linter:0.8.5 to pick up envoy-future 1.40.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1359351 (https://phabricator.wikimedia.org/T436338) [18:18:09] ^ hi releng! would appreciate a prompt look at those two if you have a moment, we'd like to move efficiently with this envoy update [18:19:03] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12480537 (10thcipriani) [18:28:03] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12480582 (10thcipriani) [18:31:27] rzl: Taking a look [18:32:43] (03CR) 10Ahmon Dancy: [C:03+2] helm-linter: Update envoy-future to v1.40.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1359350 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [18:33:51] (03Merged) 10jenkins-bot: helm-linter: Update envoy-future to v1.40.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1359350 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [18:34:03] dancy: much thanks! [18:34:18] rzl: I'll build the new image now [18:38:09] apergos: We haven't done the third-party release in a bit, so haven't had to think about it :D I'd say: check in with the person who did the last one. The last time we generated keys rsa 4096 was recommended. But I think the defaults are now ECC 25519 (which seems like the right choice). [18:39:43] the person who did the last one didn't know either and just used the defaults [18:39:57] which is how I wound up in here asking the question :-) [18:40:15] apergos: the last time we as releng did stuff with gpg keys, we used https://blog.liw.fi/posts/2017/05/29/using_a_yubikey_4_for_ensafening_one_s_encryption/ which still seems relevant modulo the ECC key part [18:40:56] ok, I'll read, process, and then write something on the relevant task. thanks! [18:41:12] this reminds me that i think i have an expired key situation to deal with [18:41:18] which i am... not going to think about right now [18:41:23] !log journalctl --vacuum-size 256M on deployment-mediawiki15.deployment-prep.eqiad1.wikimedia.cloud to free up space [18:41:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:41:44] !log journalctl --vacuum-size 256M on deployment-jobrunner06.deployment-prep.eqiad1.wikimedia.cloud to free up space [18:41:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:41:54] I'd guess any lurking debian devs probably have good opinions in the gpg key space [18:42:02] if you can corner one [18:44:05] oh I forgot to ask about expiry too, woops. asking now in case you have any thoughts [18:46:35] expiry of 1y is recommended. you'll gave to edit your key to update expiry if need. it's (as you can tell from brennen 's comment) an easy thing to forget. To update (gpg --edit-key then type "expire" at the > and re-export) [18:46:48] we were keeping keys in https://gerrit.wikimedia.org/r/plugins/gitiles/pgp-public-keys/ [18:47:06] since all the public key exchanges went caput from bogus signatures [18:47:16] (kaput?) [18:47:21] (went away) [18:49:30] i think i may still have a pending patch there to update my key from the _last_ time i thought about expiry [18:49:43] which is probably a good indicator of the shape this process is in [18:49:47] testing the new defaults in generation: ECC and 25519 are the defaults for the main key + signing key, so that seems good. [18:51:38] (03open) 10dancy: scap pull: Do not print a stack trace when rsync fails [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1384 [18:51:55] (03update) 10dancy: scap pull: Do not print a stack trace when rsync fails [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1384 [18:52:49] this is all great info, plenty for me to work with. [18:54:37] (03merge) 10dancy: scap pull: Do not print a stack trace when rsync fails [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1384 [19:01:21] I kinda think that folks are moving away from gpg and more like to short-lived certs (e.g. sigstore used by a buncha stuff now modulo linux distros) [19:02:11] er s/and more/to more/ [19:03:08] er...trying to say: people use sigstore for software releases and that's good. [19:03:27] rzl: Having issues with image building. Still debugging [19:03:56] I'm not a fan of the whole cert infra [19:04:55] dancy: appreciate it. note we moved the package from bullseye-wikimedia up to bookworm-wikimedia, I don't think that will affect your image build but mentioning it in case I missed something [19:05:42] dancy: ah man it does, sorry, I missed that in dockerfiles/helm-linter/envoy-future.list [19:05:49] sending a followup patch [19:06:42] do you know if that needs a new changelog entry also? [19:06:55] apergos: same, currently seems too centralized, at the same time: gpg keys have always been tricky to use and once the public registries stopped accepting signatures, less useful. [19:07:18] I *think* not if you haven't already built 0.8.5, but I can just as easily make it 0.8.5-1 anyway [19:07:28] or 0.8.6 I mean [19:07:36] uh huh the web of trust is one problem, and the gpg 'stuff the registries' issue is another [19:07:44] ^ [19:09:17] rzl: I haven't been able to build anything yet [19:09:53] (so, no changelog update needed, I think) [19:10:16] ack [19:11:21] (03PS1) 10RLazarus: helm-linter: Switch envoy-future to bookworm-wikimedia [integration/config] - 10https://gerrit.wikimedia.org/r/1359360 (https://phabricator.wikimedia.org/T436338) [19:11:46] dancy: ^ see if that fixes it. sorry for the wasted time [19:12:14] (03CR) 10Ahmon Dancy: [C:03+2] helm-linter: Switch envoy-future to bookworm-wikimedia [integration/config] - 10https://gerrit.wikimedia.org/r/1359360 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [19:13:04] (03PS1) 10Ahmon Dancy: fab: Remove docker pull docker-registry.wikimedia.org/buster [integration/config] - 10https://gerrit.wikimedia.org/r/1359361 [19:13:35] (03Merged) 10jenkins-bot: helm-linter: Switch envoy-future to bookworm-wikimedia [integration/config] - 10https://gerrit.wikimedia.org/r/1359360 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [19:14:12] (03PS2) 10Ahmon Dancy: fab: Remove docker pull docker-registry.wikimedia.org/buster [integration/config] - 10https://gerrit.wikimedia.org/r/1359361 [19:26:42] (03CR) 10Ahmon Dancy: [C:03+2] fab: Remove docker pull docker-registry.wikimedia.org/buster [integration/config] - 10https://gerrit.wikimedia.org/r/1359361 (owner: 10Ahmon Dancy) [19:28:06] (03Merged) 10jenkins-bot: fab: Remove docker pull docker-registry.wikimedia.org/buster [integration/config] - 10https://gerrit.wikimedia.org/r/1359361 (owner: 10Ahmon Dancy) [19:29:07] Ha, good spot. [19:29:51] dancy: It’d be lovely to kill ci-buster and all its children soon. [19:30:00] Agreed! [19:48:41] James_F: it is possible the docker pulls could be removed from `./fab` [19:49:03] it worked around docker-pkg not refreshing the base images, but I think it does now [19:49:08] but I haven't checked [19:49:16] hashar: I think that fix hasn’t been released yet. [19:49:21] hashar: Or maybe it has? [19:49:40] * hashar gently pushes James to yet another rabbit hole [19:49:56] I’m in about two dozen already. :-) [19:50:33] !gerrit I5844d86054da9f64d523e54e2fb001288fddbf1e [19:50:33] https://gerrit.wikimedia.org/r/#/q/I5844d86054da9f64d523e54e2fb001288fddbf1e [19:50:35] T200720 [19:50:36] T200720: docker-pkg should attempt to pull dependent images from the registry - https://phabricator.wikimedia.org/T200720 [19:50:37] maybe [19:51:02] Hmmmmmm. Maybe? [19:51:04] that is from november 2018? 🤷 [19:51:07] ❓ [19:51:09] ! [19:51:09] So new? [19:51:14] yeah quite recent [19:51:42] I don't know whether that one actually fixes it [19:52:17] (03approved) 10volker-e: tyops.js: Report line numbers that start at 1 [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/10 (owner: 10jforrester) [19:52:19] cause the base images ( eg `FROM docker-registry.wikimedia.org/bookworm:latest`) are not defined by docker-pkg [19:52:25] and I have no clue what is its behavior [19:52:36] Magic. [19:52:57] I classify Magic. along Hope. and Beliefs [19:53:17] which is on the other shelf than the one Engineering is placed at :b [19:54:05] anyway that change was probably to pull/refresh from registry the images managed by docker-pkg [19:54:10] but maybe not the other images [19:54:33] anyway [19:54:37] meanwhile ./fab pulls :b [19:55:38] I currently have 328 open tabs (up from my normal ~50). I think I’ll spend the rest of today closing things out. [19:56:02] sounds good! [20:04:53] hashar: thank you for sorting the deploy this morning [20:05:37] RhinosF1: you are welcome! I'll watch the windows more carefully tomorrow and on thursday :) [20:07:37] hashar: there's a window now without a deployer yet if you're not already long past your day unfortunately [20:07:56] yeah we had a complicated week last week :b [20:08:20] Complicated is one way to describe last week [20:08:43] Thank you to all of you who've been dealing with the fallout [20:08:51] I don't want to repeat last week anytime soon [20:08:58] +1 [20:12:54] brennen: looking at performance/WikimediaDebug , it looks like you have pushed a signed tag for 3.4.0 :) [20:14:08] it might have broken the extension on Firefox, the button is always gray for me [20:17:15] hashar: the last tag i pushed for wikimediadebug wasn't signed, because that's when i learned my key was expired. it's on my list to fix that. [20:17:37] great! :-] [20:17:55] did you test the new 3.4.0 on Firefox? [20:17:56] i don't think it's actually broken on firefox, and it was definitely tested on both browsers, but, uh, i'll check again. [20:18:03] great! [20:18:10] so it is probably my setup being broken in some magic way [20:18:45] i do remember thinking something was a bit different visually. i'll have a look when i am back at my desktop. [20:18:48] current status: tacos. [20:18:58] enjoy the meal! [20:19:04] and good luck with refreshing the gpg key! [20:20:04] Hmm, trusted-runner pushing to docker registry seems broken? [20:21:34] Ah, I see the thread. [20:32:12] (03PS1) 10Ahmon Dancy: helm-linter: Use bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1359390 [20:32:52] think i ran into that earlier. [20:35:32] (03CR) 10Ahmon Dancy: [C:03+2] helm-linter: Use bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1359390 (owner: 10Ahmon Dancy) [20:36:54] (03Merged) 10jenkins-bot: helm-linter: Use bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1359390 (owner: 10Ahmon Dancy) [20:36:55] (03merge) 10jforrester: tyops.js: Report line numbers that start at 1 [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/10 [20:36:59] (03update) 10jforrester: tyops.js: Keep the case of a case-insensitive typo in the fix [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/11 [20:37:04] (03update) 10jforrester: tyops.js: Keep the case of a case-insensitive typo in the fix [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/11 [20:37:10] (03update) 10jforrester: tyops.js: Keep the case of a case-insensitive typo in the fix [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/11 [20:56:56] (03CR) 10Ahmon Dancy: [C:03+2] "Successfully published image docker-registry.discovery.wmnet/releng/helm-linter:0.8.5" [integration/config] - 10https://gerrit.wikimedia.org/r/1359390 (owner: 10Ahmon Dancy) [20:57:22] !log Updated docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1359390 and friends [20:57:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:57:48] (03CR) 10Ahmon Dancy: [C:03+2] jjb: [helm-lint] Update to helm-linter:0.8.5 to pick up envoy-future 1.40.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1359351 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [20:59:16] (03Merged) 10jenkins-bot: jjb: [helm-lint] Update to helm-linter:0.8.5 to pick up envoy-future 1.40.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1359351 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [21:00:15] (03CR) 10Ahmon Dancy: [C:03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1359351 (https://phabricator.wikimedia.org/T436338) (owner: 10RLazarus) [21:07:24] 10Continuous-Integration-Config, 10a Wikimedia CLI, 10dev-images, 05MW-1.43-notes, and 4 others: Rebuild releng/* dev/* images for PHP upgrade - https://phabricator.wikimedia.org/T441915#12481619 (10hashar) [21:08:53] 10Continuous-Integration-Config, 10a Wikimedia CLI, 10dev-images, 05MW-1.43-notes, and 4 others: Rebuild releng/* dev/* images for PHP upgrade - https://phabricator.wikimedia.org/T441915#12481620 (10hashar) https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1357693 (for `fundraising/REL1_43`) was the la... [21:10:00] (03approved) 10volker-e: tyops.js: Keep the case of a case-insensitive typo in the fix [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/11 (owner: 10jforrester) [21:10:28] (03merge) 10volker-e: tyops.js: Keep the case of a case-insensitive typo in the fix [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/11 (owner: 10jforrester) [21:27:15] (03update) 10addshore: Draft: Add interactive documentation review and selective publishing [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/724 [21:28:02] (03update) 10addshore: Draft: Add interactive documentation review and selective publishing [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/724 [21:28:08] (03update) 10addshore: Add interactive documentation review and selective publishing [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/724 [21:29:01] helm-lint in CI for operations/deployment-charts seems to be broken. It fails every time with `Errno::ENOENT: No such file or directory - create-chart` [21:32:55] dancy: We might need to revert the helm-linter CI change (or fix). [21:34:22] post-merge CI in gerrit seems to not be working, too; stuck on "waiting for jobs" https://gerrit.wikimedia.org/r/c/mediawiki/services/mobileapps/+/1359392?tab=checks [21:40:48] (03merge) 10addshore: Add interactive documentation review and selective publishing [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/724 [21:41:17] dbrant: it is waiting in the queue https://integration.wikimedia.org/zuul/#q=1359392 [21:41:33] postmerge has a lower priority compared to the rest [21:42:10] fun [21:42:15] the job has just started! [21:43:16] ahh ty [21:43:42] you are welcome! :) [21:43:47] thank you for having reported it! [21:44:13] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12481792 (10Southparkfan) [21:47:37] 10Continuous-Integration-Config, 10a Wikimedia CLI, 10dev-images, 05MW-1.43-notes, and 4 others: Rebuild releng/* dev/* images for PHP upgrade - https://phabricator.wikimedia.org/T441915#12481846 (10Jdforrester-WMF) See also the follow-on work on {T442495} for libbrotli1 (rather than PHP versions). [21:55:41] !log Beta Cluster: remove PHP 8.3 config remnants on mediawiki and jobrunner hosts; packages were in dpkg 'rc' state (i.e. not installed, but config files are present) - T442482 [21:55:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:55:46] T442482: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482 [21:57:48] 10GitLab (Infrastructure), 06Release-Engineering-Team (Radar), 06Collaboration-Services: Trusted runners failing to connect to docker registry - https://phabricator.wikimedia.org/T439046#12481885 (10thcipriani) 05Resolved→03Open OK, we had a recurrence of this, fixed with some help, but I'm pretty sure I... [21:58:13] 10GitLab (Infrastructure), 06Release-Engineering-Team (Radar), 06Collaboration-Services: Trusted runners failing to connect to docker registry - https://phabricator.wikimedia.org/T439046#12481889 (10thcipriani) p:05High→03Medium (resetting priority < high since its working now) [21:59:25] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12481891 (10Southparkfan) >>! In T442482#12475803, @Andrew wrote: > Here is the raw output from the cumin run of the audit script on all VMs. Inter... [22:02:49] (03open) 10jforrester: Move the typo check out of the grunt task into lib/tyops.js, and Add a command-line tool, bin/tyops.js [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/12 [22:02:49] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12481913 (10Southparkfan) [22:02:51] (03update) 10jforrester: Move the typo check out of the grunt task into lib/tyops.js, and add a command-line tool, bin/tyops.js [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/12 [22:02:56] (03merge) 10jforrester: Move the typo check out of the grunt task into lib/tyops.js, and add a command-line tool, bin/tyops.js [repos/ci-tools/grunt-tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/grunt-tyops/-/merge_requests/12 [22:03:58] thanks a lot for the quick turnaround dancy <3 sorry that turned out to be more work than expected [22:10:42] !log Beta Cluster: install mediawiki-libs-shellbox "2026-10-22-141212-video85" tag - T442482 [22:10:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:10:48] T442482: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482 [22:20:25] 06Release-Engineering-Team, 10Catalyst, 10dev-images, 10MediaWiki-Docker: dev-images: Handle common PHP ini settings in shared file(s) - https://phabricator.wikimedia.org/T442649 (10brennen) 03NEW [22:20:45] (03open) 10brennen: Set session.upload_progress = Off in php-fpm images [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/134 (https://phabricator.wikimedia.org/T441915 https://phabricator.wikimedia.org/T442649) [22:21:41] (03update) 10jforrester: Add an agent skill, skills/tyops/SKILL.md [repos/ci-tools/tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/tyops/-/merge_requests/13 [22:21:42] (03update) 10jforrester: Rename the package from grunt-tyops to tyops [repos/ci-tools/tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/tyops/-/merge_requests/14 [22:21:43] (03update) 10jforrester: Release v0.2.0 [repos/ci-tools/tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/tyops/-/merge_requests/15 [22:21:47] (03open) 10jforrester: Add an agent skill, skills/tyops/SKILL.md [repos/ci-tools/tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/tyops/-/merge_requests/13 [22:21:48] (03open) 10jforrester: Rename the package from grunt-tyops to tyops [repos/ci-tools/tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/tyops/-/merge_requests/14 [22:21:51] (03open) 10jforrester: Release v0.2.0 [repos/ci-tools/tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/tyops/-/merge_requests/15 [22:23:06] (03update) 10brennen: Set session.upload_progress = Off in php-fpm images [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/134 (https://phabricator.wikimedia.org/T441915 https://phabricator.wikimedia.org/T442649) [22:27:11] (03update) 10jforrester: Release v0.2.0 [repos/ci-tools/tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/tyops/-/merge_requests/15 [22:27:13] (03update) 10jforrester: Release v0.2.0 [repos/ci-tools/tyops] - 10https://gitlab.wikimedia.org/repos/ci-tools/tyops/-/merge_requests/15 [22:31:23] (03approved) 10jforrester: Set session.upload_progress = Off in php-fpm images [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/134 (https://phabricator.wikimedia.org/T441915 https://phabricator.wikimedia.org/T442649) (owner: 10brennen) [22:31:58] ~ docker run --quiet --pull always --rm -it --entrypoint /usr/bin/envoy docker-registry.wikimedia.org/releng/helm-linter --version [22:31:58] /usr/bin/envoy version: 86af01fdec40c31d025c495a4aa303c4126b2fd0/1.40.1/Clean/RELEASE/BoringSSL [22:32:01] perfect 👍 [22:32:14] rzl: Not perfect. :-( [22:32:29] uh oh [22:32:30] rzl: The helm-lint job for deployment-charts seems broken, sadly. [22:32:46] oh, looking [22:32:56] E.g. https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1359402 -> https://integration.wikimedia.org/ci/job/helm-lint/37008/console [22:33:04] 17:46:38 Errno::ENOENT: No such file or directory - create-chart [22:34:03] hmm, yeah. let me see [22:34:38] Also if you have a spare 5 mins a merge on https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/1359399 would be great (and a build even better). [22:35:33] ack, let me see if I can un-break charts CI first but I'll look at that before I go offline for the day [22:35:40] <3 [22:35:48] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12481997 (10Southparkfan) [22:36:03] I can repro "No such file or directory - create-chart" with `rake run_locally` on an empty change, so that's good at least [22:36:26] create-chart is a sextant subcommand so I think we're failing to install sextant in the image correctly, still digging [22:39:04] yeah definitely that, `sextant -h` works in 0.8.4 but not 0.8.5 [22:39:45] there's a line in the Rakefile that's supposed to print a nice error if sextant is missing, so I guess that's broken too, but that can go on the pile for later [22:40:13] rzl: Ping me if needed. I'm available for another hour [22:40:49] dancy: thanks! I see in https://gerrit.wikimedia.org/r/1359390 that you switched to pipx for sextant, but it looks like it didn't have the desired effect, I'm still working on why [22:41:42] it's a symlink to /root/.local/pipx/venvs/sextant/bin/sextant [22:41:42] rzl: Ah, permissions issue. It's installed in a root-only directory [22:41:55] I'll experiment [22:41:59] ahh that'll do it [22:45:25] dancy: is that just pipx install --global? [22:45:53] I didn't notice that option! I'll give it a shot [22:46:04] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team, 13Patch-For-Review: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12482048 (10thcipriani) [22:46:29] Oh, yeah, the perils of Python in Debian. [22:51:39] or s/ in Debian// for that matter [22:52:01] look, it's perils all the way down [22:52:13] (i know this one is specific, but there are so many perils) [22:52:27] +1 [22:54:55] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team, 13Patch-For-Review: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12482102 (10thcipriani) [22:56:51] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team, 13Patch-For-Review: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12482104 (10Southparkfan) [22:58:07] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team, 13Patch-For-Review: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12482107 (10Southparkfan) [22:59:09] aside: weirdly if I make any local noop change to the Rakefile, I get "Please install sextant: pip3 install sextant" as expected. if I run on a completely clean commit I get "Errno::ENOENT: No such file or directory - create-chart" [23:00:19] so I conjecture that `which("sextant")` must be returning nil in the first case as expected, but "" or something in the second case. but it'll be annoying to debug [23:00:41] (03PS1) 10Ahmon Dancy: helm-linter: Install sextant in /pipx [integration/config] - 10https://gerrit.wikimedia.org/r/1359407 [23:01:10] (03PS2) 10Ahmon Dancy: helm-linter: Install sextant in /pipx [integration/config] - 10https://gerrit.wikimedia.org/r/1359407 [23:02:29] (03CR) 10Ahmon Dancy: [C:03+2] helm-linter: Install sextant in /pipx [integration/config] - 10https://gerrit.wikimedia.org/r/1359407 (owner: 10Ahmon Dancy) [23:03:36] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team, 13Patch-For-Review: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12482120 (10Southparkfan) [23:03:41] (03Merged) 10jenkins-bot: helm-linter: Install sextant in /pipx [integration/config] - 10https://gerrit.wikimedia.org/r/1359407 (owner: 10Ahmon Dancy) [23:04:22] !log Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1359407 [23:04:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:08:00] (03CR) 10Ahmon Dancy: [C:03+2] "docker-registry.discovery.wmnet/releng/helm-linter:0.8.5-s1" [integration/config] - 10https://gerrit.wikimedia.org/r/1359407 (owner: 10Ahmon Dancy) [23:09:20] (03PS1) 10Ahmon Dancy: jjb: [helm-lint] Update to helm-linter:0.8.5-s1 [integration/config] - 10https://gerrit.wikimedia.org/r/1359408 [23:09:42] (03CR) 10Ahmon Dancy: [C:03+2] jjb: [helm-lint] Update to helm-linter:0.8.5-s1 [integration/config] - 10https://gerrit.wikimedia.org/r/1359408 (owner: 10Ahmon Dancy) [23:10:03] dancy: nice, `rake run_locally` looks good now! [23:10:18] yay! [23:11:12] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team, 13Patch-For-Review: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12482187 (10Southparkfan) [23:11:15] (03Merged) 10jenkins-bot: jjb: [helm-lint] Update to helm-linter:0.8.5-s1 [integration/config] - 10https://gerrit.wikimedia.org/r/1359408 (owner: 10Ahmon Dancy) [23:11:47] rzl: The --global option wasn't available in that old version of pipx from debian. [23:11:59] ohh interesting [23:12:55] James_F: helm-linter Repaired [23:13:17] pipx 1.1.0 wow, yeah [23:14:49] Excellent. [23:16:16] dancy: whew, now I think we're actually all good :) thank you again! [23:22:37] 06Release-Engineering-Team, 10Catalyst, 10dev-images, 10MediaWiki-Docker, 13Patch-For-Review: dev-images: Handle common PHP ini settings in shared file(s) - https://phabricator.wikimedia.org/T442649#12482227 (10brennen) The dev-images MR above is a good example of something we should handle in fewer places. [23:22:45] (03merge) 10brennen: Set session.upload_progress = Off in php-fpm images [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/134 (https://phabricator.wikimedia.org/T441915 https://phabricator.wikimedia.org/T442649) [23:27:49] (03PS1) 10Jforrester: Zuul: [labs/tools/wikiinfo] Add mediawiki-i18n-check template [integration/config] - 10https://gerrit.wikimedia.org/r/1359410 (https://phabricator.wikimedia.org/T441762) [23:28:14] (03CR) 10Jforrester: [C:03+2] Zuul: [labs/tools/wikiinfo] Add mediawiki-i18n-check template [integration/config] - 10https://gerrit.wikimedia.org/r/1359410 (https://phabricator.wikimedia.org/T441762) (owner: 10Jforrester) [23:29:39] (03Merged) 10jenkins-bot: Zuul: [labs/tools/wikiinfo] Add mediawiki-i18n-check template [integration/config] - 10https://gerrit.wikimedia.org/r/1359410 (https://phabricator.wikimedia.org/T441762) (owner: 10Jforrester) [23:30:00] !log Zuul: [labs/tools/wikiinfo] Add mediawiki-i18n-check template, for T441762 [23:30:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:30:05] T441762: Please add mediawiki-i18n-check-docker to repository labs/tools/wikiinfo - https://phabricator.wikimedia.org/T441762 [23:30:25] 10Continuous-Integration-Config, 10WikiInfo, 13Patch-For-Review: Please add mediawiki-i18n-check-docker to repository labs/tools/wikiinfo - https://phabricator.wikimedia.org/T441762#12482245 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF Done. [23:33:40] 10Beta-Cluster-Infrastructure, 06tools-infrastructure-team: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12482268 (10Southparkfan) [23:33:46] !log Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/134 (T441915, T442649) [23:33:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:33:53] T441915: Rebuild releng/* dev/* images for PHP upgrade - https://phabricator.wikimedia.org/T441915 [23:33:54] T442649: dev-images: Handle common PHP ini settings in shared file(s) - https://phabricator.wikimedia.org/T442649 [23:41:59] rzl: Thanks!