[01:34:30] 10Continuous-Integration-Config, 10CommonPasswords, 10phan, 07ci-test-error, 07Performance Issue: Tests on mediawiki/libs/CommonPasswords failing with timeouts - https://phabricator.wikimedia.org/T394821#10858042 (10Daimona) >>! In T394821#10845912, @Reedy wrote: > There's probably scope for an #upst... [07:01:38] Any idea why Izno has changed our board type to a group: https://phabricator.wikimedia.org/project/manage/7280/? Is that even correct? [07:11:57] Nikerabbit: maybe you can ask Izno or revert? :) The same was done some minutes later on another project: [07:11:57] "set the icon for Ladies-That-FOSS-Wikidata to Group" [07:14:19] there is no chat on Phab :/ [07:17:08] Nikerabbit: then I don't know how we here can find what they might have thought about it :b [07:17:11] you can always try https://www.mediawiki.org/wiki/Special:EmailUser/Izno ? [07:17:23] (reachable by following their Phabricator profile) [07:17:43] yeah I'll do that [07:25:14] (03CR) 10Hashar: [C:03+2] "That is a nice one, thanks 😊" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1149824 (https://phabricator.wikimedia.org/T395190) (owner: 10Krinkle) [07:25:36] (03Merged) 10jenkins-bot: popup: Fix forgetful "backend" dropdown [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1149824 (https://phabricator.wikimedia.org/T395190) (owner: 10Krinkle) [07:25:56] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for Zuul upgrade project - https://phabricator.wikimedia.org/T393873#10858158 (10MoritzMuehlenhoff) 05Resolved→03Ope... [07:29:09] (03CR) 10Hashar: [C:03+2] Add debug instrumentation [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1149825 (https://phabricator.wikimedia.org/T395190) (owner: 10Krinkle) [07:29:28] (03Merged) 10jenkins-bot: Add debug instrumentation [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1149825 (https://phabricator.wikimedia.org/T395190) (owner: 10Krinkle) [08:13:11] (03PS4) 10Hashar: build: Fix ESLint warnings [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1149826 (owner: 10Krinkle) [08:13:11] (03PS1) 10Hashar: build: explicitly set npm package name [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1151119 [08:13:48] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10858282 (10MatthewVernon) >>! In T378922#10848007, @Jelto wrote: >>>! In T378922#10847339, @MatthewVerno... [08:15:38] (03CR) 10Hashar: [C:03+1] "Ideally I'd configure ESLint to turn warnings to errors to prevent regression but I could not find a quick way to do it. But another chang" [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1149826 (owner: 10Krinkle) [08:17:59] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10858295 (10Jelto) Thanks a lot @MatthewVernon, I can confirm the buckets are gone. I'll re-create the bu... [08:18:36] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10858296 (10Jelto) [08:22:30] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10858303 (10MatthewVernon) Yeah, the apus cluster isn't ideal for buckets with a very large number of obj... [08:34:16] 10GitLab (Infrastructure), 10Ceph, 06collaboration-services, 10Data-Persistence-Backup, and 3 others: Migrate gitlab storage to apus (also: backups from S3?) - https://phabricator.wikimedia.org/T378922#10858327 (10Jelto) [09:34:50] (03PS1) 10Hslater: Skip REL1_44 jobs in bluespice template [integration/config] - 10https://gerrit.wikimedia.org/r/1151145 [09:46:31] (03PS1) 10Phuedx: Zuul: [mediawiki/extensions/WikimediaEvents] Add MetricsPlatform dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1151150 (https://phabricator.wikimedia.org/T393918) [10:58:56] 10GitLab (Infrastructure), 06collaboration-services: Check GitLab artifact retention time - https://phabricator.wikimedia.org/T395014#10858878 (10Jelto) p:05Triage→03High a:03Jelto [12:20:07] 10GitLab (Infrastructure), 06collaboration-services: Check GitLab artifact retention time - https://phabricator.wikimedia.org/T395014#10859138 (10Jelto) I've done a bit more research using [https://docs.gitlab.com/user/storage\_management\_automation/#manage-cicd-pipeline-storage](https://docs.gitlab.com/user/... [12:26:22] (03CR) 10Hashar: [C:04-1] "I do not see why you would remove CI entirely for `REL1_44`. If there are tests now failing we should fix them, if BlueSpice does not supp" [integration/config] - 10https://gerrit.wikimedia.org/r/1151145 (owner: 10Hslater) [12:27:04] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/WikimediaEvents] Add MetricsPlatform dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1151150 (https://phabricator.wikimedia.org/T393918) (owner: 10Phuedx) [12:28:07] something isn't right with "Depends-On" dependencies when a patch has been cherry-picked to other branches. e.g. https://gerrit.wikimedia.org/r/c/mediawiki/skins/MinervaNeue/+/1151163 says "Starting gate-and-submit-1_44 jobs.", even though it's on master, because the dependency has been cherry-picked to 1.44 (and already merged on master) [12:28:23] i don't think i've seen that before, but maybe it's a known problem [12:29:19] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikimediaEvents] Add MetricsPlatform dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1151150 (https://phabricator.wikimedia.org/T393918) (owner: 10Phuedx) [12:30:40] hashar: hellooo, I'm back on my opensearch in wikibase ci quest! it turns out that the last attempt to finally enable it only failed (T394376) because a necessary wikibase patch hadn't been merged yet. do you think we could give that another try? [12:30:40] T394376: Wikibase CI broken: Failed to parse mapping [_doc]: analyzer [extract_wb_property] has not been configured in mappings - https://phabricator.wikimedia.org/T394376 [12:31:16] hashar: i.e. merge https://gerrit.wikimedia.org/r/c/integration/config/+/1150758 which is essentially resubmitting the patch that you had made before? [12:32:12] (03CR) 10Jforrester: Skip REL1_44 jobs in bluespice template (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1151145 (owner: 10Hslater) [12:32:26] (03PS1) 10WMDE-leszek: Zuul: [mediawiki/extensions/Wikibase] Enable Open Search for apitests jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1150758 (https://phabricator.wikimedia.org/T386691) [12:52:47] jakob_WMDE: welcome back! [12:53:08] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/Wikibase] Enable Open Search for apitests jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1150758 (https://phabricator.wikimedia.org/T386691) (owner: 10WMDE-leszek) [12:53:27] jakob_WMDE: it is great to see you have found the issue! [12:54:54] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Wikibase] Enable Open Search for apitests jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1150758 (https://phabricator.wikimedia.org/T386691) (owner: 10WMDE-leszek) [12:57:16] hashar: hehe yes! hopefully no issues this time, but it was all working on the missing patch when you rechecked it https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1140156/8#message-351cbb5730b52b3f8aa09f6e0a34a58a4c1d3e10 [13:02:36] jakob_WMDE: I have deployed the change [13:02:45] wohoo thanks!! [13:02:57] (03CR) 10Hashar: [C:03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1150758 (https://phabricator.wikimedia.org/T386691) (owner: 10WMDE-leszek) [13:05:29] 10Gerrit: Gerrit trips on complicated string for PHP syntax highlighting - https://phabricator.wikimedia.org/T395324 (10ihurbain) 03NEW [13:16:11] 10Gerrit: Gerrit trips on complicated string for PHP syntax highlighting - https://phabricator.wikimedia.org/T395324#10859342 (10hashar) Gerrit uses https://highlightjs.org/ and I once managed to fix a bug in it (T372404). Thus it is certainly fixable :) [13:24:07] hashar: yayyy, it's working now. thanks for all the help! :D [13:30:58] 10Gerrit: Gerrit trips on complicated string for PHP syntax highlighting - https://phabricator.wikimedia.org/T395324#10859379 (10hashar) [13:33:04] 10Gerrit: Gerrit trips on complicated string for PHP syntax highlighting - https://phabricator.wikimedia.org/T395324#10859384 (10hashar) At a quick glance, it does not like the semicolon in the string inside an anonymous function. That makes the bug very similar to T372404 and I bet the fix would be similar to h... [13:34:44] jakob_WMDE: awesome. Can you possibly announce it on wikitech-l? My guess is dcausse/ebernhardson would also be interested in it for CirrusSearch and maybe Elastica [13:38:48] hashar: sure, can do! [14:44:39] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: Recent incidents of buildkitd's storage volume filling up - https://phabricator.wikimedia.org/T395097#10859698 (10dancy) I think this is related to the `publish-build` job of https://gitlab.wikimedia.org/repos/sre/wikitech-static-docker/-/blob/main/.gitl... [15:04:11] (03merge) 10dancy: Allow deployments to specify custom k8s clusters [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/795 (https://phabricator.wikimedia.org/T388761) (owner: 10swfrench) [15:04:55] (03open) 10dancy: Release 4.170.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/820 [15:07:17] (03merge) 10dancy: Release 4.170.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/820 [15:12:25] happy workiversary, dduvall [15:13:03] greg-g: thanks! :D [15:48:43] 🎉 [15:48:55] 🍩 [15:51:08] ooh that donut looks good [15:59:33] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for Zuul upgrade project - https://phabricator.wikimedia.org/T393873#10860046 (10Dzahn) uhm, yea, that is a mistake. no... [16:00:23] there is no emoji for :foie_gras: [16:00:24] :) [16:00:41] haha [16:01:08] 🪿 [16:01:30] :-] [16:02:58] hashar: i will never forget that Annecy foie gras [16:10:54] and suddenly, but weirdly enough surprising nobody, foie gras on -releng [16:17:12] my friend keeps raving about "Happy Foie" developed by this German chef. it's supposed to be just like the original but without having to force-feed the geese https://happyfoie.com/ [16:17:27] then they combined it with .. marzipan [16:18:29] 'instead of stuffing animals, he refined the livers of free-range organic animals with high-quality fats." [16:27:43] color me intrigued. [16:28:59] 10Phabricator, 06Security-Team, 07Security: Audit members of acl*security for more than 12 months of no activity (May 2025) - https://phabricator.wikimedia.org/T368224#10860274 (10sbassett) [16:48:46] 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review: Refactor `build-images.py` to use a common code image and `docker buildx` - https://phabricator.wikimedia.org/T392526#10860403 (10Scott_French) After merging https://gerrit.wikimedia.org/r/1146091, we were able to update thirdparty/docker for... [16:51:47] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for Zuul upgrade project - https://phabricator.wikimedia.org/T393873#10860424 (10ops-monitoring-bot) Cookbook cookbooks... [17:07:30] 06Project-Admins, 06Release-Engineering-Team, 10MediaWiki-extensions-General, 10MediaWiki-General, and 2 others: Create a security pre-release Phabricator policy manageable by the Security Team - https://phabricator.wikimedia.org/T393403#10860524 (10Mstyles) 05Open→03Resolved @Aklapper using the #a... [17:29:03] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for Zuul upgrade project - https://phabricator.wikimedia.org/T393873#10860686 (10ops-monitoring-bot) Cookbook cookbooks... [17:48:04] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 3 others: eqiad/codfw: 6 VM request for Zuul upgrade project - https://phabricator.wikimedia.org/T393873#10860807 (10Dzahn) zuul2001 has been reimaged with... [18:02:29] (03CR) 10Hashar: [C:04-1] Skip REL1_44 jobs in bluespice template (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1151145 (owner: 10Hslater) [18:02:50] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 3 others: eqiad/codfw: 6 VM request for Zuul upgrade project - https://phabricator.wikimedia.org/T393873#10860911 (10Dzahn) new roles created and added to... [18:03:12] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 3 others: eqiad/codfw: 6 VM request for Zuul upgrade project - https://phabricator.wikimedia.org/T393873#10860913 (10Dzahn) 05Open→03Resolved [18:05:17] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 06cloud-services-team: Recent incidents of buildkitd's storage volume filling up - https://phabricator.wikimedia.org/T395097#10860917 (10thcipriani) >>! In T395097#10859698, @dancy wrote: > I think this is related to the `publish-b... [18:10:32] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 06cloud-services-team: Recent incidents of buildkitd's storage volume filling up - https://phabricator.wikimedia.org/T395097#10860942 (10dancy) >>! In T395097#10860917, @thcipriani wrote: >>>! In T395097#10859698, @dancy wrote: >>... [18:13:33] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 06cloud-services-team: Recent incidents of buildkitd's storage volume filling up - https://phabricator.wikimedia.org/T395097#10860953 (10dancy) 05Open→03In progress p:05Triage→03Low a:03dancy [18:22:07] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: give contint-roots access to new zuul VMs (was: create new admin group for "zuul devs") - https://phabricator.wikimedia.org/T394819#10860993 (10Dzahn) 05In progress→0... [18:59:52] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Phabricator "Impossible Cluster Write" - https://phabricator.wikimedia.org/T393931#10861127 (10thcipriani) 05Open→03Invalid Haven't run into this since. I think the explanation was immediate overload which we looked into. Nothing else to do. [19:00:00] (03update) 10dancy: spiderpig: Automatically handle db schema changes at startup [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/819 [19:00:14] (03update) 10dancy: spiderpig: Automatically handle db schema changes at startup [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/819 [19:02:28] (03merge) 10dancy: spiderpig: Automatically handle db schema changes at startup [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/819 [19:04:12] (03update) 10dancy: spiderpig: Support MediaWiki train deployments [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/808 (https://phabricator.wikimedia.org/T392610) (owner: 10dduvall) [20:04:52] 10Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic, 13Patch-For-Review: Separate Gerrit https and ssh/git hostnames - https://phabricator.wikimedia.org/T394271#10861275 (10LSobanski) >>! In T394271#10850552, @thcipriani wrote: > Splitting traffic will mean acute, short-term pain (there... [21:14:40] 10Gerrit, 10Release-Engineering-Team (Radar), 06collaboration-services, 06Traffic, 13Patch-For-Review: Separate Gerrit https and ssh/git hostnames - https://phabricator.wikimedia.org/T394271#10861431 (10A_smart_kitten) [21:20:00] Project beta-update-databases-eqiad build #85026: 04FAILURE in 0.48 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/85026/ [22:22:38] Yippee, build fixed! [22:22:38] Project beta-update-databases-eqiad build #85027: 09FIXED in 2 min 38 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/85027/ [22:33:42] (03open) 10thcipriani: backport: git remove extra -c flag [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/821 [22:33:44] (03update) 10thcipriani: backport: git remove extra -c flag [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/821 [22:37:38] (03update) 10thcipriani: backport: git remove extra -c flag [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/821 [22:54:04] 10Phabricator-Bot-Requests, 10Release-Engineering-Team (Doing 😎): Create Phabricator bot account for Cloud VPS monitoring - https://phabricator.wikimedia.org/T395104#10861661 (10Aklapper) a:03Aklapper API Token available in non-public P76541. Please resolve this task once it works as expected. [22:56:52] (03merge) 10dancy: backport: git remove extra -c flag [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/821 (owner: 10thcipriani) [22:57:51] (03open) 10dancy: Release y [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/822 [22:59:08] (03merge) 10dancy: Release y [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/822 [23:00:50] (03open) 10dancy: Revert "Release y" [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/823 [23:00:53] (03update) 10dancy: Revert "Release y" [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/823 [23:02:09] (03merge) 10dancy: Revert "Release y" [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/823 [23:05:59] (03open) 10dancy: Release 4.171.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/824 [23:07:05] (03merge) 10dancy: Release 4.171.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/824