[00:38:23] 10Beta-Cluster-Infrastructure, 10GitLab (CI & Job Runners), 06collaboration-services, 10m3api: Unable to connect to Beta Cluster from WMCS GitLab CI runners - https://phabricator.wikimedia.org/T397591#10981715 (10Dzahn) I deployed Bryan's changes that had already been reviewed by Jelto. After merging on p... [00:46:25] 10Beta-Cluster-Infrastructure, 10GitLab (CI & Job Runners), 06collaboration-services, 10m3api: Unable to connect to Beta Cluster from WMCS GitLab CI runners - https://phabricator.wikimedia.org/T397591#10981720 (10Dzahn) To see an actual change in `iptables -L DOCKER-ISOLATION` I had to `systemctl restart f... [00:48:57] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Provision Kubernetes cluster and bastion using OpenTofu and Magnum - https://phabricator.wikimedia.org/T396936#10981723 (10Dzahn) firewall changes deployed -> T397591#10981699 [00:58:16] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Provision Kubernetes cluster and bastion using OpenTofu and Magnum - https://phabricator.wikimedia.org/T396936#10981729 (10Dzahn) ferm restarted on all wmcs runners and verified they have the iptables rule now for `enc-1.cloudinfra.eqi... [00:59:31] 10Beta-Cluster-Infrastructure, 10GitLab (CI & Job Runners), 06collaboration-services, 10m3api: Unable to connect to Beta Cluster from WMCS GitLab CI runners - https://phabricator.wikimedia.org/T397591#10981733 (10Dzahn) This should work now. ferm restarted on all wmcs gitlab-runners. They all have the ipt... [01:26:45] (03update) 10dancy: api.py: Revise spiderpig login session handling [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/880 (https://phabricator.wikimedia.org/T397457) [01:27:43] (03update) 10dancy: api.py: Revise spiderpig login session handling [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/880 (https://phabricator.wikimedia.org/T397457) [06:08:17] 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-Phabricator: Add the 'other assignee' field to the Phabricator test instance - https://phabricator.wikimedia.org/T398732#10981975 (10A_smart_kitten) Thanks @Dzahn! I've done that now - for the record, I've left the new value I've changed tha... [06:59:34] 10Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T392179#10982072 (10Dreamy_Jazz) [07:00:06] 10Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T392179#10982076 (10Dreamy_Jazz) [07:19:40] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upgrade GitLab to major version 18 - https://phabricator.wikimedia.org/T394382#10982093 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.upgrade was started by jelto@cumin1003 on GitLab host gitlab1004.wikimedia.org with rea... [07:30:38] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upgrade GitLab to major version 18 - https://phabricator.wikimedia.org/T394382#10982114 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.upgrade started by jelto@cumin1003 on GitLab host gitlab1004.wikimedia.org with reason:... [07:31:27] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upgrade GitLab to major version 18 - https://phabricator.wikimedia.org/T394382#10982120 (10Jelto) [07:33:52] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services: Upgrade GitLab to major version 18 - https://phabricator.wikimedia.org/T394382#10982133 (10Jelto) Both replicas have been upgraded successfully to version 18. I'll upgrade production tomorrow at 12:00 UTC. The cookbook should be... [07:40:36] (03open) 10jelto: gitlab-runner: bump image version to alpine-v18.0.3 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/480 (https://phabricator.wikimedia.org/T394382) [07:40:59] (03update) 10jelto: gitlab-runner: bump image version to alpine-v18.0.3 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/480 (https://phabricator.wikimedia.org/T394382) [07:41:56] 10GitLab (Infrastructure), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Upgrade GitLab to major version 18 - https://phabricator.wikimedia.org/T394382#10982168 (10Jelto) [08:16:52] 10Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.45.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T392179#10982285 (10Aklapper) [08:28:54] (03PS1) 10Tiziano Fogli: Review access change [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1167145 [08:33:21] 10Gerrit: Users can not delete their sandbox branches - https://phabricator.wikimedia.org/T398921 (10hashar) 03NEW [08:35:00] (03PS1) 10Hashar: Review access change [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1167147 [08:35:15] (03open) 10jelto: set session_expire_delay to 90 days [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/73 (https://phabricator.wikimedia.org/T398791) [08:36:49] (03PS2) 10Hashar: Allow users to clean stall branches in refs/sandbox/${username} [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1167147 (https://phabricator.wikimedia.org/T398921) [08:39:10] (03CR) 10Hashar: [V:03+2 C:03+2] Allow users to clean stall branches in refs/sandbox/${username} [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1167147 (https://phabricator.wikimedia.org/T398921) (owner: 10Hashar) [08:40:01] (03PS1) 10Hashar: Review access change [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1167150 [08:42:20] (03PS2) 10Hashar: Remove specific force push to refs/sandbox/* branches [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1167150 (https://phabricator.wikimedia.org/T398921) [08:43:51] (03Abandoned) 10Tiziano Fogli: Review access change [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1167145 (owner: 10Tiziano Fogli) [08:44:07] (03CR) 10Hashar: [V:03+2 C:03+2] Remove specific force push to refs/sandbox/* branches [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1167150 (https://phabricator.wikimedia.org/T398921) (owner: 10Hashar) [09:27:04] 10Gerrit, 13Patch-For-Review: Users can not delete their sandbox branches - https://phabricator.wikimedia.org/T398921#10982677 (10hashar) When an access list has a reference containing `${username}`, it is special cased by Gerrit to restrict interacts to that specific user. All-Projects granted {nav Create Re... [09:27:13] 10Gerrit, 13Patch-For-Review: Users can not delete their sandbox branches - https://phabricator.wikimedia.org/T398921#10982679 (10hashar) 05Open→03Resolved a:03hashar [11:32:31] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Remove remnants of Phatality ID in Phabricator - https://phabricator.wikimedia.org/T398397#10983223 (10Aklapper) [11:33:17] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Remove remnants of Phatality ID in Phabricator - https://phabricator.wikimedia.org/T398397#10983224 (10Aklapper) [11:38:40] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Remove remnants of Phatality ID in Phabricator - https://phabricator.wikimedia.org/T398397#10983250 (10Aklapper) [12:01:10] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 07Performance Issue: Drop a bunch of Maniphest Custom Fields from fulltext indexing - https://phabricator.wikimedia.org/T398399#10983376 (10Aklapper) [12:22:34] (03PS1) 10Jforrester: Docker: [mediawiki-phan-taint-check-demo] Move to bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1167207 (https://phabricator.wikimedia.org/T383337) [12:23:26] (03CR) 10Jforrester: [C:03+2] Docker: [mediawiki-phan-taint-check-demo] Move to bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1167207 (https://phabricator.wikimedia.org/T383337) (owner: 10Jforrester) [12:25:16] (03Merged) 10jenkins-bot: Docker: [mediawiki-phan-taint-check-demo] Move to bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1167207 (https://phabricator.wikimedia.org/T383337) (owner: 10Jforrester) [12:31:38] !log Docker: [mediawiki-phan-taint-check-demo] Move to bookworm, for T383337 [12:31:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:31:40] T383337: Migrate all CI jobs from Bullseye to Bookworm or later and drop Bullseye testing support - https://phabricator.wikimedia.org/T383337 [12:47:23] (03PS1) 10Jforrester: jjb: [mw-tools-phan-demos-publish] Switch to bookworm-based image [integration/config] - 10https://gerrit.wikimedia.org/r/1167210 (https://phabricator.wikimedia.org/T383337) [12:47:44] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate all CI jobs from Bullseye to Bookworm or later and drop Bullseye testing support - https://phabricator.wikimedia.org/T383337#10983594 (10Jdforrester-WMF) [12:48:26] (03CR) 10Jforrester: [C:03+2] "Deployed and tested; it now works again!" [integration/config] - 10https://gerrit.wikimedia.org/r/1167210 (https://phabricator.wikimedia.org/T383337) (owner: 10Jforrester) [12:49:03] Daimona: ^^ means that https://doc.wikimedia.org/mediawiki-tools-phan-SecurityCheckPlugin/master/demos/ is updated for the first time in a month or so. [12:49:36] (03Merged) 10jenkins-bot: jjb: [mw-tools-phan-demos-publish] Switch to bookworm-based image [integration/config] - 10https://gerrit.wikimedia.org/r/1167210 (https://phabricator.wikimedia.org/T383337) (owner: 10Jforrester) [12:49:39] Oh, was it broken? I think I used it kinda recently [12:50:02] It was broken when we stopped supporting 7.4, as the CI image was buster-based and so used PHP 7.4. [12:55:01] Hmmmm I guess. [12:55:26] If something breaks but nobody notices, is the thing really broken? [12:56:00] I noticed. :-) But yes. [13:06:01] Daimona: I can confirm that if nobody notices: it is ain't broken :] [13:06:19] Indeed ;) [13:06:50] I have looked at MediaWikiFarm AGAIN and this time I got CI to pass https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MediaWikiFarm/+/1163866/ [13:07:19] and I found out you tried to fix the tests 5+ years ago ( https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MediaWikiFarm/+/545263 ) [13:07:33] I'll reach out to them, I am not sure whether that extension is still actively used [13:07:40] hashar: OMG. [13:07:53] * hashar calls ER for James [13:08:02] Nice work. Let's land it now and tell them to consider it after it's too late. [13:08:31] that is one of the last extension still using UnitTestsList hook, which I have an interest in seeing it disappear [13:08:31] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MediaWikiFarm/+/1163866/ [13:08:51] the others are DonationInterface (I haven't looked at it yet) and Wikibase which really is TWO extension in a single repo and that causes troubles [13:09:01] Yeah. [13:09:37] I've C+2'ed both. [13:09:43] I still have nightmares from when I attempted that fix years ago [13:09:56] There's lots more still to fix. [13:09:58] well [13:10:00] But it passes CI. [13:10:12] So… Daimona, if you want to try your wider fixes now… :-D [13:10:24] ;] [13:10:44] Oh noes [13:10:55] a question I have for both of you is which channel can I join to chat about mediawiki / testing etc? [13:11:12] beside here of course :] [13:11:12] This one, more than any, in practice. [13:11:17] yea [13:11:48] I am looking at #mw-testing-framework-reviews [13:11:57] I guess folks interested are already here anyway [13:20:18] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 07Performance Issue: Drop a bunch of Maniphest Custom Fields from fulltext indexing - https://phabricator.wikimedia.org/T398399#10983720 (10Aklapper) [13:25:07] James_F: merci! [13:25:26] Bien sûr. [13:26:56] Yep, that's my understanding too [13:27:50] and I have reached out to seb35 by email [13:28:17] then https://wiki-valley.com/wiki/Sp%C3%A9cial:Version shows that they are using MediaWikiFarm from September 2019 :] [13:28:39] Oh dear. [13:28:45] ?? [13:29:31] it is stable software :-] [13:29:47] No 'b'. Stale software. [13:30:06] join the light james [13:30:08] join the light! [13:30:20] "if it ain't broke, don't fix it!" 🎶 [13:30:34] If it ain't broke, you've clearly not QA'ed it enough. [13:30:45] 🎶 "old software, is good old stable software" 🎸 [13:30:49] I should start a cult [13:31:34] you don't need QA when you don't have bug, and can't have bug if there is nowhere to report them [13:32:00] and I am back to what Daimona was saying: "if nobody notices, it ain't broken"! [13:32:12] anyway joke aside. I am going to look at DonationInterface [13:32:27] hashar: Don't cry. [13:32:39] at least it is not written in Go [13:33:01] Flawless logic [13:33:47] there is no #hashar tag in Phabricator, thus no issues 🤪 [13:45:13] json does not support comments [13:45:20] nor does it support trailing commas [13:45:23] Indeed. [13:45:36] * hashar files a task to switch extension registration to yaml [13:45:46] Or JSONC, just to be unhelpful. [13:45:56] i propose a custom file format [13:46:15] based on the mediawiki template syntax [13:46:17] you know, if I had become the CEO we would have switched to XML [13:46:35] use that as MCR content on the wikis [13:46:50] Surely we can all agreed on OWL+RDFa in a JSON encoding? [13:46:52] and do the rendering via an XSLT to wikitext (I am sure parsoid team would have loved that) [13:47:26] add a grain of SOAP! [13:47:51] * James_F is having flashbacks. [13:47:55] Not good ones. [13:48:05] taavi: I can sense the avalanche of curly braces and the file format unexpectedly being Turing complete [13:49:26] 10Phabricator, 06collaboration-services: Update Phabricator 429 message - https://phabricator.wikimedia.org/T398316#10983974 (10Jelto) We could use a short version of https://phabricator.wikimedia.org/maniphest/task/edit/form/75/?title=Phabricator%20throttling%20issues%20&description=I%27m%20reporting%20issues... [13:54:05] and that is where I am wondering whether an extension can have settings that are only loaded for tests [14:05:21] 06Project-Admins: Create project tag for Reader Experiments - https://phabricator.wikimedia.org/T398973 (10lwatson) 03NEW [14:09:05] 06Project-Admins: Create project tag for Reader Experiments - https://phabricator.wikimedia.org/T398973#10984174 (10Aklapper) Is there a code repository, or group prefix in a code hosting system (or anything like that) to link to? The mentioned teams to link are #reader_growth_team and #reader_experience_team I... [14:09:49] We should be friendlier to new-generation developers. Why don't we rewrite MediaWiki in Minecraft? [14:10:14] I am working on it [14:10:31] :] [14:11:07] Noice [14:12:36] Minecraft is in progress: https://phabricator.wikimedia.org/T388913 [14:13:36] RiiR! [14:16:19] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Remove remnants of Phatality ID in Phabricator ("error.id" custom field) - https://phabricator.wikimedia.org/T398397#10984219 (10Aklapper) [14:16:21] Hah! It's just a matter of scopecreeping that task a lil then [14:22:04] 10Deployments, 10Release-Engineering-Team (Radar), 06serviceops, 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10984246 (10akosiaris) a:03akosiaris No new reports, I 'll resolve, feel free to reopen. [14:22:22] 10Deployments, 10Release-Engineering-Team (Radar), 06serviceops, 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10984248 (10akosiaris) 05Open→03Resolved [14:26:42] damn [14:27:40] 06Project-Admins: Create project tag for Reader Experiments - https://phabricator.wikimedia.org/T398973#10984284 (10lwatson) Thanks, @Aklapper. Yes, those are the teams! The code repository doesn't exist yet. [14:42:30] 10Phabricator (Upstream), 07Upstream: Phabricator does not warn about (apparent) duplicates when creating a new task - https://phabricator.wikimedia.org/T45#10984358 (10Chealer) [14:46:07] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services: Deploy Phabricator/Phorge 2025-07-08 - https://phabricator.wikimedia.org/T398976 (10brennen) 03NEW [14:47:15] (03open) 10brennen: update submodules for 2025-07-08 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/74 (https://phabricator.wikimedia.org/T398976) [14:47:59] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2025-07-08 - https://phabricator.wikimedia.org/T398976#10984389 (10brennen) [14:52:22] 10Phabricator, 06DBA: Switchover m3 (phabricator) master db1213 -> db1250 - https://phabricator.wikimedia.org/T398818#10984411 (10brennen) I'd like to aim for 2025-07-14 for the migration. [15:00:40] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2025-07-08 - https://phabricator.wikimedia.org/T398976#10984438 (10brennen) 05Open→03Declined Pushing this one. [15:00:55] (03merge) 10brennen: update submodules for 2025-07-08 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/74 (https://phabricator.wikimedia.org/T398976) [15:01:07] 10Phabricator: Unclear what the point of tokens in Phabricator is - https://phabricator.wikimedia.org/T899#10984445 (10Chealer) >>! In T899#10981188, @Aklapper wrote: > @Chealer: What did you reopen? Uh… this report? 🤨 > We don't plan to disable the tokens feature. We don't plan to clarify what tokens are... [15:07:23] 06Project-Admins: Create project tag for Reader Experiments - https://phabricator.wikimedia.org/T398973#10984476 (10lwatson) I requested a new repo `mediawiki/extensions/ReaderExperiments` https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests#mediawiki/extensions/ReaderExperiments [15:08:59] 10Phabricator: Unclear what the point of tokens in Phabricator is - https://phabricator.wikimedia.org/T899#10984481 (10Aklapper) Sure. As the maintainer of Wikimedia Phabricator, I do not plan to address this issue. Thus this issue shall remain in "declined" state. Thanks for your understanding. [15:11:06] 06Project-Admins, 10Release-Engineering-Team (Doing 😎): Create project tag for Reader Experiments - https://phabricator.wikimedia.org/T398973#10984495 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #reader-experiments has been created: https://phabricator.wikimedia.org/project/v... [15:13:37] 10Phabricator (Upstream), 07Upstream: Phabricator does not warn about (apparent) duplicates when creating a new task - https://phabricator.wikimedia.org/T45#10984505 (10Aklapper) @Chealer: Please do not rename feature request titles in this way. It's not a bug ("does not warn"). It's a feature ("should suggest... [15:14:07] 10Phabricator (Upstream), 07Upstream: Phabricator should suggest possible duplicates when creating a new task - https://phabricator.wikimedia.org/T45#10984520 (10Aklapper) [15:24:23] (03open) 10bd808: kubernetes: build a v1.28 cluster [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/17 (https://phabricator.wikimedia.org/T396936 https://phabricator.wikimedia.org/T398643) [15:36:05] 10Phabricator (Upstream), 07Upstream: Phabricator should suggest possible duplicates when creating a new task - https://phabricator.wikimedia.org/T45#10984683 (10Chealer) @Aklapper: This is not a feature, but―like the vast majority of tickets―an issue report, with an implicit (sometimes explicit) request to so... [15:46:46] (03PS1) 10Hashar: Script to clean user starred changes [integration/gerrit-admin] - 10https://gerrit.wikimedia.org/r/1167248 (https://phabricator.wikimedia.org/T355794) [15:48:12] (03CR) 10Hashar: [C:03+2] Script to clean user starred changes [integration/gerrit-admin] - 10https://gerrit.wikimedia.org/r/1167248 (https://phabricator.wikimedia.org/T355794) (owner: 10Hashar) [15:48:38] (03Merged) 10jenkins-bot: Script to clean user starred changes [integration/gerrit-admin] - 10https://gerrit.wikimedia.org/r/1167248 (https://phabricator.wikimedia.org/T355794) (owner: 10Hashar) [15:53:27] 10Phabricator, 06collaboration-services: Update Phabricator 429 message - https://phabricator.wikimedia.org/T398316#10984771 (10Aklapper) See also https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Blocked_help for user docs for a similar issue. [15:58:48] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 07Performance Issue: Drop a bunch of Maniphest Custom Fields from fulltext indexing - https://phabricator.wikimedia.org/T398399#10984805 (10Aklapper) [16:14:13] (03PS1) 10Pppery: Zuul: Add Osuji Pius Nakachi to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1167255 [16:15:03] 10Phabricator: Unclear what the point of tokens in Phabricator is - https://phabricator.wikimedia.org/T899#10984885 (10Chealer) >>! In T899#10984481, @Aklapper wrote: > Sure. As the maintainer of Wikimedia Phabricator, I do not plan to address this issue. Nobody asked you to do that. Once again, I hope you... [16:18:07] 10Phabricator: Unclear what the point of tokens in Phabricator is - https://phabricator.wikimedia.org/T899#10984913 (10Dzahn) @Chealer What are you trying to fix please? [16:20:52] (03approved) 10dduvall: api.py: Revise spiderpig login session handling [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/880 (https://phabricator.wikimedia.org/T397457) (owner: 10dancy) [16:25:45] 10Phabricator, 06collaboration-services: Update Phabricator 429 message - https://phabricator.wikimedia.org/T398316#10984969 (10Dzahn) We want to strike a delicate balance here between NOT letting abusers know in too much detail why they are throttled and ALSO letting legit users have a way to get unblocked an... [16:26:29] (03merge) 10dancy: Revert "spiderpig: Define `access_log_middleware` prior to `SessionMiddleware`" [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/879 [16:26:31] (03update) 10dancy: api.py: Revise spiderpig login session handling [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/880 (https://phabricator.wikimedia.org/T397457) [16:27:05] (03update) 10dancy: api.py: Revise spiderpig login session handling [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/880 (https://phabricator.wikimedia.org/T397457) [16:27:10] (03update) 10dancy: api.py: Revise spiderpig login session handling [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/880 (https://phabricator.wikimedia.org/T397457) [16:27:18] !log T289318: Add ATS routing to profile::trafficserver::backend::mapping_rules in Hiera (Horizon pupet prefix: cache-text) for a wmcloud version of config-master.wikimedia.beta.wmflabs.org [16:27:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:27:20] T289318: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318 [16:27:48] (03merge) 10dancy: api.py: Revise spiderpig login session handling [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/880 (https://phabricator.wikimedia.org/T397457) [16:28:30] !log T289318: Add ATS routing to profile::trafficserver::backend::mapping_rules in Hiera (Horizon pupet prefix: cache-text) for a wmcloud version of stream.wikimedia.beta.wmflabs.org at stream.wikimedia.beta.wmcloud.org [16:28:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:28:56] 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-Phabricator: Add the 'other assignee' field to the Phabricator test instance - https://phabricator.wikimedia.org/T398732#10984982 (10Dzahn) 05Open→03Resolved Thank you. I will call it resolved. The paste you linked to does not seem... [16:34:29] (03open) 10dancy: Release 4.187.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/881 [16:36:09] (03update) 10bd808: kubernetes: build a v1.28 cluster [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/17 (https://phabricator.wikimedia.org/T396936 https://phabricator.wikimedia.org/T398643) [16:36:24] !log T289318: Change purge_host_regex in Hiera (Horizon pupet prefix: cache-text) to include upload.wikimedia.beta.wmcloud.org and maps.wikimedia.beta.wmcloud.org [16:36:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:36:26] T289318: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318 [16:36:33] (03merge) 10dancy: Release 4.187.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/881 [16:37:02] 06Release-Engineering-Team, 06collaboration-services, 10VPS-project-Phabricator: Add the 'other assignee' field to the Phabricator test instance - https://phabricator.wikimedia.org/T398732#10985108 (10Aklapper) >>! In T398732#10981512, @Dzahn wrote: > You can be bold with this one. If you just want to ad... [16:38:57] !log T289318: Change purge_host_regex in Hiera (Horizon pupet prefix: cache-upload) to include upload.wikimedia.beta.wmcloud.org and maps.wikimedia.beta.wmcloud.org [16:38:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:48:36] 06Release-Engineering-Team, 10Scap (SpiderPig 🕸️): Add a favicon for spiderpig.wikimedia.org - https://phabricator.wikimedia.org/T394104#10985170 (10dancy) 05Open→03Resolved Deployed via scap 4.187.0. It may take a while for the favicon.ico to be retrieved by browsers that have previously visited spid... [16:48:59] 10Beta-Cluster-Infrastructure, 10MW-1.45-notes (1.45.0-wmf.9; 2025-07-08), 13Patch-For-Review: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318#10985176 (10Krinkle) >>! In T289318#10985093, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-releng),... [16:50:57] 06Release-Engineering-Team, 10Scap (SpiderPig 🕸️): SpiderPig session / login broken - https://phabricator.wikimedia.org/T397457#10985249 (10dancy) 05Open→03Resolved a:03dancy Fix deployed in scap 4.187.0. Lemme know if you have any further issues. Thanks to @thcipriani for creating a setup for repr... [16:56:10] (03CR) 10Jforrester: [C:03+2] Zuul: Add Osuji Pius Nakachi to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1167255 (owner: 10Pppery) [16:56:17] 10Phabricator, 06DBA: Switchover m3 (phabricator) master db1213 -> db1250 - https://phabricator.wikimedia.org/T398818#10985293 (10brennen) Gah, sorry, correction: 2025-07-15. 14th is Monday, aiming for Tuesday the 15th. [16:56:23] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services: Deploy Phabricator/Phorge 2025-07-08 - https://phabricator.wikimedia.org/T398976#10985295 (10Dzahn) seems resolved instead of declined? [16:57:57] (03Merged) 10jenkins-bot: Zuul: Add Osuji Pius Nakachi to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1167255 (owner: 10Pppery) [16:57:58] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services: Deploy Phabricator/Phorge 2025-07-08 - https://phabricator.wikimedia.org/T398976#10985298 (10Aklapper) We didn't do a deployment today [17:01:31] !log Zuul: Add Osuji Pius Nakachi to CI allowlist [17:01:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:05:39] (03merge) 10bd808: kubernetes: build a v1.28 cluster [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/17 (https://phabricator.wikimedia.org/T396936 https://phabricator.wikimedia.org/T398643) [17:16:54] !log T289318: Change profile::tlsproxy::envoy in Hiera (Horizon pupet prefix: deployment-ms-fe) to include upload.wikimedia.beta.wmcloud.org [17:16:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:16:57] T289318: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318 [17:40:39] !log T289318, T396088: Add DNS for irc.wikimedia.beta.wmcloud.org pointing to deployment-ircd03, same as irc.beta.wmflabs.org (while at it, addig missing "wikimedia" part to the new new wmcloud domain to match prod). [17:40:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:40:43] T289318: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318 [17:40:44] T396088: irc.beta.wmflabs.org is unresponsive - https://phabricator.wikimedia.org/T396088 [18:03:10] !log T289318: Change DNS for upload.wikimedia.beta.wmflabs.org from cache-upload to cache-text to expose new redirect. [18:03:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:03:12] T289318: Move *.beta.wmflabs.org to *.beta.wmcloud.org - https://phabricator.wikimedia.org/T289318 [18:08:00] 10Phabricator, 06collaboration-services: consider moving aphlict admin port to https / envoy - https://phabricator.wikimedia.org/T340169#10985591 (10Dzahn) While the upstream docs say "each server runs a "client" service and an "admin" service" I don't see an explanation yet what the "admin service" actually d... [18:10:32] Krinkle: some of the service names (DNS records) you removed in https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Deployment-prep/Help&diff=0&oldid=2321660 seem to still be in the Designate config. [18:11:11] bd808: do you mean the first row about cache-text and config-master/wikidata? [18:11:45] config-master.wikimedia.beta.wmflabs.org, m.wikidata.beta.wmflabs.org, wikidata.beta.wmflabs.org all are still in DNS [18:13:02] yeah, I removed those because it seemed redundant to list everything out. It wasn't complete anyway. [18:13:14] cache-text is defacto where everything goes that isn't listed elsewhere. [18:13:57] config-master browser hasn't worked for a couple of months in beta. wikidata isn't particularly notable. [18:14:33] we could list performance, and wikipedia, and commons, and wiivoyage, and www. I don't see the benefit of that particular table being a complete list though. Don't feel strongly about it though, it just seemed easy to clean up [18:15:29] well... ok. At some point I was trying to make that page document what the config is reguardless of if it is "right" so we could actually figure out what to remove. [18:18:03] Things like config-master.wikimedia.beta.wmflabs.org not working are true, but do not invalidate that there is a DNS record pointed to the 185.15.56.36 public IP for it [18:22:46] (03open) 10bd808: ci: Add pre-commit config and fix violations [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/18 [18:22:58] bd808: ack, there's well over a hundred DNS records pointing to x.36 [18:23:46] most for wikis and their m-dot, but also wildcards covering non-wikis such as performance.wikimedia and stream.wikimedia which doesnt have its own DNS but does travelt through cache-text [18:29:20] > Error: Unable to delete proxy: stream-beta.wmflabs.org [18:30:33] s/well over a hundred/~80/ (2x20 on beta.wmflabs, 2x20 pages on beta.wmcloud) [18:34:50] (03approved) 10brennen: set session_expire_delay to 90 days [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/73 (https://phabricator.wikimedia.org/T398791) (owner: 10jelto) [19:08:19] 10Phabricator, 06collaboration-services: consider moving aphlict admin port to https / envoy - https://phabricator.wikimedia.org/T340169#10985929 (10Dzahn) 18:16 < andre> https://we.phorge.it/rP9a7ad972cd2637e566ac7598357b88364d6f2b64 might not be the greatest read at the fireplace but that's when code got spl... [19:39:08] 10Phabricator: Unclear what the point of tokens in Phabricator is - https://phabricator.wikimedia.org/T899#10986043 (10Chealer) Greetings Daniel, >>! In T899#10984913, @Dzahn wrote: > @Chealer What are you trying to fix please? First, I do not consider this as a bug report. I did mention one minor defect,... [19:56:29] (03update) 10bd808: ci: Add pre-commit config and fix violations [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/18 [19:58:44] (03update) 10bd808: ci: Add pre-commit config and fix violations [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/18 [20:02:44] (03update) 10bd808: ci: Add pre-commit config and fix violations [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/18 [20:06:33] (03update) 10bd808: ci: Add pre-commit config and fix violations [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/18 [20:07:13] trial and error debugging in that gitlab MR. such fun [20:09:26] (03update) 10bd808: ci: Add pre-commit config and fix violations [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/18 [20:11:28] 10Phabricator: Unclear what the point of tokens in Phabricator is - https://phabricator.wikimedia.org/T899#10986114 (10Dzahn) >>! In T899#10986042, @Chealer wrote: > Greetings Daniel, Greetings, Chealer. > First, I do not consider this as a bug report. Well, it was reported by another user over 10 years a... [20:20:00] James_F: How does a project name in jjb/project-pipelines.yaml refer to a project name in Gerrit? [20:20:29] for example I see project called "codesearch" or "miscweb" but those are not repo names [20:20:56] repo names are stuff like "operations/container/miscweb" [20:21:15] and in Gerrit language those are also projects [20:22:27] oh, I think I just answered my own question.. by finding a second change in zuul/layout.yaml that does this [20:26:14] (03PS1) 10Dzahn: add pipelines for codesearch-sourcebot [integration/config] - 10https://gerrit.wikimedia.org/r/1167288 (https://phabricator.wikimedia.org/T268199) [20:30:27] (03PS1) 10Dzahn: zuul: add config for codesearch-sourcebot [integration/config] - 10https://gerrit.wikimedia.org/r/1167289 (https://phabricator.wikimedia.org/T268199) [21:21:14] 10Release-Engineering-Team (Priority Backlog 📥), 07OKR-Work: Move nightly image build from releases-jenkins to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T398873#10986402 (10bd808) https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/blob/master/conf/releasing/casc/jobs/publishMWSingleVer... [21:43:33] 10Release-Engineering-Team (Priority Backlog 📥), 07OKR-Work: Move nightly image build from releases-jenkins to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T398873#10986462 (10dancy) a:03dancy [21:51:26] 10Release-Engineering-Team (Priority Backlog 📥), 07OKR-Work: Move nightly image build from releases-jenkins to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T398873#10986478 (10bd808) [22:12:46] 10Release-Engineering-Team (Priority Backlog 📥), 07OKR-Work: Move nightly image build from releases-jenkins to deployment.eqiad.wmnet - https://phabricator.wikimedia.org/T398873#10986501 (10dancy) @dduvall There's a hazard in the way that the `MediaWiki branch and publish WMF single-version image` job handles... [22:22:17] 10GitLab (Auth & Access), 10Release-Engineering-Team (Radar), 10CAS-SSO, 06collaboration-services, 06Infrastructure-Foundations: Add GitLab to offboarding workflow - https://phabricator.wikimedia.org/T339843#10986537 (10Dzahn) Since infrastructure-security owns the offboarding workflow (and has the offbo... [22:25:26] 06Release-Engineering-Team, 06collaboration-services, 10mwcli, 06serviceops-radar: Create /nonexistent directory for nobody user in golang images - https://phabricator.wikimedia.org/T331209#10986577 (10Dzahn) ping @Addshore This ticket is still open but possibly resolved since 2023. thoughts? [22:36:11] 10Release-Engineering-Team (Radar), 06collaboration-services, 06SRE: Redirect revisions from svn.wikimedia.org to https://static-codereview.wikimedia.org - https://phabricator.wikimedia.org/T119846#10986675 (10Dzahn) I am uploading a patch for that. I did notice though that the "SVN repo browser" part is st... [22:58:29] (03update) 10bd808: ci: Add pre-commit config and fix violations [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/18 [23:11:44] (03update) 10bd808: ci: Add pre-commit config and fix violations [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/18 [23:25:04] (03update) 10bd808: ci: Add pre-commit config and fix violations [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/18 [23:52:26] (03update) 10bd808: ci: Add pre-commit config and fix violations [repos/releng/zuul/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/merge_requests/18