[06:59:42] 06Release-Engineering-Team, 06collaboration-services: gerrit: Throttle traffic to gitiles - https://phabricator.wikimedia.org/T420595#11731234 (10ABran-WMF) p:05Medium→03High [07:08:05] 06Release-Engineering-Team, 06collaboration-services: gerrit: Throttle traffic to gitiles - https://phabricator.wikimedia.org/T420595#11731239 (10ABran-WMF) from the dumps extracted by @thcipriani: ` - 2026-03-19 20:45:32... [07:43:03] 06Release-Engineering-Team, 06collaboration-services: gerrit: Throttle traffic to gitiles - https://phabricator.wikimedia.org/T420595#11731276 (10ABran-WMF) I've created 2 new url patterns for gitiles: - https://requestctl.wikimedia.org/pattern/url/gitiles_coslty_urls_primary - https://requestctl.wikimedia.org... [08:35:06] 06Release-Engineering-Team, 06MediaWiki-Engineering, 10MediaWiki-extensions-WikimediaMaintenance, 10Wiki-Setup (Create): addWiki.php fails with CannotReplaceActiveServiceException for DBLoadBalancerFactory - https://phabricator.wikimedia.org/T420531#11731391 (10A_smart_kitten) [09:15:19] (03PS1) 10Phedenskog: phpunit: Support parallel execution for default test suite [integration/quibble] - 10https://gerrit.wikimedia.org/r/1256264 [09:18:07] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Phabricator does not show archived milestones as archived - https://phabricator.wikimedia.org/T420605#11731488 (10Aklapper) p:05Triage→03Low a:03Aklapper [09:25:40] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for PieWriter - https://phabricator.wikimedia.org/T420709 (10PieWriter) 03NEW [09:33:15] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for PieWriter - https://phabricator.wikimedia.org/T420709#11731579 (10PieWriter) 05Open→03Resolved [09:33:37] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for PieWriter - https://phabricator.wikimedia.org/T420709#11731580 (10PieWriter) a:03PieWriter [09:37:12] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for PieWriter - https://phabricator.wikimedia.org/T420709#11731587 (10PieWriter) Once I connected my LDAP to phabricator, it was automatically approved. Thanks [10:13:52] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): RuntimeException: Undefined offset: 2 in PhutilMediaWikiAuthAdapter - https://phabricator.wikimedia.org/T420711 (10Aklapper) 03NEW [10:44:39] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥), 10Wikimedia-Phabricator-Extensions: RuntimeException: Undefined offset: 2 in PhutilMediaWikiAuthAdapter - https://phabricator.wikimedia.org/T420711#11731771 (10Aklapper) Added exception handling in https://gitlab.wikimedia.org/repos/phabricator/... [11:05:10] (03PS1) 10Bhsd: Zuul: [CodeMirror] Add AbuseFilter dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1256318 (https://phabricator.wikimedia.org/T399673) [11:07:12] (03CR) 10CI reject: [V:04-1] Zuul: [CodeMirror] Add AbuseFilter dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1256318 (https://phabricator.wikimedia.org/T399673) (owner: 10Bhsd) [11:08:06] (03PS2) 10Bhsd: Zuul: [CodeMirror] Add AbuseFilter dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1256318 (https://phabricator.wikimedia.org/T399673) [11:49:31] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Phabricator does not render archived milestone tags as archived - https://phabricator.wikimedia.org/T420605#11731924 (10Aklapper) 05Open→03Stalled Upstreamed as https://we.phorge.it/T16567; Patch proposed in https://we.phorge.it/... [13:54:57] 10Beta-Cluster-Infrastructure, 06Content-Transform-Team: Maps don't display on the beta cluster; tile server offline on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420299#11732259 (10Jgiannelos) I think that the issue here is that kartotherian used to be... [14:07:48] 10Beta-Cluster-Infrastructure: Some production sites on the beta cluster were unable to be selected as import targets - https://phabricator.wikimedia.org/T410109#11732316 (10TTO) I suspect the conversion of the import form to OOUI was done badly. Have a look at the `data-ooui` attribute on the subproject dropd... [14:26:00] 10Beta-Cluster-Infrastructure, 06Content-Transform-Team: Maps don't display on the beta cluster; tile server offline on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420299#11732404 (10Jgiannelos) I think to bring again deployment-prep up to speed with pro... [14:44:28] 10Beta-Cluster-Infrastructure, 06Content-Transform-Team: Maps don't display on the beta cluster; tile server offline on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420299#11732461 (10Jgiannelos) On a different note (cc @ihurbain) since we barely do any d... [14:57:20] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/AbuseFilter] Add dependency on CodeMirror [integration/config] - 10https://gerrit.wikimedia.org/r/1256318 (https://phabricator.wikimedia.org/T399673) (owner: 10Bhsd) [14:57:22] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/AbuseFilter] Add dependency on CodeMirror [integration/config] - 10https://gerrit.wikimedia.org/r/1256318 (https://phabricator.wikimedia.org/T399673) (owner: 10Bhsd) [14:59:02] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/AbuseFilter] Add dependency on CodeMirror [integration/config] - 10https://gerrit.wikimedia.org/r/1256318 (https://phabricator.wikimedia.org/T399673) (owner: 10Bhsd) [14:59:29] !log Zuul: [mediawiki/extensions/AbuseFilter] Add dependency on CodeMirror, for T399673 [14:59:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:59:32] T399673: Add CodeMirror mode for AbuseFilter syntax - https://phabricator.wikimedia.org/T399673 [15:06:09] 10GitLab, 06Release-Engineering-Team: GitLab Private Repository Request for: MediaWiki JS CSS Scan Signatures, MediaWiki JS CSS Scan Tool - https://phabricator.wikimedia.org/T420646#11732572 (10sbassett) >>! In T420646#11730518, @brennen wrote: > Seems to fit [[ https://www.mediawiki.org/wiki/GitLab/Hosting_a_... [15:08:05] 06Release-Engineering-Team, 06collaboration-services: gerrit: Throttle traffic to gitiles - https://phabricator.wikimedia.org/T420595#11732576 (10ABran-WMF) [15:09:34] 06Release-Engineering-Team, 06collaboration-services: gerrit: Throttle traffic to gitiles - https://phabricator.wikimedia.org/T420595#11732577 (10Dzahn) Yesterday we changed `gerrit_throttle_gitiles` from throttling to actually `dropping`. Technically it should be renamed (which Arnaudb just did, thanks) The... [15:25:54] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: Replace nginx ingress with traefik for gitlab-cloud-runners - https://phabricator.wikimedia.org/T420743 (10dancy) 03NEW [15:27:08] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: Replace nginx ingress with traefik for gitlab-cloud-runners - https://phabricator.wikimedia.org/T420743#11732670 (10dancy) [15:37:49] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11732690 (10hashar) //@ABran-WMF and I further exchanged about on Thursday... [15:38:17] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11732691 (10hashar) ### Config troubleshooting The event mpm was designed... [15:42:33] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11732733 (10hashar) //I think that concludes this task (debugging), and we... [15:44:29] (03open) 10dancy: Replace nginx ingress with traefik [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/562 (https://phabricator.wikimedia.org/T420743) [15:44:33] (03update) 10dancy: Replace nginx ingress with traefik [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/562 (https://phabricator.wikimedia.org/T420743) [15:49:38] (03update) 10dancy: Replace nginx ingress with traefik [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/562 (https://phabricator.wikimedia.org/T420743) [15:58:18] (03update) 10dancy: Replace nginx ingress with traefik [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/562 (https://phabricator.wikimedia.org/T420743) [16:08:05] 10Beta-Cluster-Infrastructure, 06Content-Transform-Team: Maps don't display on the beta cluster; tile server offline on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420299#11732858 (10bd808) >>! In T420299#11732460, @Jgiannelos wrote: > Can we point mw on... [16:35:14] 10Beta-Cluster-Infrastructure, 06Content-Transform-Team: Maps don't display on the beta cluster; tile server offline on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420299#11733039 (10MSantos) I agree with @Jgiannelos and @ihurbain. We should reduce compl... [17:02:57] 06Release-Engineering-Team (Priority Backlog 📥), 07OKR-Work: wmf/next branch cut job on releases-jenkins and systemd timer on deployment server times overlap - https://phabricator.wikimedia.org/T404399#11733120 (10bd808) >>! In T404399#11174624, @bd808 wrote: > Setting the branch cut job to 01:00 UTC and the b... [17:17:15] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Fix access check for multiple env deletion endpoint - https://phabricator.wikimedia.org/T420596#11733189 (10jnuche) 05Open→03Resolved Fix is in production [17:26:34] PROBLEM - jenkins_service_running on contint1003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [17:34:29] 10Phabricator, 06collaboration-services, 10VPS-project-Phabricator, 13Patch-For-Review: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11733274 (10A_smart_kitten) 05Open→03In progress a:03A_smart_kitten >>! In T388022#11719821, @A... [18:18:34] RECOVERY - jenkins_service_running on contint1003 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [18:24:27] ^ this is me working on it. not the prod jenkins. fixed downtimes too [19:00:59] (03PS1) 10MusikAnimal: Add Alien333 to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1256488 (https://phabricator.wikimedia.org/T402086) [19:01:16] (03PS2) 10MusikAnimal: Zuul: add Alien333 to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/1256488 (https://phabricator.wikimedia.org/T402086) [20:43:15] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#11733945 (10Dzahn) [20:55:28] 06Release-Engineering-Team (Priority Backlog 📥): Collect data on merge to deploy cycle times for the rolling `wmf/next` branch so we can report on Pretrain impacts - https://phabricator.wikimedia.org/T420790 (10bd808) 03NEW [21:07:43] 06Release-Engineering-Team (Priority Backlog 📥): Collect data on merge to deploy cycle times for the rolling `wmf/next` branch so we can report on Pretrain impacts - https://phabricator.wikimedia.org/T420790#11734025 (10bd808) One thing we will need to be aware of in creating this data pipeline is that with the... [21:16:52] 10Continuous-Integration-Infrastructure, 07Jenkins: Verify the Jenkins Gearman plugin works under Java 21 - https://phabricator.wikimedia.org/T420178#11734030 (10hashar) I have finally went to test the plugin and based my work on what I have done with T271683. This is what I did: **Setup Gearman** and start... [21:20:04] 10Continuous-Integration-Infrastructure, 07Jenkins: Verify the Jenkins Gearman plugin works under Java 21 - https://phabricator.wikimedia.org/T420178#11734039 (10hashar) 05Open→03Resolved a:03hashar **Test** Admire a script I wrote several years ago (January 2022?): ` name=invoke.py,lang=python #!/u... [21:20:32] 10GitLab, 06Release-Engineering-Team: GitLab Private Repository Request for: MediaWiki JS CSS Scan Signatures, MediaWiki JS CSS Scan Tool - https://phabricator.wikimedia.org/T420646#11734049 (10brennen) 05Open→03Resolved Created: - https://gitlab.wikimedia.org/repos/security/mw-jss-css-scan-tool - htt... [21:21:27] !log Unblock 103.159.218.0/24 (T420530) [21:21:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:22:57] 10Continuous-Integration-Infrastructure, 07Jenkins, 06collaboration-services: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109#11734060 (10hashar) I have validated the Jenkins Gearman plugin does work with Java 21 from Trixie (T420178). One less blocker! [21:49:12] 06Release-Engineering-Team (Priority Backlog 📥): Extend Kubernetes Logstash annotations to include the OCI container image and tag - https://phabricator.wikimedia.org/T420794 (10bd808) 03NEW [21:50:15] 06Release-Engineering-Team (Priority Backlog 📥): Extend Kubernetes Logstash annotations to include the OCI container image and tag - https://phabricator.wikimedia.org/T420794#11734131 (10bd808) In a prior discussion of this need @akosiaris said: > This will require to alter the chart to add the extra labels. It'... [22:07:30] Yippee, build fixed! [22:07:30] Project beta-scap-sync-world build #250050: 09FIXED in 2 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/250050/ [22:52:56] (03open) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331) [22:56:46] (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331) [23:00:23] (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331) [23:06:27] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11734360 (10thcipriani) Two thoughts on this: - I worry it would confuse newcomers and add admin load if you had to be trusted to do many common tasks. That said, anti-v... [23:11:39] (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331) [23:12:38] 10Beta-Cluster-Infrastructure, 06Content-Transform-Team: Maps don't display on the beta cluster; tile server offline on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420299#11734371 (10bd808) >>! In T420299#11733039, @MSantos wrote: > I agree with @Jgianne... [23:12:38] (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331) [23:15:19] (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331) [23:18:41] (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331) [23:27:14] 10Phabricator: Move "Task search" to the top of the default phabricator sidebar nav - https://phabricator.wikimedia.org/T420801 (10thcipriani) 03NEW [23:35:12] Project beta-code-update-eqiad build #592525: 04FAILURE in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/592525/ [23:40:26] (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331) [23:45:22] Yippee, build fixed! [23:45:22] Project beta-code-update-eqiad build #592526: 09FIXED in 2 min 21 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/592526/ [23:55:52] (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331)