[00:07:45] 10Phabricator, 06Release-Engineering-Team: Temporarily replace the Phabricator logo for Pride Month (2026) - https://phabricator.wikimedia.org/T427739#12078507 (10brennen) 05In progress→03Resolved Reset to standard logo. See you all on next year's task... [00:10:03] (03update) 10brennen: Draft: Update fundraising ML image to use WSGI [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/109 (owner: 10ejegg) [04:08:57] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12078788 (10AlexisJazz) ` Request served via deployment-cache-text08 deployment-cache-text08, Varnish XID 644382721 Error: 503, Backend fetch failed at... [05:33:13] 10Phabricator, 06DBA: Switchover m3 (phabricator) master (db1250 -> db1228) - https://phabricator.wikimedia.org/T430158#12078832 (10Marostegui) [05:39:56] 10Phabricator, 06DBA, 13Patch-For-Review: Switchover m3 (phabricator) master (db1250 -> db1228) - https://phabricator.wikimedia.org/T430158#12078850 (10Marostegui) [05:42:05] 10Phabricator, 06DBA, 13Patch-For-Review: Switchover m3 (phabricator) master (db1250 -> db1228) - https://phabricator.wikimedia.org/T430158#12078851 (10Marostegui) test [05:43:08] 10Phabricator, 06DBA, 13Patch-For-Review: Switchover m3 (phabricator) master (db1250 -> db1228) - https://phabricator.wikimedia.org/T430158#12078852 (10Marostegui) [05:43:39] 10Phabricator, 06DBA, 13Patch-For-Review: Switchover m3 (phabricator) master (db1250 -> db1228) - https://phabricator.wikimedia.org/T430158#12078853 (10Marostegui) [05:43:51] 10Phabricator, 06DBA, 13Patch-For-Review: Switchover m3 (phabricator) master (db1250 -> db1228) - https://phabricator.wikimedia.org/T430158#12078854 (10Marostegui) 05Open→03Resolved This was done [05:59:30] (03update) 10kbach: Rules: allow valid media types and handle polymorphic properties [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/14 (https://phabricator.wikimedia.org/T428149) (owner: 10milazg) [06:00:44] 10Continuous-Integration-Infrastructure (Zuul upgrade): Enable Zuul OpenTelemetry tracing - https://phabricator.wikimedia.org/T430900 (10hashar) 03NEW [06:02:28] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, and 2 others: Get traces/logs from ATS on Gerrit CI errors - https://phabricator.wikimedia.org/T430901 (10ABran-WMF) 03NEW [06:33:21] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, and 2 others: Get traces/logs from ATS on Gerrit CI errors - https://phabricator.wikimedia.org/T430901#12078972 (10ABran-WMF) **(Analysis below written by Claude - with Fable 5 xhigh effort)** I analyz... [07:08:15] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 10observability: Enable Zuul OpenTelemetry tracing - https://phabricator.wikimedia.org/T430900#12078985 (10ABran-WMF) [07:10:51] 10Gerrit, 06collaboration-services, 13Patch-For-Review: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#12078994 (10ABran-WMF) 05Open→03Resolved >>! In T411583#12074620, @ABran-WMF wrote: > Since we're now out of the woods, I think this task can be considered `Resolved` once [[... [07:30:35] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, and 2 others: Get traces/logs from ATS on Gerrit CI errors - https://phabricator.wikimedia.org/T430901#12079033 (10ABran-WMF) p:05Triage→03High [07:31:43] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, and 2 others: Fetches from Gerrit aborted due to: GnuTLS recv error (-54): Error in the pull function - https://phabricator.wikimedia.org/T420865#12079049 (10ABran-WMF) 05Resolved→03Stalled Marking... [07:32:55] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, and 2 others: Fetches from Gerrit aborted due to: GnuTLS recv error (-54): Error in the pull function - https://phabricator.wikimedia.org/T420865#12079063 (10ABran-WMF) 05Stalled→03Resolved [07:33:18] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, and 2 others: Gerrit: network errors in CI - https://phabricator.wikimedia.org/T429749#12079065 (10ABran-WMF) 05In progress→03Stalled Marking this `Stalled` while {T430901} is being handled [07:38:20] (03Abandoned) 10Arnaudb: jjb: trace git HTTPS transport on gated quibble jobs to debug gerrit resets [integration/config] - 10https://gerrit.wikimedia.org/r/1305393 (https://phabricator.wikimedia.org/T429749) (owner: 10Arnaudb) [08:10:10] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Translate: Beta cluster is not working due to error in Translate\TtmServer\ServiceCreationFailure - https://phabricator.wikimedia.org/T430882#12079220 (10atsuko) It can be the case if `'service' => $wmgLocalServices[$wmgTranslateK8SBackend . '-dnsdisc']` is... [08:24:52] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 06collaboration-services, 10Data-Services, 13Patch-For-Review: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12079302 (10hashar) The Zuul executor refers to the list of no proxy dom... [09:11:38] 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 13Patch-For-Review, 05Release, 05Train Deployments: 1.47.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T423918#12079507 (10Aklapper) Deployed to final group2. Things look fine so far when it comes to numbers. {T430916} worries me a... [09:19:54] (03CR) 10Hashar: "v14-recheck no clue whether Zuul does something on pre review patches made to a config project. I think I have witnessed a config error i" [integration/config] - 10https://gerrit.wikimedia.org/r/1306484 (owner: 10Hashar) [09:27:39] (03CR) 10Hashar: "On the `mediawiki/core` noop child change, Zuul reports https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1306498/1#message-7d50376c6f582f" [integration/config] - 10https://gerrit.wikimedia.org/r/1306484 (owner: 10Hashar) [09:30:28] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for Open Data Ivoir - https://phabricator.wikimedia.org/T430467#12079643 (10Aklapper) a:03Aklapper Reopening as T429857 seems to take its time, so let's create this manually. [09:31:26] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, and 2 others: Get traces/logs from ATS on Gerrit CI errors - https://phabricator.wikimedia.org/T430901#12079653 (10Joe) (**Analysis below written by Giuseppe 1.0 on moderate effort**) :P There is at l... [09:32:46] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for Open Data Ivoir - https://phabricator.wikimedia.org/T430467#12079664 (10Aklapper) 05Declined→03Resolved Requested public project tag #tool-odivoir has been created: https://phabricator.wikimedia.org/project/view/8853/ (In c... [09:32:56] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for Open Data Ivoir (Tool-odivoir) - https://phabricator.wikimedia.org/T430467#12079674 (10Aklapper) [09:45:40] 10Phabricator, 06cloud-services-team, 10Striker, 06tools-platform-team: Striker dev environment needs a new Phabricator base image - https://phabricator.wikimedia.org/T340080#12079776 (10dcaro) [09:50:07] 06Release-Engineering-Team (Doing 😎), 06cloud-services-team, 10Striker, 06tools-platform-team: [striker] Failing to create new phabricator projects - https://phabricator.wikimedia.org/T430120#12079791 (10Aklapper) a:03Aklapper [09:58:33] 06Release-Engineering-Team (Deployment Training Requests): Deployment training request for ssastry - https://phabricator.wikimedia.org/T353383#12079825 (10Aklapper) 05Open→03Resolved Assuming this took place; please reopen if there is still more to do. :) [09:59:12] 06Release-Engineering-Team (Deployment Training Requests): Deployment training request for Sergio Gimeno - https://phabricator.wikimedia.org/T320722#12079830 (10Aklapper) 05Open→03Resolved Assuming this took place at some point; please reopen if there is still more to do. :) [09:59:54] 06Release-Engineering-Team (Doing 😎), 06cloud-services-team, 10Striker, 06tools-platform-team: [striker] Failing to create new phabricator projects - https://phabricator.wikimedia.org/T430120#12079857 (10taavi) [10:10:34] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Translate, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): Beta cluster is not working due to error in Translate\TtmServer\ServiceCreationFailure - https://phabricator.wikimedia.org/T430882#12079916 (10atsuko) [10:10:35] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Translate, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): Beta cluster is not working due to error in Translate\TtmServer\ServiceCreationFailure - https://phabricator.wikimedia.org/T430882#12079917 (10atsuko) 05Open→03In progress [10:28:52] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎), 06cloud-services-team, 10Striker: Striker: Always set all IO to none and hidden for internal URIs when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341#12079994 (10Aklapper) This got deployed today; I should veri... [10:34:38] Hi! if I need to deploy mw-config to a beta cluster, do I need to go thru scap, or should I just +2 the change? The change in question: https://gerrit.wikimedia.org/r/1307099 [10:47:17] !log zuul1002: running Puppet agent to drop `wikimediacloud.org` from `no_proxy` | T430479 [10:47:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:47:19] T430479: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479 [10:48:02] atsukoito: those can be merged / deployed at anytime afaik [10:48:17] scap / spiderpig should be smart enough to detect it only affects beta cluster [10:48:24] it would update the repo on the deployment server [10:48:32] but otherwise does not trigger a sync on the production cluster [10:48:36] HOPEFULLY / IIRC [10:48:48] it is always wise to sync up in -operations in case someone is doing an operation [10:48:58] there might be a policy/doc about that somewhere [10:49:34] I can totally squeeze it in the next deployment window and run scap/spiderpig process. [10:50:09] or that yeah, thought here is some delay [10:50:21] and that is to unbreak the beta cluster isn't it? [10:50:30] yes, to unbreak beta [10:50:52] lets switch to -operations and process it :] [10:51:14] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Translate, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: Beta cluster is not working due to error in Translate\TtmServer\ServiceCreationFailure - https://phabricator.wikimedia.org/T430882#12080036 (10atsuko) p:05Triage→03High [10:53:15] !og zuul1002: restarted zuul-executor service | T430479 [10:53:15] T430479: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479 [11:01:30] hashar: `10:57:04 Skipping sync since all commits were beta/labs-only changes. Operation completed.` yeah, it did handle it alright [11:06:43] atsukoito: awesome!!! [11:06:49] thank you for the notice [11:16:44] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Translate, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: Beta cluster is not working due to error in Translate\TtmServer\ServiceCreationFailure - https://phabricator.wikimedia.org/T430882#12080060 (10atsuko) Confirming that `$wgTran... [12:01:19] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Translate, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: Beta cluster is not working due to error in Translate\TtmServer\ServiceCreationFailure - https://phabricator.wikimedia.org/T430882#12080220 (10atsuko) 05In progress→03R... [12:08:05] hashar: does beta cluster work for you? I see that most of the pages still doesn't load, but I don't see much in logstash.. not sure it is related to the change [12:08:21] we did initially, it wasn't 2 days ago [12:10:58] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 06collaboration-services, 10Data-Services: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12080271 (10hashar) The ansible playbook can now reach out the WMCS OpenStack API. There is an... [12:30:39] 06Release-Engineering-Team (Doing 😎), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T423918#12080505 (10Aklapper) 05Open→03Resolved 🔥This is fine.🔥 [12:35:14] 06Release-Engineering-Team (Doing 😎), 06cloud-services-team, 10Striker, 06tools-platform-team: [striker] Failing to create new phabricator projects - https://phabricator.wikimedia.org/T430120#12080517 (10Aklapper) 05Open→03Resolved Closing as @StrikerBot succeeded in creating https://phabricator.wi... [12:46:54] (03update) 10aklapper: Draft: Set IO_NONE by default on repos; always expose observed URI [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/103 (https://phabricator.wikimedia.org/T405596) [13:13:22] 10Continuous-Integration-Infrastructure, 10MediaWiki-Vendor, 07Composer: Upgrade composer to >= 2.10.1 - https://phabricator.wikimedia.org/T428570#12080675 (10Jdforrester-WMF) a:03Jdforrester-WMF [13:21:42] (03PS1) 10Jforrester: Docker: [composer-scratch] Upgrade composer to 2.10.2 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/1307145 (https://phabricator.wikimedia.org/T428570) [13:21:44] (03PS1) 10Jforrester: jjb: Update jobs to use images with composer 2.10.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1307146 (https://phabricator.wikimedia.org/T428570) [13:23:27] (03CR) 10Jforrester: [C:03+2] Docker: [composer-scratch] Upgrade composer to 2.10.2 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/1307145 (https://phabricator.wikimedia.org/T428570) (owner: 10Jforrester) [13:25:09] (03Merged) 10jenkins-bot: Docker: [composer-scratch] Upgrade composer to 2.10.2 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/1307145 (https://phabricator.wikimedia.org/T428570) (owner: 10Jforrester) [13:26:49] (03PS2) 10Jforrester: jjb: Update jobs to use images with composer 2.10.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1307146 (https://phabricator.wikimedia.org/T428570) [13:27:20] !log Docker: [composer-scratch] Upgrade composer to 2.10.2 and cascade, for T428570 [13:27:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:27:22] T428570: Upgrade composer to >= 2.10.1 - https://phabricator.wikimedia.org/T428570 [13:46:06] 10Continuous-Integration-Infrastructure, 10MediaWiki-Vendor, 07Composer, 13Patch-For-Review: Upgrade composer to >= 2.10.1 - https://phabricator.wikimedia.org/T428570#12080856 (10Reedy) [13:49:02] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 10Horizon, 10Openstack-Magnum, 06tools-infrastructure-team: Can't retrieve Magnum cluster certificate and config via Horizon - https://phabricator.wikimedia.org/T430680#12080892 (10hashar) **SUCCESS*** thank you @Andrew... [13:51:01] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 06collaboration-services, 10Data-Services: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12080912 (10hashar) The Cloud Config file is stored in https://gerrit.wikimedia.org/g/integrat... [13:56:26] (03CR) 10Jforrester: [C:03+2] jjb: Update jobs to use images with composer 2.10.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1307146 (https://phabricator.wikimedia.org/T428570) (owner: 10Jforrester) [13:57:08] 10Continuous-Integration-Infrastructure, 10MediaWiki-Vendor, 07Composer, 13Patch-For-Review: Upgrade composer to >= 2.10.1 - https://phabricator.wikimedia.org/T428570#12080967 (10Jdforrester-WMF) [13:57:59] 10Continuous-Integration-Infrastructure, 10MediaWiki-Vendor, 07Composer, 13Patch-For-Review: Upgrade composer to >= 2.10.1 - https://phabricator.wikimedia.org/T428570#12080974 (10Jdforrester-WMF) a:05Jdforrester-WMF→03None The bits I can do are now done. [13:58:35] (03Merged) 10jenkins-bot: jjb: Update jobs to use images with composer 2.10.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1307146 (https://phabricator.wikimedia.org/T428570) (owner: 10Jforrester) [14:05:08] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥), 06collaboration-services, 13Patch-For-Review: Disable IO for diffusion repositories - https://phabricator.wikimedia.org/T405596#12081025 (10Aklapper) Some progress: As `SELECT COUNT(r.name) AS pulls, r.name FROM phabricator_rep... [14:16:08] 10Phabricator, 06DBA: Improve Phabricator update reliability - https://phabricator.wikimedia.org/T430936#12081102 (10Aklapper) [14:22:54] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 06collaboration-services, 10Data-Services: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12081131 (10hashar) For comparison with an application credential for the `zuul` project I hav... [14:40:48] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, and 2 others: Get traces/logs from ATS on Gerrit CI errors - https://phabricator.wikimedia.org/T430901#12081230 (10ABran-WMF) Thanks for taking a look! For the sake of thoroughness, I asked Claude to fe... [15:11:13] 06Project-Admins, 06Moderator-Tools-Team: Create project tag for ModeratorToolkit - https://phabricator.wikimedia.org/T430980 (10DMburugu) 03NEW [15:12:05] 06Project-Admins, 06Moderator-Tools-Team: Create project tag for ModeratorToolkit - https://phabricator.wikimedia.org/T430980#12081483 (10DMburugu) [15:12:52] 06Project-Admins, 06Moderator-Tools-Team: Create project tag for ModeratorToolkit - https://phabricator.wikimedia.org/T430980#12081484 (10DMburugu) [15:17:38] 06Project-Admins, 06Release-Engineering-Team (Doing 😎), 10MediaWiki-extensions-ModeratorToolkit, 06Moderator-Tools-Team: Create project tag for ModeratorToolkit - https://phabricator.wikimedia.org/T430980#12081506 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project tag #MediaWiki... [15:29:34] 10Beta-Cluster-Infrastructure: Unbalanced handling of beta cluster traffic - https://phabricator.wikimedia.org/T430984 (10dancy) 03NEW [15:31:10] 06Project-Admins: Create a #traffic-rate-limit tag - https://phabricator.wikimedia.org/T424643#12081635 (10BCornwall) What about a more general tag, such as #waf, that would cover more components than just rate limiting? [15:40:05] 06Project-Admins, 06Release-Engineering-Team (Doing 😎), 10MediaWiki-extensions-ModeratorToolkit, 06Moderator-Tools-Team: Create project tag for ModeratorToolkit - https://phabricator.wikimedia.org/T430980#12081667 (10Samwalton9-WMF) Added to {H437}. [15:46:15] 10Beta-Cluster-Infrastructure: Unbalanced handling of beta cluster traffic - https://phabricator.wikimedia.org/T430984#12081696 (10dancy) 05Open→03Invalid @bd808 explained to me that the two nodes serve different purposes. `deployment-mediawiki13` is the single node api server cluster and `deployment-me... [15:46:51] 06Project-Admins: Create a #traffic-rate-limit tag - https://phabricator.wikimedia.org/T424643#12081703 (10Aklapper) What does waf stand for? [15:48:00] 10Beta-Cluster-Infrastructure: Unbalanced handling of beta cluster traffic - https://phabricator.wikimedia.org/T430984#12081711 (10bd808) The balancing would be in the Beta cluster CDN edge somewhere rather then the WMCS shared proxies. I think all of `*.beta.wmcloud.org` is the CDN copy on it's own IP. I s... [16:03:38] 10Continuous-Integration-Config, 10MediaWiki-extensions-ModeratorToolkit, 06Moderator-Tools-Team: Set up continuous integration on the ModeratorToolkit extension - https://phabricator.wikimedia.org/T430991 (10DMburugu) 03NEW [16:04:38] 10Continuous-Integration-Config, 10MediaWiki-extensions-ModeratorToolkit, 06Moderator-Tools-Team: Set up continuous integration on the ModeratorToolkit extension - https://phabricator.wikimedia.org/T430991#12081825 (10DMburugu) [16:04:59] 10Continuous-Integration-Config, 10MediaWiki-extensions-ModeratorToolkit, 06Moderator-Tools-Team: Set up continuous integration on the ModeratorToolkit extension - https://phabricator.wikimedia.org/T430991#12081827 (10DMburugu) [16:05:18] 10Continuous-Integration-Config, 10MediaWiki-extensions-ModeratorToolkit, 06Moderator-Tools-Team: Set up continuous integration on the ModeratorToolkit extension - https://phabricator.wikimedia.org/T430991#12081831 (10DMburugu) [16:41:03] 10Continuous-Integration-Infrastructure (Zuul upgrade): zuul-client encrypt fails to retrieve public key - https://phabricator.wikimedia.org/T431003 (10hashar) 03NEW [16:46:52] 10Continuous-Integration-Infrastructure (Zuul upgrade): zuul-client encrypt fails to retrieve public key - https://phabricator.wikimedia.org/T431003#12082130 (10hashar) [16:46:57] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06cloud-services-team, 06collaboration-services, 10Data-Services: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12082129 (10hashar) [16:50:26] 10Continuous-Integration-Infrastructure (Zuul upgrade): zuul-client encrypt fails to retrieve public key - https://phabricator.wikimedia.org/T431003#12082161 (10Dzahn) Is that curl command generated by zuul-client or something else or is that just you testing? Where does "curl -X 'GET' \ 'https://zuul.wikimed... [16:52:56] (03PS1) 10Hashar: zuul: change OpenStack secret [integration/config] - 10https://gerrit.wikimedia.org/r/1307204 (https://phabricator.wikimedia.org/T430479) [16:53:41] 10Continuous-Integration-Infrastructure (Zuul upgrade): zuul-client encrypt fails to retrieve public key - https://phabricator.wikimedia.org/T431003#12082172 (10Dzahn) a claim is made that: //'"This happens to prevent directory traversal attacks. Apache evaluates the URL and intercepts it before your proxy rules... [16:54:20] (03CR) 10Hashar: [C:03+2] zuul: change OpenStack secret [integration/config] - 10https://gerrit.wikimedia.org/r/1307204 (https://phabricator.wikimedia.org/T430479) (owner: 10Hashar) [16:54:20] 10Continuous-Integration-Infrastructure (Zuul upgrade): zuul-client encrypt fails to retrieve public key - https://phabricator.wikimedia.org/T431003#12082173 (10Dzahn) We might want `AllowEncodedSlashes On` in httpd config if we want to allow them. [16:56:08] (03Merged) 10jenkins-bot: zuul: change OpenStack secret [integration/config] - 10https://gerrit.wikimedia.org/r/1307204 (https://phabricator.wikimedia.org/T430479) (owner: 10Hashar) [16:58:30] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Translate, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: Beta cluster is not working due to error in Translate\TtmServer\ServiceCreationFailure - https://phabricator.wikimedia.org/T430882#12082194 (10atsuko) Also, Beta cluster i... [17:02:52] 06Project-Admins: Create a #traffic-rate-limit tag - https://phabricator.wikimedia.org/T424643#12082210 (10BCornwall) Web Application Firewall - While the primary function of such things is for security purposes, most modern WAFs also provide rate-limiting (e.g. https://docs.aws.amazon.com/waf/latest/developergu... [17:30:41] 10Phabricator, 06Release-Engineering-Team: Default Phorge search box refreshes / blanks after a paste - https://phabricator.wikimedia.org/T431012 (10brennen) 03NEW [17:32:30] 10Phabricator, 06Release-Engineering-Team: Default Phorge search box refreshes / blanks after a paste - https://phabricator.wikimedia.org/T431012#12082355 (10brennen) [17:42:22] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12082393 (10bd808) >>! In T420833#12078788, @AlexisJazz wrote: > What would happen if https://en.wikipedia.org/wiki/Basic_access_authentication was enab... [17:46:00] 06Gerrit-Privilege-Requests: Request membership in toollabs-trusted & wmcs-trusted groups for tlepage - https://phabricator.wikimedia.org/T431014 (10TLepage-WMF) 03NEW [17:46:31] 06Gerrit-Privilege-Requests: Request membership in toollabs-trusted & wmcs-trusted groups for tlepage - https://phabricator.wikimedia.org/T431014#12082419 (10TLepage-WMF) [17:47:11] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12082421 (10bd808) `lang=shell-session root@deployment-mediawiki14:~# ./small-ban-hammer.sh 1 +1 | sort -n -k 4 | grep '# 2 hits' | wc -l 53136 ` 2 hits... [17:52:22] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: zuul-client encrypt fails to retrieve public key - https://phabricator.wikimedia.org/T431003#12082433 (10Dzahn) I tested this. When allowing encoded slashes using "On" there is still a 404 but now it does not come from Apache httpd any... [17:53:57] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: zuul-client encrypt fails to retrieve public key - https://phabricator.wikimedia.org/T431003#12082436 (10Dzahn) When we use `AllowEncodedSlashes NoDecode` to ensure it also does not try to decode them and just pass them on to zuul, it... [17:58:06] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎), 06cloud-services-team, 10Striker: Striker: Always set all IO to none and hidden for internal URIs when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341#12082465 (10Aklapper) Meh, not fixed. Per https://phabricato... [18:03:07] 06Release-Engineering-Team (Doing 😎), 06WMF-NDA-Requests: New joiner request to access #wmf-nda - https://phabricator.wikimedia.org/T431006#12082478 (10Aklapper) 05Open→03Resolved a:03Aklapper Done! [18:08:22] 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: zuul-client encrypt fails to retrieve public key - https://phabricator.wikimedia.org/T431003#12082488 (10Dzahn) @hashar fixed! ` [zuul1001:~] $ curl -X 'GET' \ 'https://zuul.wikimedia.org/api/tenant/wikimedia/key/integration%2Fconf... [18:09:18] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: zuul-client encrypt fails to retrieve public key - https://phabricator.wikimedia.org/T431003#12082490 (10Dzahn) [18:09:37] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: zuul-client encrypt fails to retrieve public key - https://phabricator.wikimedia.org/T431003#12082505 (10Dzahn) a:03Dzahn [18:36:11] 10Phabricator, 06Release-Engineering-Team: Default Phorge search box refreshes / blanks after a paste - https://phabricator.wikimedia.org/T431012#12082641 (10Aklapper) I don't think I've experienced this so far in Chromium or Firefox (but I usually run latest versions). What does "quickly" mean here exactly? D... [18:41:16] 10Phabricator, 06Release-Engineering-Team: Default Phorge search box refreshes / blanks after a paste - https://phabricator.wikimedia.org/T431012#12082663 (10dancy) >>! In T431012#12082641, @Aklapper wrote: > I don't think I've experienced this so far in Chromium or Firefox (but I usually run latest versions).... [18:52:42] 10Phabricator, 06Release-Engineering-Team: Default Phorge search box refreshes / blanks after a paste - https://phabricator.wikimedia.org/T431012#12082702 (10dancy) I captured a video of the "refresh" behavior. {F91586365} [18:59:29] 10Phabricator, 06Release-Engineering-Team: Default Phorge search box refreshes / blanks after a paste - https://phabricator.wikimedia.org/T431012#12082741 (10Aklapper) Huh, strange. Now I'm curious what the ["network" tab of the browser's developer tools](https://developers.google.com/web/tools/chrome-devtools... [19:20:01] (03PS1) 10Ottomata: phan: Add CentralAuth as EventBus phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1307228 (https://phabricator.wikimedia.org/T430967) [19:39:57] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T430830#12082918 (10thcipriani) a:05dancy→03jeena [19:40:28] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T430831#12082921 (10thcipriani) a:05dduvall→03dancy [19:41:08] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T430832#12082926 (10thcipriani) a:05jeena→03dduvall [20:48:11] 10MediaWiki-Releasing, 06Security-Team, 07SecTeam-Processed, 07Security: Write and send supplementary release announcement for extensions and skins with security patches (1.43.9/1.44.6/1.45.4) - https://phabricator.wikimedia.org/T421273#12083284 (10Mstyles) Email from T411394#11796980 has been sent to vari... [21:01:15] 10MediaWiki-Releasing, 06Security-Team, 07SecTeam-Processed, 07Security: Write and send supplementary release announcement for extensions and skins with security patches (1.43.9/1.44.6/1.45.4) - https://phabricator.wikimedia.org/T421273#12083320 (10sbassett) 05In progress→03Resolved a:03Mstyles >... [21:01:53] 10Phabricator, 06Release-Engineering-Team: Default Phorge search box refreshes / blanks after a paste - https://phabricator.wikimedia.org/T431012#12083327 (10dancy) >>! In T431012#12082741, @Aklapper wrote: > Huh, strange. Now I'm curious what the ["network" tab of the browser's developer tools](https://develo... [21:27:23] James_F: https://beta-update.wmcloud.org/202607022120.log [21:27:26] (03open) 10taavi: phab: Replace deprecated endpoints [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/116 (https://phabricator.wikimedia.org/T427620) [21:49:03] (03open) 10dancy: requirements: Bump vulnerable dependencies to address Dependabot alerts [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1214 [21:51:19] (03open) 10dancy: package-lock.json: Updates from npm audit fix [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1215 [21:57:00] 06Release-Engineering-Team (Priority Backlog 📥): [FY25-26 WE6.7.3] Automated supervision - https://phabricator.wikimedia.org/T427968#12083551 (10bd808) [22:26:04] 10Diffusion, 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Title of commit message cut off in Diffusion after a full stop - https://phabricator.wikimedia.org/T130557#12083669 (10Aklapper) Fixed in upstream (after tagging of the 2026.27 release). [22:49:45] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Priority Backlog 📥), 10Cloud-VPS (Debian Bullseye Deprecation), 07Epic: Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12083722 (10bd808) [22:50:20] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Priority Backlog 📥), 10Cloud-VPS (Debian Bullseye Deprecation), 07Epic: Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12083724 (10bd808)