[00:04:05] (03open) 10samwilson: Use HTTP client from Api class for Wikidata, to set User-Agent [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/7 [00:17:41] (03CR) 10Lokal Profil: Recreate source tables as part of deployment (032 comments) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1259245 (owner: 10Jean-Frédéric) [00:30:01] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Add PTR record for azwikimedia - https://phabricator.wikimedia.org/T421025 (10Nemoralis) 03NEW [00:30:47] 10Cloud-VPS (Project-requests): Request creation of azwikimedia VPS project - https://phabricator.wikimedia.org/T417736#11741620 (10Nemoralis) [00:30:51] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Add floating IP and vanity domain for azwikimedia project - https://phabricator.wikimedia.org/T419582#11741621 (10Nemoralis) [00:43:40] 10Tool-wikimonitor: Secure Persistent “Remember Me” Login with Split-Key Encryption for Wikimedia OAuth2 Tokens - https://phabricator.wikimedia.org/T420946#11741633 (10Gerges) p:05Triage→03Low [01:12:00] RESOLVED: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [01:18:03] FIRING: [2x] PuppetDisabled: Puppet disabled on cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [01:42:05] 10Cloud-VPS (Project-requests): Request creation of etherpads3 VPS project - https://phabricator.wikimedia.org/T420532#11741720 (10komla) @bd808 this has been done! Sorry about the delay [01:42:13] 10Cloud-VPS (Project-requests): Request creation of etherpads3 VPS project - https://phabricator.wikimedia.org/T420532#11741721 (10komla) 05Open→03Resolved [01:47:48] FIRING: [2x] PuppetDisabled: Puppet disabled on cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [02:17:48] RESOLVED: PuppetDisabled: Puppet disabled on cloudbackup1002-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [02:25:11] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Add PTR record for azwikimedia - https://phabricator.wikimedia.org/T421025#11741787 (10Andrew) users had not actually allocated a floating IP, but I have now done so. It is: 185.15.56.85 [02:35:29] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Add PTR record for azwikimedia - https://phabricator.wikimedia.org/T421025#11741802 (10Andrew) 05Open→03Resolved a:03Andrew ` andrew@bookworm:~/tofu-infra/resources/eqiad1-r/cloudinfra$ dig -x 185.15.56.85 ; <<>> DiG 9.18.33-1~deb12u2-Debian <<>>... [04:12:43] (03CR) 10MusikAnimal: [C:03+2] "All tests pass now when I run `composer test` locally (with `app.is_wmf` set), so we're all good :)" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1255922 (https://phabricator.wikimedia.org/T322466) (owner: 10Alien4444) [04:13:22] (03Merged) 10jenkins-bot: Check replicas are there when the dblists say so [labs/xtools] - 10https://gerrit.wikimedia.org/r/1255922 (https://phabricator.wikimedia.org/T322466) (owner: 10Alien4444) [04:17:45] (03CR) 10MusikAnimal: [C:03+2] Add discord channel and invite links to footer [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257431 (owner: 10Alien4444) [04:18:21] (03Merged) 10jenkins-bot: Add discord channel and invite links to footer [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257431 (owner: 10Alien4444) [06:22:31] (03CR) 10Lokal Profil: Recreate source tables as part of deployment (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1259245 (owner: 10Jean-Frédéric) [07:03:12] (03PS1) 10Krinkle: cvn-api: Set expose_php=On in php.ini [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1259738 [07:03:54] (03CR) 10Krinkle: [V:03+2 C:03+2] cvn-api: Set expose_php=On in php.ini [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1259738 (owner: 10Krinkle) [07:12:58] (03PS1) 10Krinkle: build: Raise dev deps for PHP 8.2+, make PHP 8.5 passing [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1259749 (https://phabricator.wikimedia.org/T419075) [07:13:16] (03CR) 10CI reject: [V:04-1] build: Raise dev deps for PHP 8.2+, make PHP 8.5 passing [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1259749 (https://phabricator.wikimedia.org/T419075) (owner: 10Krinkle) [07:20:27] (03PS2) 10Krinkle: build: Raise dev deps for PHP 8.2+, make PHP 8.5 passing [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1259749 (https://phabricator.wikimedia.org/T419075) [07:20:42] (03CR) 10Krinkle: "check experimental" [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1259749 (https://phabricator.wikimedia.org/T419075) (owner: 10Krinkle) [07:47:52] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, and 2 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11742071 (10Aklapper) That custom Phabricator code would be in https://gitlab.wikimedia.org/repos/... [07:52:52] (03CR) 10Krinkle: "recheck" [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1259749 (https://phabricator.wikimedia.org/T419075) (owner: 10Krinkle) [07:53:11] (03CR) 10Krinkle: [C:03+2] build: Raise dev deps for PHP 8.2+, make PHP 8.5 passing [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1259749 (https://phabricator.wikimedia.org/T419075) (owner: 10Krinkle) [07:53:41] (03Merged) 10jenkins-bot: build: Raise dev deps for PHP 8.2+, make PHP 8.5 passing [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/1259749 (https://phabricator.wikimedia.org/T419075) (owner: 10Krinkle) [08:05:50] (03merge) 10arthurtaylor: Reduce job-time variance using weighted average [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/12 (https://phabricator.wikimedia.org/T420884) [08:45:22] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: rabbitmqctl list_queues in eqiad/codfw times out after 60s - https://phabricator.wikimedia.org/T420923#11742202 (10fgiunchedi) After some investigation I found the following: 1. rabbitmq cli utils use tcp ports [[ https://www.rabbitmq.com/docs/networking#... [09:23:33] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11742308 (10Dondersmooi) The actual OAuth server URL is stored in the database (PROPERTY_MEDIAWIKI... [09:43:05] (03approved) 10dcaro: toolforge_deploy: add restore-all [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/314 [09:43:19] (03merge) 10dcaro: toolforge_deploy: add restore-all [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/314 [09:51:48] (03approved) 10dcaro: deploy: reply to HEAD requests [toolforge-repos/sample-dotnet-buildpack-app] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-dotnet-buildpack-app/-/merge_requests/2 [09:51:53] (03merge) 10dcaro: deploy: reply to HEAD requests [toolforge-repos/sample-dotnet-buildpack-app] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-dotnet-buildpack-app/-/merge_requests/2 [10:30:08] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [10:31:18] !log filippo@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [10:32:52] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [10:34:02] !log filippo@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) on deployment codfw1dev for all services [10:35:26] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for service: project,nova,glance,keystone,cinder,neutron,trove,magnum,octavia,heat,swift [10:39:37] !log filippo@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for service: project,nova,glance,keystone,cinder,neutron,trove,magnum,octavia,heat,swift [10:44:28] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11742630 (10A_smart_kitten) FWIW, changing the OAuth domain used by Phabricator from mediawiki.org... [10:45:29] 10Tool-wikimonitor, 07good first task, 07patch-welcome: Unable to Edit SpEL Condition from Mobile - https://phabricator.wikimedia.org/T420409#11742648 (10BHUVANSH855) Hi @Gerges , I’m a new contributor preparing for GSoC 2026. I’ve reviewed the issue and the discussion so far. I’d like to contribute by spe... [10:45:32] 06cloud-services-team, 10Toolforge, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Allow codewords to refer to known IP-ranges in oauth consumer registration - https://phabricator.wikimedia.org/T421049 (10Effeietsanders) 03NEW [10:47:05] (03update) 10dcaro: inject_buildpacks: skip dotnet if builtin exits [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/84 [10:48:48] 06cloud-services-team, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Allow codewords to refer to known IP-ranges in oauth consumer registration - https://phabricator.wikimedia.org/T421049#11742690 (10taavi) [10:54:08] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11742712 (10A_smart_kitten) (IIUC, it would also mean that the logo & site-name in the "Log In or... [10:55:19] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11742715 (10taavi) Meh. Doing Phabricator authentication via mw.o does have some minor advantages,... [11:06:38] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11742745 (10Dondersmooi) >In T420599#11742712, @A_smart_kitten wrote: > True; however, I guess my... [11:07:32] 10Tool-wsindex, 10Wikisource Reader App (Android), 10Outreachy (Round 31): Outreachy 31: Improve the Wikisource Reader App - https://phabricator.wikimedia.org/T405593#11742752 (10Bodhisattwa) [11:12:52] (03approved) 10fnegri: core: update jobs in storage too [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/277 (owner: 10dcaro) [11:13:48] (03merge) 10dcaro: core: update jobs in storage too [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/277 [11:16:09] 10Tool-wsindex, 10Wikisource Reader App (Android), 10Outreachy (Round 31): Outreachy 31: Improve the Wikisource Reader App - https://phabricator.wikimedia.org/T405593#11742783 (10Bodhisattwa) 05Open→03Resolved a:03Muguro The final summary report is published - https://www.mediawiki.org/wiki/Outreac... [11:17:42] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.479-20260324111403-11f5772c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1181 [11:17:47] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.479-20260324111403-11f5772c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1181 [11:39:19] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#11742854 (10fnegri) `s3` and `x3` traffic is now routed to `clouddb102[23]`: `lang=shell-session fnegri@cumin1003:~$ sudo confctl select servic... [11:46:03] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Move all openstack rabbitmq queues to quorum - https://phabricator.wikimedia.org/T421054 (10fgiunchedi) 03NEW [11:47:32] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Data-Services, 06Data-Persistence: clouddb1013 crashed after the upgrade to mariadb 10.11.16 - https://phabricator.wikimedia.org/T420177#11742880 (10fnegri) I have depooled `clouddb1017@s3`, all `s3` traffic is now going to `clouddb1022` and `clouddb1023`, which a... [12:29:52] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:31:29] (03close) 10dcaro: support start and end params in logs endpoint [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/267 (https://phabricator.wikimedia.org/T400917) (owner: 10raymond-ndibe) [12:42:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [12:51:20] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:56:56] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Move all openstack rabbitmq queues to quorum - https://phabricator.wikimedia.org/T421054#11743154 (10Andrew) > because they are declared classic and not quorum by default. In theory, openstack is creating these queues as quorum; ` [oslo_messaging_rabbi... [13:02:16] 10Tool-centralnotice-banner-editor: Add undo functionality to the editor - https://phabricator.wikimedia.org/T421061 (10Oyelola_Victoria) 03NEW [13:04:16] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:10:32] 10Tool-centralnotice-banner-editor: Add option to set border, margin, and padding for all sides at once - https://phabricator.wikimedia.org/T421063 (10Oyelola_Victoria) 03NEW [13:11:47] 10Tool-wikimonitor, 07good first task, 07patch-welcome: Unable to Edit SpEL Condition from Mobile - https://phabricator.wikimedia.org/T420409#11743250 (10Vishaldevops) @Gerges i have started working on the issue, also have added a patch of code in the apps,js program to resolve the issue. Basically earlier... [13:21:11] 10Tool-centralnotice-banner-editor: Banner-level and text-level link options conflict with each other - https://phabricator.wikimedia.org/T421068 (10Oyelola_Victoria) 03NEW [13:37:01] 10Tool-centralnotice-banner-editor: Automatically strip protocol from URLs - https://phabricator.wikimedia.org/T421070 (10Oyelola_Victoria) 03NEW [13:41:45] (03update) 10fnegri: WIP: Introduce diff-mode [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/10 (https://phabricator.wikimedia.org/T351637) [13:41:49] (03update) 10fnegri: WIP: Introduce diff-mode [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/10 (https://phabricator.wikimedia.org/T351637) [13:43:50] 10Tool-centralnotice-banner-editor: Add image lookup within the editor - https://phabricator.wikimedia.org/T421072 (10Oyelola_Victoria) 03NEW [13:49:58] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11743666 (10A_smart_kitten) >>! In T420599#11742745, @Dondersmooi wrote: > My understanding is tha... [13:51:10] (03approved) 10dcaro: jobs-api: bump to 0.0.479-20260324111403-11f5772c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1181 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:51:13] (03merge) 10dcaro: jobs-api: bump to 0.0.479-20260324111403-11f5772c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1181 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:00:44] 10Tool-centralnotice-banner-editor: Add link to text modal overflows and cuts off in small screens - https://phabricator.wikimedia.org/T421101 (10Oyelola_Victoria) 03NEW [14:02:14] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11743968 (10bd808) >>! In T420599#11730366, @bd808 wrote: > The clients could be reconfigured, but... [14:07:19] 10Tool-wikimonitor, 07good first task, 07patch-welcome: Display flagged edits count in browser tab title (dynamic update) - https://phabricator.wikimedia.org/T420850#11744111 (10Praffq) a:03Praffq @Gerges I'll like to work on this issue. Should I raise a PR directly on github repo for it? [14:28:34] 10Tool-wikimonitor, 07good first task, 07patch-welcome: Unable to Edit SpEL Condition from Mobile - https://phabricator.wikimedia.org/T420409#11744257 (10Vishaldevops) {F73565632} @Gerges I checked whther this works using mobile view in google dev tool and yes i am able to type and the cursor appears, is the... [14:34:01] (03update) 10dcaro: replace job images with web images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/263 (https://phabricator.wikimedia.org/T415322) (owner: 10raymond-ndibe) [14:34:22] 10Tool-wikimonitor, 07good first task, 07patch-welcome: Unable to Edit SpEL Condition from Mobile - https://phabricator.wikimedia.org/T420409#11744285 (10Vishaldevops) @Gerges I tested whether the patch i added fixed the issue using the mobile device emulation in Chrome DevTools. The editor now correctly... [14:35:08] 10Tool-centralnotice-banner-editor: Some image fields overflow and cut off in small screens - https://phabricator.wikimedia.org/T421120 (10Oyelola_Victoria) 03NEW [14:35:15] (03update) 10dcaro: values.yaml: replace job image variants with webservice image variants [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/18 (https://phabricator.wikimedia.org/T415322) (owner: 10raymond-ndibe) [14:38:06] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, and 3 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11744332 (10Dondersmooi) >> In T420599#11743968, @bd808 wrote: > Developers, including the folks w... [14:42:41] 10Tool-wikimonitor, 07good first task, 07patch-welcome: Unable to Edit SpEL Condition from Mobile - https://phabricator.wikimedia.org/T420409#11744359 (10Gerges) @Vishaldevops, Good to hear that. please submit a pull request so we can test it on the beta server. [15:05:20] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-30 (2026-03-24 to 2026-04-07)): Fix linter issues discovered during implementation of the OAD example - https://phabricator.wikimedia.org/T414974#11744496 (10HCoplin-WMF) [15:34:33] (03update) 10fnegri: Introduce diff-mode [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/10 (https://phabricator.wikimedia.org/T351637) [15:34:40] (03update) 10fnegri: Introduce diff-mode [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/10 (https://phabricator.wikimedia.org/T351637) [15:55:13] (03update) 10fnegri: Introduce diff-mode [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/10 (https://phabricator.wikimedia.org/T351637) [15:57:42] 10Tool-wikimonitor, 07good first task, 07patch-welcome: Unable to Edit SpEL Condition from Mobile - https://phabricator.wikimedia.org/T420409#11744798 (10Vishaldevops) @Gerges i have submitted a pull request for review https://github.com/wiki-connect/wikimonitor/pull/32 Thank you. [16:13:03] 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure, 07Jenkins: Large temporary quota increase for integration project to ease Jenkins migration - https://phabricator.wikimedia.org/T421139 (10hashar) 03NEW [16:15:00] 10Cloud-VPS (Quota-requests), 10Continuous-Integration-Infrastructure, 07Jenkins: Large temporary quota increase for integration project to ease Jenkins migration - https://phabricator.wikimedia.org/T421139#11744946 (10hashar) [16:27:35] PROBLEM - Host cloudbackup2003 is DOWN: PING CRITICAL - Packet loss = 100% [16:30:32] 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Power Supply - Status - issue on cloudbackup2003:9290 - https://phabricator.wikimedia.org/T420948#11745154 (10Jhancock.wm) there were a few power supplies that went down in the same rack. it wasn't a breaker trip. all on different channels on the PDUs. I... [16:32:37] RECOVERY - Host cloudbackup2003 is UP: PING OK - Packet loss = 0%, RTA = 30.24 ms [16:38:12] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, and 2 others: Inconsistent OAuth endpoint for Wikimedia Global Account (SUL) across services - https://phabricator.wikimedia.org/T420599#11745208 (10A_smart_kitten) >>! In T420599#11742745, @Dondersmooi wrote: > I thought the purpose o... [17:07:49] 10VPS-project-Codesearch: Codesearch stuck at Feb 12th? - https://phabricator.wikimedia.org/T421147 (10Michael) 03NEW [17:20:07] (03approved) 10fnegri: inject_buildpacks: skip dotnet if builtin exits [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/84 (owner: 10dcaro) [17:23:37] (03merge) 10dcaro: inject_buildpacks: skip dotnet if builtin exits [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/84 [17:25:51] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-builder: bump to 0.0.143-20260324172359-c67d1d81 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1182 (https://phabricator.wikimedia.org/T380127) [17:25:58] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-builder: bump to 0.0.143-20260324172359-c67d1d81 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1182 (https://phabricator.wikimedia.org/T380127) [17:37:33] (03update) 10fnegri: Introduce diff-mode [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/10 (https://phabricator.wikimedia.org/T351637) [17:39:08] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [17:43:38] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [17:44:44] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [17:49:41] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [17:53:16] (03approved) 10dcaro: builds-builder: bump to 0.0.143-20260324172359-c67d1d81 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1182 (https://phabricator.wikimedia.org/T380127) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:53:20] (03merge) 10dcaro: builds-builder: bump to 0.0.143-20260324172359-c67d1d81 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1182 (https://phabricator.wikimedia.org/T380127) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:54:20] 10VPS-project-Codesearch: Codesearch stuck at Feb 12th? - https://phabricator.wikimedia.org/T421147#11745763 (10Krinkle) Yeah, the "Deployed" index is 3 weeks behind. https://codesearch.wmcloud.org/deployed/?q=lcobucci%2Fclock%7CLcobucci.%5B%5EJ%5D#MediaWiki%20core vs https://codesearch.wmcloud.org/core/?q=lc... [18:00:35] (03PS1) 10Krinkle: write_config: clarify "skipping restart" log message [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1260063 (https://phabricator.wikimedia.org/T421147) [18:01:28] 10VPS-project-Codesearch, 13Patch-For-Review: Codesearch stuck at Feb 12th? - https://phabricator.wikimedia.org/T421147#11745811 (10Dzahn) ` Hound is still starting up, please wait a few minutes for the initial indexing to complete. See for more information. ` [18:01:39] (03CR) 10CI reject: [V:04-1] write_config: clarify "skipping restart" log message [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1260063 (https://phabricator.wikimedia.org/T421147) (owner: 10Krinkle) [18:03:19] 10VPS-project-Codesearch, 13Patch-For-Review: Codesearch stuck at Feb 12th? - https://phabricator.wikimedia.org/T421147#11745820 (10Dzahn) The deployed index should have been re-built. Any difference? [18:03:52] 10Cloud-VPS (Quota-requests), 06collaboration-services, 10Continuous-Integration-Infrastructure, 07Jenkins: Large temporary quota increase for integration project to ease Jenkins migration - https://phabricator.wikimedia.org/T421139#11745822 (10Dzahn) [18:19:15] 10VPS-project-Codesearch, 13Patch-For-Review: Codesearch stuck at Feb 12th? - https://phabricator.wikimedia.org/T421147#11745877 (10A_smart_kitten) >>! In T421147#11745820, @Dzahn wrote: > The deployed index should have been re-built. Any difference? I still see e.g. the difference noted in T421147#11745763 b... [19:19:33] (03PS1) 10MusikAnimal: .gitreview: set default branch to main [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260077 [19:21:09] (03CR) 10MusikAnimal: [C:03+2] .gitreview: set default branch to main [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260077 (owner: 10MusikAnimal) [19:21:53] (03Merged) 10jenkins-bot: .gitreview: set default branch to main [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260077 (owner: 10MusikAnimal) [20:20:47] 06cloud-services-team, 10Data-Services, 10Datasets-General-or-Unknown, 06Traffic, 13Patch-For-Review: Move dumps.wikimedia.org HTTP service behind CDN edge - https://phabricator.wikimedia.org/T306550#11746514 (10BBlack) ## On the technical front: * Both the object size (0.5GB) and the total dataset siz... [21:10:27] (03PS1) 10Novem Linguae: set up eslint-config-wikimedia [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260115 (https://phabricator.wikimedia.org/T392531) [21:10:45] (03PS2) 10Novem Linguae: set up eslint-config-wikimedia [labs/xtools] - 10https://gerrit.wikimedia.org/r/1260115 (https://phabricator.wikimedia.org/T392531) [22:02:17] FIRING: JobUnavailable: Reduced availability for job rabbitmq in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [22:07:17] RESOLVED: JobUnavailable: Reduced availability for job rabbitmq in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [22:55:55] (03PS4) 10Alien4444: Add proofreadpage quality support [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012)