[01:48:51] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Privacy Engineering, and 2 others: Add global_edit_count to wikireplicas - https://phabricator.wikimedia.org/T344108#12078674 (10aranyap) The Product Safety & Integrity team has reviewed this table and is considering this to be a low privacy ris... [02:38:22] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Privacy Engineering, and 2 others: Add global_edit_count to wikireplicas - https://phabricator.wikimedia.org/T344108#12078728 (10Soda) >>! In T344108#12078673, @aranyap wrote: > The Product Safety & Integrity team has reviewed this table and is... [04:50:44] (03open) 10countcount: Add a detail parameter to /info and /ipinfo to select returned categories [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/29 [05:01:50] (03update) 10countcount: Add a detail parameter to /info and /ipinfo to select returned categories [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/29 [05:02:11] (03update) 10countcount: Add a detail parameter to /info and /ipinfo to select returned categories [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/29 [05:07:49] (03merge) 10countcount: Add a detail parameter to /info and /ipinfo to select returned categories [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/29 [05:15:33] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/28 [05:50:44] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 06Tech-Docs-Team, 07OKR-Work: Fix description and example issues for transform paths in the MediaWiki REST API OAD (part 1) - https://phabricator.wikimedia.org/T428151#12078863 (10KBach) 05Open→03In progress a:03KBach [06:55:51] (03open) 10countcount: Add a sulrenames detail category tracking pre-SUL local renames [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/30 [07:04:02] (03merge) 10countcount: Add a sulrenames detail category tracking pre-SUL local renames [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/30 [07:15:15] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/28 [07:19:31] (03open) 10countcount: Add an all detail category selecting all detail categories [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/31 [07:21:08] (03merge) 10countcount: Add an all detail category selecting all detail categories [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/31 [07:29:48] 06tools-platform-team: [jobs-cli] Use the new `status` fields instead of the old short/long statuses - https://phabricator.wikimedia.org/T430904 (10dcaro) 03NEW [07:30:16] (03update) 10dcaro: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) (owner: 10damian) [07:30:51] 10Toolforge, 06tools-platform-team: [jobs-cli] Use the new `status` fields instead of the old short/long statuses - https://phabricator.wikimedia.org/T430904#12079039 (10dcaro) [07:31:01] 10Toolforge, 06tools-platform-team: [jobs-api] flaky unit test test_get_one_off_job_status - https://phabricator.wikimedia.org/T430550#12079041 (10dcaro) [07:31:14] 10Toolforge, 06tools-platform-team: [jobs-api] when restarting or deleting a job that does not exist on runtime, you get an exception 404 - https://phabricator.wikimedia.org/T430524#12079043 (10dcaro) [07:31:31] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Drive community adoption of Toolforge Push-to-Deploy - https://phabricator.wikimedia.org/T430181#12079045 (10dcaro) [07:31:50] 10Toolforge, 06tools-platform-team: How to Deploy on Toolforge Workshop - https://phabricator.wikimedia.org/T427961#12079054 (10taavi) [07:31:54] 10Toolforge, 06tools-platform-team: Drive community adoption of Toolforge Push-to-Deploy - https://phabricator.wikimedia.org/T430181#12079055 (10taavi) [07:31:56] 10Toolforge, 06tools-platform-team: [toolforge-weld] update bump_version.sh - https://phabricator.wikimedia.org/T429235#12079058 (10taavi) [07:31:59] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api] using "--filelog" without "--mount" saves invalid jobs into storage - https://phabricator.wikimedia.org/T429907#12079057 (10taavi) [07:32:03] 10Toolforge, 06tools-platform-team: [builds-api] Add support for using a commit hash as ref - https://phabricator.wikimedia.org/T429227#12079059 (10taavi) [07:33:56] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12079072 (10dcaro) [07:33:59] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Drive community adoption of Toolforge Push-to-Deploy - https://phabricator.wikimedia.org/T430181#12079071 (10taavi) [07:34:01] 06cloud-services-team, 10Cloud-VPS: TLS Certificate Provisioning for wikiba.se Subdomain - https://phabricator.wikimedia.org/T430057#12079074 (10taavi) Please set up the AAAA record as incicated on the help page Bryan linked above. [07:37:38] (03open) 10countcount: Rename the sulrenames detail category to localrenames [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/32 [07:39:09] (03merge) 10countcount: Rename the sulrenames detail category to localrenames [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/32 [07:58:54] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:08:09] (03merge) 10taavi: Rename test URL query parameter to avoid conflictsRename testing URL query parameter [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/18 (https://phabricator.wikimedia.org/T421542) (owner: 10shadabgdg) [08:14:13] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review, 07patch-welcome: Toolforge 404 handler displays wrong tool name from ?url= query parameter - https://phabricator.wikimedia.org/T421542#12079238 (10taavi) 05Open→03Resolved [08:15:41] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/28 [08:16:54] (03update) 10dcaro: runtime: remove diff_with_running_job [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/336 [08:19:12] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:19:25] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Provision cloudvirt with firewall - https://phabricator.wikimedia.org/T430310#12079274 (10taavi) [08:19:39] 10Cloud-VPS, 06tools-infrastructure-team: Stop hardcoding cloudcephosd networking data in puppet - https://phabricator.wikimedia.org/T429932#12079276 (10taavi) [08:19:45] 10Cloud-VPS, 06tools-infrastructure-team: [ceph] Stats not reporting slow operations - https://phabricator.wikimedia.org/T430846#12079278 (10taavi) [08:20:16] 10Cloud-VPS, 06tools-infrastructure-team: Implement magnum monitoring - https://phabricator.wikimedia.org/T429557#12079279 (10taavi) [08:24:52] 06cloud-services-team, 10Data-Services, 06collaboration-services, 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12079302 (10hashar) The Zuul executor refers to the list of no proxy dom... [08:37:15] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [08:39:13] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:41:43] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [08:42:46] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:55:20] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [08:56:49] (03approved) 10dcaro: jobs-api: bump to 0.0.523-20260701162132-d3541ac2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1314 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:56:58] (03merge) 10dcaro: jobs-api: bump to 0.0.523-20260701162132-d3541ac2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1314 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [09:06:18] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: puppet failing on metricsinfra-prometheus-2.metricsinfra due to expired crl - https://phabricator.wikimedia.org/T429298#12079489 (10taavi) I rotated the Puppet CA there recently; I wonder if there's a missing step from that process to refresh t... [09:12:33] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Data-Platform-SRE, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918 (10ayounsi) 03NEW p:05Triage→03Medium The #Cloud-Services project tag is not intended to have any tasks. Please check the list on h... [09:18:03] (03CR) 10Gergő Tisza: [C:03+1] Add datalist suggestions to the file path filters [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306099 (https://phabricator.wikimedia.org/T197802) (owner: 10Simon04) [09:22:24] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Data-Platform-SRE, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079602 (10ayounsi) @cmooney @Papaul can one of you take that upgrade ? And update the date/time to anything that suits you in that week. [09:25:03] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Data-Platform-SRE, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079609 (10CWilliams-WMF) [09:25:31] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Data-Platform-SRE, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079614 (10CWilliams-WMF) [09:28:03] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Data-Platform-SRE, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079636 (10ayounsi) [09:30:44] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Data-Platform-SRE, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079648 (10CWilliams-WMF) [09:33:56] 10Tool-odivoir: Développement de la version 1.0 de la plateforme Open Data Ivoir - https://phabricator.wikimedia.org/T430463#12079680 (10Aklapper) [09:34:00] 10Tool-odivoir: Traduire la documentation de la plateforme Open Data Ivoir en allemand - https://phabricator.wikimedia.org/T430891#12079683 (10Aklapper) [09:34:01] 10Tool-odivoir: Traduire la documentation de la plateforme Open Data Ivoir en anglais - https://phabricator.wikimedia.org/T430890#12079684 (10Aklapper) [09:34:06] 10Tool-odivoir: Traduire de la documentation de la plateforme Open Data Ivoir en espagnol - https://phabricator.wikimedia.org/T430905#12079682 (10Aklapper) [09:34:09] 10Tool-odivoir, 10GAWA: Ajouter du contenus sur la page de documentation de la plateforme Open Data Ivoir - https://phabricator.wikimedia.org/T430887#12079685 (10Aklapper) [09:34:13] 10Tool-odivoir: Structurer la page de documentation de la plateforme Open Data Ivoir sur méta-wiki - https://phabricator.wikimedia.org/T430767#12079687 (10Aklapper) [09:34:17] 10Tool-odivoir, 10GAWA: Concevoir la page d'accueil de la plateforme Open Data Ivoir - https://phabricator.wikimedia.org/T430474#12079689 (10Aklapper) [09:34:21] 10Tool-odivoir, 10GAWA: Créer la page Méta-Wiki de la plateforme Open Data Ivoir - https://phabricator.wikimedia.org/T430766#12079688 (10Aklapper) [09:34:25] 10Tool-odivoir, 10GAWA: Conception des interfaces pour afficher les statistiques par thématique - https://phabricator.wikimedia.org/T430522#12079693 (10Aklapper) [09:34:40] 10Tool-odivoir, 10GAWA: Conception des blocs iframes pour l'intégration des graphiques et cartes - https://phabricator.wikimedia.org/T430521#12079694 (10Aklapper) [09:34:44] 10Tool-odivoir, 10GAWA: Intégration des données Wikidata sur l'orpaillage clandestin en Côte d'Ivoire - https://phabricator.wikimedia.org/T430519#12079695 (10Aklapper) [09:34:48] 10Tool-odivoir, 10GAWA: Créer la page A propos de la plateforme Open Data Ivoir - https://phabricator.wikimedia.org/T430475#12079696 (10Aklapper) [09:39:21] (03CR) 10Majavah: [C:03+2] update blubber buildkit image to v1.8.0 [labs/striker] - 10https://gerrit.wikimedia.org/r/1296518 (https://phabricator.wikimedia.org/T321316) (owner: 10Jelto) [09:42:22] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Data-Platform-SRE, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079752 (10Marostegui) db2160 needs just a downtime. dbproxy will fail during the maintenance (just irc notification) and they'll recove... [09:42:23] (03CR) 10Majavah: [C:03+2] "T340080 means that I can't test this on a more modern Phorge. However, I did check it with the current version and it works." [labs/striker] - 10https://gerrit.wikimedia.org/r/1305875 (https://phabricator.wikimedia.org/T430120) (owner: 10Aklapper) [09:42:45] (03Merged) 10jenkins-bot: update blubber buildkit image to v1.8.0 [labs/striker] - 10https://gerrit.wikimedia.org/r/1296518 (https://phabricator.wikimedia.org/T321316) (owner: 10Jelto) [09:42:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:45:29] (03Merged) 10jenkins-bot: Remove bogus attachment parameter 'uris' from Phabricator API calls [labs/striker] - 10https://gerrit.wikimedia.org/r/1305875 (https://phabricator.wikimedia.org/T430120) (owner: 10Aklapper) [09:45:40] 06cloud-services-team, 10Striker, 06tools-platform-team, 10Phabricator: Striker dev environment needs a new Phabricator base image - https://phabricator.wikimedia.org/T340080#12079776 (10dcaro) [09:45:57] 06cloud-services-team, 10Striker, 06tools-platform-team: Rebuild Striker demo server - https://phabricator.wikimedia.org/T329687#12079778 (10dcaro) [09:47:54] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079785 (10ayounsi) [09:48:03] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079787 (10Marostegui) [09:48:49] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079788 (10ayounsi) [09:49:04] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079789 (10ayounsi) [09:50:07] 06cloud-services-team, 10Striker, 06tools-platform-team, 06Release-Engineering-Team (Doing 😎): [striker] Failing to create new phabricator projects - https://phabricator.wikimedia.org/T430120#12079791 (10Aklapper) a:03Aklapper [09:57:48] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:58:02] (03PS4) 10Majavah: Set all IO to none and hidden for internal URIs of Diffusion repos [labs/striker] - 10https://gerrit.wikimedia.org/r/1202259 (https://phabricator.wikimedia.org/T409341) (owner: 10Aklapper) [09:58:35] (03CR) 10Majavah: [C:03+2] Set all IO to none and hidden for internal URIs of Diffusion repos [labs/striker] - 10https://gerrit.wikimedia.org/r/1202259 (https://phabricator.wikimedia.org/T409341) (owner: 10Aklapper) [09:59:54] 06cloud-services-team, 10Striker, 06tools-platform-team, 06Release-Engineering-Team (Doing 😎): [striker] Failing to create new phabricator projects - https://phabricator.wikimedia.org/T430120#12079857 (10taavi) [09:59:55] 06cloud-services-team, 10Striker, 06tools-platform-team: [striker] Unable to create phabricator project for tool odivoir – HTTP 500 Internal Server Error on toolsadmin - https://phabricator.wikimedia.org/T429857#12079855 (10taavi) →14Duplicate dup:03T430120 [10:01:59] (03Merged) 10jenkins-bot: Set all IO to none and hidden for internal URIs of Diffusion repos [labs/striker] - 10https://gerrit.wikimedia.org/r/1202259 (https://phabricator.wikimedia.org/T409341) (owner: 10Aklapper) [10:04:13] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Data-Platform-SRE, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079891 (10cmooney) a:03cmooney [10:04:30] (03update) 10dcaro: [values.yaml] add support for webservice-defaults [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/23 (https://phabricator.wikimedia.org/T348755) (owner: 10raymond-ndibe) [10:04:34] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079893 (10cmooney) a:03cmooney [10:05:07] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/18 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:05:11] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/18 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:06:12] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Data-Platform-SRE, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12079906 (10cmooney) [10:09:10] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079912 (10jcrespo) [10:10:32] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance toolsbeta-test-k8s-worker-nfs-8 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [10:10:47] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079920 (10jcrespo) backup2012: it is repo backups, should be ok for a small downtime, at most 1 backup run will fail, it will b... [10:12:42] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12079927 (10cmooney) [10:13:57] 10Data-Services, 06tools-platform-team, 06Data-Persistence: clouddb1017 getting stuck during shutdown - https://phabricator.wikimedia.org/T427060#12079931 (10ops-monitoring-bot) Rebooting clouddb1017.eqiad.wmnet [10:18:02] RESOLVED: InstanceDown: Project toolsbeta instance toolsbeta-test-k8s-worker-nfs-8 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:19:01] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Data-Services, 06Data-Platform-SRE, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12079944 (10fgiunchedi) Ok so I spent some time getting a better understanding of NFS and how... [10:20:04] 10Data-Services, 06tools-platform-team, 06Data-Persistence: clouddb1017 getting stuck during shutdown - https://phabricator.wikimedia.org/T427060#12079946 (10ops-monitoring-bot) Reboot of clouddb1017.eqiad.wmnet completed [10:22:02] 10Data-Services, 06tools-platform-team, 06Data-Persistence: clouddb1017 getting stuck during shutdown - https://phabricator.wikimedia.org/T427060#12079956 (10fnegri) 05Open→03Resolved Tested a reboot and this time it worked fine, the step `Stopping mariadb@s1.service` completed in a few seconds. [10:27:29] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 06RoadToWiki, 07patch-welcome: Toolforge 404 handler displays wrong tool name from ?url= query parameter - https://phabricator.wikimedia.org/T421542#12079988 (10Shadabgdg) [10:28:52] 06cloud-services-team, 10Striker, 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎): Striker: Always set all IO to none and hidden for internal URIs when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341#12079994 (10Aklapper) This got deployed today; I should veri... [10:30:32] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance toolsbeta-test-k8s-worker-nfs-8 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [10:35:17] (03open) 10jelto: explicitly export PYTHONPATH in install script [repos/cloud/cloud-vps/horizon/deploy] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/merge_requests/10 [10:37:15] (03update) 10jelto: explicitly export PYTHONPATH in install script [repos/cloud/cloud-vps/horizon/deploy] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/merge_requests/10 [10:40:26] (03update) 10jelto: explicitly export PYTHONPATH in install script [repos/cloud/cloud-vps/horizon/deploy] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/merge_requests/10 [11:01:35] (03open) 10samwilson: Move targetLang fetching to within the change handler [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/12 (https://phabricator.wikimedia.org/T430917) [11:07:48] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [11:11:53] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06tools-infrastructure-team: clean up nova IDs for newer cloudvirts - https://phabricator.wikimedia.org/T430933 (10fgiunchedi) 03NEW [11:27:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cloudcumin1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [11:29:36] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12080121 (10TLepage-WMF) [11:55:15] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/84 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:55:22] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/84 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:55:24] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/160 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:55:35] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/160 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:01:21] 06cloud-services-team, 10Cloud-VPS: Openstack services should use standard HTTPS port - https://phabricator.wikimedia.org/T377055#12080224 (10taavi) a:03taavi [12:03:03] (03update) 10dcaro: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 [12:08:26] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1307121 (owner: 10L10n-bot) [12:10:57] 06cloud-services-team, 10Data-Services, 06collaboration-services, 10Continuous-Integration-Infrastructure (Zuul upgrade): Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12080271 (10hashar) The ansible playbook can now reach out the WMCS OpenStack API. There is an... [12:15:51] (03open) 10countcount: Add a collector binary that harvests all pre-SUL local renames into ToolsDB [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/33 [12:20:56] FIRING: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [12:27:05] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Data-Platform-SRE, and 7 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12080487 (10cmooney) [12:32:57] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/74 [12:35:14] 06cloud-services-team, 10Striker, 06tools-platform-team, 06Release-Engineering-Team (Doing 😎): [striker] Failing to create new phabricator projects - https://phabricator.wikimedia.org/T430120#12080517 (10Aklapper) 05Open→03Resolved Closing as @StrikerBot succeeded in creating https://phabricator.wi... [12:45:16] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12080571 (10TLepage-WMF) [12:46:12] (03merge) 10countcount: Add a collector binary that harvests all pre-SUL local renames into ToolsDB [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/33 [13:15:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/28 [13:25:27] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Data-Services, 06Data-Platform-SRE, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12080727 (10ayounsi) >>! In T411248#12076020, @fgiunchedi wrote: >>>! In T411248#12074721, @ay... [13:27:13] (03update) 10dcaro: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 [13:49:02] 06cloud-services-team, 10Horizon, 10Openstack-Magnum, 06tools-infrastructure-team, 10Continuous-Integration-Infrastructure (Zuul upgrade): Can't retrieve Magnum cluster certificate and config via Horizon - https://phabricator.wikimedia.org/T430680#12080892 (10hashar) **SUCCESS*** thank you @Andrew... [13:49:10] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12080897 (10CWilliams-WMF) [13:51:01] 06cloud-services-team, 10Data-Services, 06collaboration-services, 10Continuous-Integration-Infrastructure (Zuul upgrade): Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12080912 (10hashar) The Cloud Config file is stored in https://gerrit.wikimedia.org/g/integrat... [13:58:10] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12080976 (10dcaro) [13:59:18] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12080982 (10TLepage-WMF) [14:00:55] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Data-Platform-SRE, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12080994 (10CWilliams-WMF) [14:11:33] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 89.74%) at host paws-nfs-2 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [14:15:56] FIRING: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:21:56] (03open) 10countcount: Parse localized comment-era local renames and add a backfill tool [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/34 [14:22:54] 06cloud-services-team, 10Data-Services, 06collaboration-services, 10Continuous-Integration-Infrastructure (Zuul upgrade): Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12081131 (10hashar) For comparison with an application credential for the `zuul` project I hav... [14:35:12] (03merge) 10countcount: Parse localized comment-era local renames and add a backfill tool [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/34 [14:35:41] 10Tool-sentinel: Submit a ticket to add project to translatewiki.net - https://phabricator.wikimedia.org/T430971 (10Arian_Ar) 03NEW [14:36:28] 10Tool-sentinel: Submit a ticket to add project to translatewiki.net - https://phabricator.wikimedia.org/T430971#12081205 (10Arian_Ar) p:05Triage→03Low [14:47:25] (03CR) 10Krinkle: "This also submits `repos` which does not translate across backends per-se. But, we do validate it so at least it gets ignored the same way" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306100 (https://phabricator.wikimedia.org/T376732) (owner: 10Simon04) [14:48:11] 06cloud-services-team, 06tools-infrastructure-team, 06Data-Persistence, 06Infrastructure-Foundations, and 4 others: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928#12081261 (10Gehel) [14:48:44] 06cloud-services-team (FY2025/2026-Q3-Q4), 06Infrastructure-Foundations, 10netops, 06Traffic: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12081268 (10cmooney) I merged the patch to add the lvs public IPs to the healtcheck list now, an... [14:57:19] 10Cloud-Services, 06collaboration-services, 06Data-Persistence, 06Infrastructure-Foundations, and 6 others: codfw: rack B5 maintenance - https://phabricator.wikimedia.org/T430918#12081351 (10Gehel) [14:58:16] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12081358 (10TLepage-WMF) [14:59:45] (03CR) 10Krinkle: Add datalist suggestions to the file path filters (033 comments) [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306099 (https://phabricator.wikimedia.org/T197802) (owner: 10Simon04) [15:02:19] 06cloud-services-team, 10Toolforge, 10Wikidata: Deploy a SPARQL instance for Projekt_Pressearchiv - https://phabricator.wikimedia.org/T430976 (10bking) 03NEW [15:04:17] 06cloud-services-team, 10Toolforge, 10Wikidata: Deploy a SPARQL instance for Projekt_Pressearchiv - https://phabricator.wikimedia.org/T430976#12081407 (10bking) [15:08:28] 06cloud-services-team, 10Toolforge, 10Wikidata: Deploy a SPARQL instance for Projekt_Pressearchiv - https://phabricator.wikimedia.org/T430976#12081444 (10bking) [15:16:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/28 [15:18:44] 06cloud-services-team, 10Cloud-VPS: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12081513 (10taavi) [15:19:34] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance paws-nfs-1 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:20:02] 06cloud-services-team, 10Horizon: Rebuild Horizon with new blubber version - https://phabricator.wikimedia.org/T430164#12081534 (10taavi) [15:22:32] (03open) 10countcount: Serve detail=localrenames from the ToolsDB local_renames table [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/35 [15:22:37] 10Cloud-VPS, 06tools-infrastructure-team, 10CAS-SSO, 07Security: Allowed callback URL for keystone CAS service is too wide - https://phabricator.wikimedia.org/T430957#12081558 (10taavi) 05Open→03Resolved a:03taavi [15:47:58] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12081709 (10aputhin) 05Open→03In progress [15:53:41] (03update) 10countcount: Serve detail=localrenames from the ToolsDB local_renames table [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/35 [15:54:04] 06tools-platform-team: [components-cli] Create aliases for the subcommands - https://phabricator.wikimedia.org/T430988 (10dcaro) 03NEW [15:54:06] (03update) 10countcount: Serve detail=localrenames from the ToolsDB local_renames table [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/35 [15:54:26] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [components-cli] Create aliases for the subcommands - https://phabricator.wikimedia.org/T430988#12081762 (10dcaro) [15:57:52] (03open) 10dcaro: rotate_clinic_duty: added some helper docstrings [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/4 [15:58:03] (03update) 10dcaro: rotate_clinic_duty: added some helper docstrings [repos/cloud/wmcs/utils] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/4 [15:59:34] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance paws-nfs-1 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:03:46] (03merge) 10countcount: Serve detail=localrenames from the ToolsDB local_renames table [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/35 [16:12:04] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12081873 (10ops-monitoring-bot) Rebooting db-test2001.codfw.wmnet [16:15:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/28 [16:20:20] 10Tool-wd-act: Setup toolinfo in Toolforge project - https://phabricator.wikimedia.org/T430994#12081928 (10Tekneos) [16:22:31] 10Toolforge, 06tools-platform-team: [general] 1st of the month updates - https://phabricator.wikimedia.org/T430995 (10dcaro) 03NEW [16:22:53] 10Toolforge, 06tools-platform-team: [code,ci,general] 1st of the month updates - https://phabricator.wikimedia.org/T430995#12081945 (10dcaro) [16:23:40] 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: Upgrade cert-manager past 1.15 - https://phabricator.wikimedia.org/T408786#12081946 (10dcaro) [16:23:41] 10Tool-wd-act: Work through all best practices suggestions in Wikitech's guide - https://phabricator.wikimedia.org/T430996 (10Tekneos) 03NEW [16:23:46] 06cloud-services-team, 10Toolforge, 06tools-infrastructure-team: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.33 - https://phabricator.wikimedia.org/T408785#12081957 (10dcaro) a:05dcaro→03None [16:25:00] 10Tool-wd-act: Review and possibly update the current OAuth flow for this application - https://phabricator.wikimedia.org/T430997 (10Tekneos) 03NEW [16:25:30] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12081972 (10ops-monitoring-bot) Rebooting db-test2001.codfw.wmnet [16:29:09] 06cloud-services-team, 10Cloud-VPS: Radosgw vs openstack application credentials - https://phabricator.wikimedia.org/T430999 (10Andrew) 03NEW [16:29:17] 06tools-platform-team, 13Patch-For-Review: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12082017 (10JJMC89) [16:32:15] 06cloud-services-team, 10Toolforge, 10Wikidata: Deploy a SPARQL instance for Projekt_Pressearchiv - https://phabricator.wikimedia.org/T430976#12082027 (10Jneubert) A debian VM, where the qlever .deb package can be installed, sounds very good! In my test installation, qlever-0.5.48 package installed docker se... [16:32:23] 06tools-platform-team, 10SRE-Access-Requests, 13Patch-For-Review: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12082028 (10Dzahn) [16:36:01] 06tools-platform-team, 10SRE-Access-Requests, 13Patch-For-Review: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12082044 (10dcaro) [16:37:13] 06tools-platform-team, 10SRE-Access-Requests, 13Patch-For-Review: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12082063 (10TLepage-WMF) [16:40:04] 06tools-platform-team, 10SRE-Access-Requests, 13Patch-For-Review: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12082080 (10dcaro) [16:40:39] 06tools-platform-team, 13Patch-For-Review: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12082091 (10dcaro) [16:45:11] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' (T427914) [16:45:16] T427914: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914 [16:45:17] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' (T427914) [16:45:24] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' (T427914) [16:45:30] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' (T427914) [16:45:46] !log dcaro@cloudcumin1001 tools-logging START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' (T427914) [16:45:52] !log dcaro@cloudcumin1001 tools-logging END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' (T427914) [16:45:58] !log dcaro@cloudcumin1001 toolsbeta-logging START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' (T427914) [16:46:03] !log dcaro@cloudcumin1001 toolsbeta-logging END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' (T427914) [16:46:22] !log dcaro@cloudcumin1001 paws START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' (T427914) [16:46:27] !log dcaro@cloudcumin1001 paws END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' (T427914) [16:46:31] !log dcaro@cloudcumin1001 quarry START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' (T427914) [16:46:36] !log dcaro@cloudcumin1001 quarry END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' (T427914) [16:46:57] 06cloud-services-team, 10Data-Services, 06collaboration-services, 10Continuous-Integration-Infrastructure (Zuul upgrade): Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12082129 (10hashar) [16:47:35] 06tools-platform-team, 13Patch-For-Review: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12082131 (10dcaro) [16:52:33] 06tools-platform-team, 13Patch-For-Review: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12082165 (10dcaro) [16:56:48] 06tools-platform-team, 13Patch-For-Review: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12082188 (10TLepage-WMF) [17:08:29] 06tools-platform-team, 13Patch-For-Review: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12082220 (10TLepage-WMF) [17:19:54] (03open) 10arpithhhaaa: Improve UI layout and subtitle editor [toolforge-repos/subtitler] - 10https://gitlab.wikimedia.org/toolforge-repos/subtitler/-/merge_requests/8 [17:20:07] (03close) 10arpithhhaaa: modified UI [toolforge-repos/subtitler] - 10https://gitlab.wikimedia.org/toolforge-repos/subtitler/-/merge_requests/7 [17:20:59] (03close) 10arpithhhaaa: Commons ui integration [toolforge-repos/subtitler] - 10https://gitlab.wikimedia.org/toolforge-repos/subtitler/-/merge_requests/6 [17:21:07] (03close) 10arpithhhaaa: Bugs Fixed [toolforge-repos/subtitler] - 10https://gitlab.wikimedia.org/toolforge-repos/subtitler/-/merge_requests/5 [17:35:55] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12082361 (10TLepage-WMF) [17:41:51] 10Tool-tome: Optimise Tome for images and documents - https://phabricator.wikimedia.org/T431013 (10psubhashish1) 03NEW [17:46:33] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12082420 (10TLepage-WMF) [17:47:24] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12082423 (10TLepage-WMF) [17:48:44] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12082427 (10TLepage-WMF) [17:49:28] (03update) 10dcaro: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 [17:51:21] (03update) 10dcaro: stop passing tool name [repos/cloud/toolforge/jobs-api] (use_job_kind_for_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/337 [17:51:31] (03update) 10dcaro: stop passing tool name [repos/cloud/toolforge/jobs-api] (use_job_kind_for_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/337 [17:55:31] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12082441 (10TLepage-WMF) [17:58:06] 06cloud-services-team, 10Striker, 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Doing 😎): Striker: Always set all IO to none and hidden for internal URIs when creating Diffusion repositories - https://phabricator.wikimedia.org/T409341#12082465 (10Aklapper) Meh, not fixed. Per https://phabricato... [18:16:11] FIRING: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [18:23:07] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 06Tech-Docs-Team, 07OKR-Work: OAD linter gives warnings for missing response body examples in RESTBase compatibility properties - https://phabricator.wikimedia.org/T431018 (10TBurmeister) 03NEW [18:23:54] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 06Tech-Docs-Team, 07OKR-Work: OAD linter gives warnings for missing response examples in RESTBase compatibility properties - https://phabricator.wikimedia.org/T431018#12082592 (10TBurmeister) [18:28:32] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Convert dynamicproxy from nginx to haproxy - https://phabricator.wikimedia.org/T429930#12082595 (10taavi) a:03taavi [19:07:33] 10Tool-wd-act: Initial Release - https://phabricator.wikimedia.org/T431024 (10Tekneos) 03NEW [19:08:41] 10Tool-wd-act, 07good first task: Finish fully setting up README - https://phabricator.wikimedia.org/T431025 (10Tekneos) 03NEW Thank you for tagging this task with #good_first_task for Wikimedia newcomers! Newcomers often may not be aware of things that may seem obvious to seasoned contributors, so please t... [19:10:00] 10Tool-wd-act, 07Security: Replace development OAuth token config with production config - https://phabricator.wikimedia.org/T431026 (10Tekneos) 03NEW [19:12:16] 10Tool-wd-act: Determine how to maintain all required open source compliance - https://phabricator.wikimedia.org/T431027 (10Tekneos) 03NEW [19:13:36] 06cloud-services-team, 10PAWS, 06tools-platform-team, 13Patch-For-Review: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12082821 (10Andrew) Here is a snippet of the kind of thing the attached patch would do. Might be too aggressive... ` INFO:__main__:/srv/paws/project/paws/us... [19:32:19] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/74 (owner: 10l10n-bot) [19:32:22] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/74 (owner: 10l10n-bot) [19:38:09] 10Tool-wd-act: Fix slow performance when viewing WikiData item details - https://phabricator.wikimedia.org/T431032 (10Tekneos) 03NEW [19:38:48] 06cloud-services-team, 10PAWS, 06tools-platform-team, 13Patch-For-Review: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12082915 (10BLiviero-WMF) how does it decide what to remove? (size, age, last use, patterns?) [19:40:19] 06cloud-services-team, 10PAWS, 06tools-platform-team, 13Patch-For-Review: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12082920 (10BLiviero-WMF) I'm asking also because I am not seeing documentation on wikitech about what we clean up on a regular basis [19:47:30] 10Tool-wd-act: Figure out hosting strategy, likely with ToolForge - https://phabricator.wikimedia.org/T431033 (10Tekneos) 03NEW [19:49:04] 10Tool-wd-act: Migrate code hosting, Issues, and Wiki to WMF-supported platforms - https://phabricator.wikimedia.org/T431035 (10Tekneos) 03NEW [19:50:46] 10Tool-wd-act: Do a security review of the application. In particular, look for text/code injection issues - https://phabricator.wikimedia.org/T431036 (10Tekneos) 03NEW [19:52:15] 10Tool-wd-act: Do a deep dive on Wikimedia OAuth tokens, and figure out exactly how to configure mine - https://phabricator.wikimedia.org/T431037 (10Tekneos) 03NEW [19:53:33] 10Tool-wd-act: Likely Dockerize the application - https://phabricator.wikimedia.org/T431038 (10Tekneos) 03NEW [19:53:42] 10Tools: Toolforge CDNjs mirror can no longer obtain all versions of libraries - https://phabricator.wikimedia.org/T431039 (10LucasWerkmeister) 03NEW [19:54:24] 10Tool-wd-act: Replace development web server with production-ready server - https://phabricator.wikimedia.org/T431040 (10Tekneos) 03NEW [19:55:11] 10Tools: Toolforge CDNjs mirror can no longer obtain all versions of libraries - https://phabricator.wikimedia.org/T431039#12083061 (10LucasWerkmeister) I’ll be honest, I’m very much inclined to agree with Past Lucas right now: >>! In T342519#9088404, @LucasWerkmeister wrote: > And I think at some point it will... [19:55:45] 10Tool-wd-act: Display a graphical percentage of metadata completeness for individual items - https://phabricator.wikimedia.org/T431041 (10Tekneos) 03NEW [19:56:48] 10Tool-wd-act: Figure out how/if to integrate auth into standard python/flask/library auth facilities - https://phabricator.wikimedia.org/T431042 (10Tekneos) 03NEW [19:57:51] 10Tool-wd-act: If I'm able to read data from the WikiData REST API on ToolForge with no OAuth config - start a discussion on their docs to improve the docs and mention this possibility - https://phabricator.wikimedia.org/T431044 (10Tekneos) 03NEW [20:02:50] 10Tool-wd-act: Migrate code hosting, Issues, and Wiki to WMF-supported platforms - https://phabricator.wikimedia.org/T431035#12083117 (10Tekneos) Issues have all been moved over. Still need to move Wiki and code hosting. [20:08:59] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12083139 (10ops-monitoring-bot) Rebooting db-test2001.codfw.wmnet [20:09:05] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28618176241 (https://github.com/cluebotng/component-configs/commits/c3b2ebd135508a9a0f781ed332514069fb4d8c89) [20:09:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [20:15:56] RESOLVED: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [20:15:57] RESOLVED: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [20:17:51] 10Tool-wd-act: Migrate code hosting, Issues, and Wiki to WMF-supported platforms - https://phabricator.wikimedia.org/T431035#12083174 (10Tekneos) For the Wiki migration, I should work it into a larger developer documentation section of the tools docs. See here for their guidelines on Tool docs: https://www.media... [20:24:14] 06cloud-services-team, 10Data-Services, 06tools-platform-team, 06Data-Persistence, 13Patch-For-Review: Extend sre.mysql.upgrade to work with multiinstance hosts - https://phabricator.wikimedia.org/T420203#12083198 (10CWilliams-WMF) Just adding this here for reference... TL;DR the new cookbook seems to b... [20:44:21] 10Tool-wd-act: Migrate code hosting, Issues, and Wiki to WMF-supported platforms - https://phabricator.wikimedia.org/T431035#12083273 (10Tekneos) Git repo is pushed up to GitLab now: https://gitlab.wikimedia.org/toolforge-repos/wd-act [21:12:32] 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431055 (10LibUp-bot) 03NEW [21:19:08] 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T431056 (10LibUp-bot) 03NEW [21:22:58] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) [21:23:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [21:23:02] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) [21:23:02] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412778 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) [21:23:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [21:23:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [21:25:16] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412758 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) [21:25:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [21:25:17] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412759 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) [21:25:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [21:25:33] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412753 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) [21:25:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [21:25:50] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412786 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) [21:25:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [21:26:13] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412770 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) [21:26:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [21:27:54] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) [21:27:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [21:29:27] !log tools.cluebot-syncer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412754 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) [21:29:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [21:32:30] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412856 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) [21:32:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [21:34:07] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28622412778 (https://github.com/cluebotng/component-configs/commits/748a4aefb7b83d51fb8db338ff67a1d1c7678b51) [21:34:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [21:35:52] 10Tool-wd-act: Migrate code hosting, Issues, and Wiki to WMF-supported platforms - https://phabricator.wikimedia.org/T431035#12083462 (10Tekneos) Wiki is now migrated over to the Tool's wiki page here: https://wikitech.wikimedia.org/wiki/Tool:WikiDataAuthorityControlTool. There was only 1 page to migrate over fo... [21:36:09] 10Tool-wd-act: Migrate code hosting, Issues, and Wiki to WMF-supported platforms - https://phabricator.wikimedia.org/T431035#12083463 (10Tekneos) 05Open→03Resolved [21:37:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [21:38:07] 10Tool-wd-act: Clean up and fully flesh out Tool's documentation page - https://phabricator.wikimedia.org/T431058 (10Tekneos) 03NEW [21:42:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [21:45:14] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 05MW-1.47-notes (1.47.0-wmf.10; 2026-07-07), 06MW-Interfaces-Team (MWI-Sprint-36 (2026-06-16 to 2026-06-30)): Generate a valid semver info.version for the default MediaWiki REST module - https://phabricator.wikimedia.org/T427359#12083503 (10MGoncalves-WMF... [21:49:24] 10Tool-wd-act: Setup toolinfo in Toolforge project - https://phabricator.wikimedia.org/T430994#12083520 (10Tekneos) 05Open→03Resolved [21:49:32] 10Tool-wd-act: Setup toolinfo in Toolforge project - https://phabricator.wikimedia.org/T430994#12083521 (10Tekneos) Knocked this out a minute ago. [22:05:39] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work: Fix top-level issues in the MediaWiki REST API OAD - https://phabricator.wikimedia.org/T428147#12083614 (10MGoncalves-WMF) a:03MGoncalves-WMF [22:13:05] 06cloud-services-team, 10Horizon: Horizon IDP/SSO logout doesn't really logout - https://phabricator.wikimedia.org/T374123#12083632 (10bd808) >>! In T374123#12027984, @Volans wrote: > And from a quick test logging out of IDP doesn't actually log you out, repro steps: > * Open incognito window > * Go to horizon... [22:19:24] 10Cloud-VPS (Project-requests), 10Maps: Check current usage of project maps on CloudVPS - https://phabricator.wikimedia.org/T429446#12083656 (10bd808) [22:24:21] 10Cloud-VPS (Debian Bullseye Deprecation), 10Wikimedia-Developer-Portal: Update Dev Portal staging to Debian Bookworm - https://phabricator.wikimedia.org/T429490#12083663 (10bd808) [22:32:11] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28625556671 (https://github.com/cluebotng/component-configs/commits/e95434d48b96490f3526640acde23ce59116589b) [22:32:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [22:45:45] (03open) 10don-vip: Draft: Decouple the update pipeline into four phases... [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/45 [22:46:05] (03update) 10don-vip: Draft: Decouple the update pipeline into four phases using prioritized Redis queues [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/45 [22:46:16] 06cloud-services-team, 10PAWS, 06tools-platform-team, 13Patch-For-Review: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12083706 (10Andrew) >>! In T429578#12082915, @BLiviero-WMF wrote: > how does it decide what to remove? (size, age, last use, patterns?) The initial version o... [22:46:36] (03PS6) 10Alien4444: PagesCreated: add an incoming link count column [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276066 (https://phabricator.wikimedia.org/T276520) [22:49:45] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12083722 (10bd808) [22:49:47] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12083723 (10bd808) [22:49:49] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation), 06Data-Engineering, 10Event-Platform: Upgrade (replace) beta|deployment-prep event platform VPS instances - https://phabricator.wikimedia.org/T429497#12083721 (10bd808) [22:50:13] (03PS7) 10Alien4444: PagesCreated: add an incoming link count column [labs/xtools] - 10https://gerrit.wikimedia.org/r/1276066 (https://phabricator.wikimedia.org/T276520) [22:50:19] 06cloud-services-team, 10Cloud-VPS (Debian Bullseye Deprecation): Cloud VPS Debian Bullseye deprecation - https://phabricator.wikimedia.org/T401804#12083725 (10bd808) [22:50:20] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 07Epic, 06Release-Engineering-Team (Priority Backlog 📥): Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#12083724 (10bd808) [23:27:03] (03PS1) 10MusikAnimal: AppExtension: cache git hash shown in footer of every page [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307265 (https://phabricator.wikimedia.org/T430888) [23:32:10] (03open) 10samwilson: Fix various minor eslint warnings [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/13 [23:47:30] 10Striker: User should be told name of existing Developer account when SUL is already in use - https://phabricator.wikimedia.org/T294767#12083796 (10bd808) [23:47:31] 06cloud-services-team, 10Striker: Cannot join Toolforge: Striker throws IntegrityError while saving user - https://phabricator.wikimedia.org/T429759#12083795 (10bd808) [23:54:31] (03PS1) 10MusikAnimal: RateLimitSubscriber: improve performance and use client IP [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) [23:56:00] (03PS2) 10MusikAnimal: RateLimitSubscriber: improve performance and use client IP [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888)