[01:45:26] 10Tool-gawa: [Code] Conception de la page de statistiques - https://phabricator.wikimedia.org/T401767#11137638 (10poro26) [01:53:06] 10Tool-gawa: [Code] Conception de la page d'Accueil - https://phabricator.wikimedia.org/T403438 (10poro26) 03NEW [03:09:36] 10Tool-gawa: [Code] Conception de la page d'Accueil - https://phabricator.wikimedia.org/T403438#11137695 (10poro26) 05Open→03Resolved a:03poro26 Lien de la page d'Accueil : https://gawa.toolforge.org/home Lien du code source de la page d'Accueil : https://gitlab.wikimedia.org/toolforge-repos/gawa/-/bl... [05:53:13] 10Tools, 10WMF-General-or-Unknown: "Page Watchers" from article tool links is 404 - https://phabricator.wikimedia.org/T120927#11137760 (10Pppery) 05Open→03Resolved This isn't on the sidebar on enwiki anymore, so the original bug report has been fixed. Problems with individual user scripts aren't tracke... [06:36:46] 10VPS-Projects: Mwoffliner: cloud vps object storage size quota over 90% - https://phabricator.wikimedia.org/T403410#11137794 (10Benoit74) @dcaro can you develop a bit why we suddenly get this message, and where we can find our current object storage usage and total entitled storage? I struggle to find the info... [06:40:14] 10VPS-Projects: Mwoffliner: cloud vps object storage size quota over 90% - https://phabricator.wikimedia.org/T403410#11137796 (10Audiodude) Make sure you have mwoffliner selected as your project: {F65947133} Then click "Object Store" and under that, "Containers". This is what I see: {F65947137} GIven that w... [06:40:59] 10VPS-Projects: Mwoffliner: cloud vps object storage size quota over 90% - https://phabricator.wikimedia.org/T403410#11137798 (10Audiodude) Honestly I feel like we can just ignore this task, or delete these objects, or both. [08:08:24] 06cloud-services-team, 10Cloud-VPS, 06Data-Platform-SRE, 13Patch-For-Review: Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#11137885 (10fnegri) Looping in #data-platform-sre as they have done some work on those hosts and they might have ideas. [08:08:29] 06cloud-services-team, 10Cloud-VPS, 06Data-Platform-SRE, 13Patch-For-Review: Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#11137886 (10fnegri) [08:19:16] 10VPS-Projects: Mwoffliner: cloud vps object storage size quota over 90% - https://phabricator.wikimedia.org/T403410#11137892 (10dcaro) > @dcaro can you develop a bit why we suddenly get this message, and where we can find our current object storage usage and total entitled storage? I struggle to find the info i... [08:42:03] 06cloud-services-team, 10Toolforge: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T403348#11137948 (10fgiunchedi) p:05Triage→03Medium [10:31:11] 10VPS-project-Codesearch: Codesearch down/unreachable (2025-09-02) - https://phabricator.wikimedia.org/T403434#11138318 (10A_smart_kitten) 05Open→03Resolved Seems like this might now be resolved. Unfortunately, this seems to be a recurring issue as of recently… :[ [10:41:19] 10Tool-ldap: Display production SSH keys in use by a developer - https://phabricator.wikimedia.org/T403463 (10SD0001) 03NEW [11:19:27] 10VPS-project-Codesearch: Codesearch down/unreachable (2025-09-02) - https://phabricator.wikimedia.org/T403434#11138449 (10Waddie96) Yeh been working for me all day with great response time. [11:31:27] 10VPS-Projects: Mwoffliner: cloud vps object storage size quota over 90% - https://phabricator.wikimedia.org/T403410#11138478 (10Benoit74) Thank you for the explanation, this is way clearer. I was confused believing we've made something in recent days which caused us to go over-quota. Let's cleanup these contai... [11:32:40] 10VPS-Projects: Mwoffliner: cloud vps object storage size quota over 90% - https://phabricator.wikimedia.org/T403410#11138481 (10Benoit74) 05Open→03Resolved a:03Benoit74 [11:53:22] (03merge) 10samwilson: Update repo URL in toolinfo.json [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/3 (https://phabricator.wikimedia.org/T395398) [11:53:32] (03open) 10samwilson: Use HTTP client object from API, with User-Agent set [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/4 (https://phabricator.wikimedia.org/T403435) [11:55:30] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/207 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:56:03] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/30 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:56:33] (03update) 10samwilson: Use HTTP client object from API, with User-Agent set [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/4 (https://phabricator.wikimedia.org/T403435) [11:58:45] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/207 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:58:46] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/207 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:58:57] (03update) 10dcaro: api: add `include_unset` parameter to get_job and get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/205 (https://phabricator.wikimedia.org/T402569) [11:59:10] (03open) 10arnaudb: test [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/209 [11:59:35] (03update) 10arnaudb: test [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/209 [11:59:58] (03close) 10arnaudb: test [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/209 [12:08:07] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/30 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:08:41] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/207 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:09:47] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/30 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:11:17] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.410-20250902120853-d76fc1b5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/945 [12:11:27] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.410-20250902120853-d76fc1b5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/945 [12:12:25] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.66-20250902120959-63846eb2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/943 [12:12:28] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.66-20250902120959-63846eb2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/943 [12:19:22] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:19:53] (03CR) 10Jforrester: build: Commit missing .phan/config.php file (031 comment) [labs/tools/wikiinfo] - 10https://gerrit.wikimedia.org/r/1173454 (owner: 10Krinkle) [12:20:31] (03update) 10samwilson: Use HTTP client object from API, with User-Agent set [toolforge-repos/wsexport] - 10https://gitlab.wikimedia.org/toolforge-repos/wsexport/-/merge_requests/4 (https://phabricator.wikimedia.org/T403435) [12:25:33] (03update) 10dcaro: openapi: add the internal server and some description [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/76 (https://phabricator.wikimedia.org/T402032) [12:33:50] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [12:35:32] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:38:28] 10VPS-project-Codesearch: Move Codesearch behind OAuth login - https://phabricator.wikimedia.org/T403474 (10Peachey88) 03NEW [12:41:04] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/77 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:41:09] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/77 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:43:18] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.77-20250902124122-8bb72ef6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/941 [12:43:20] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.77-20250902124122-8bb72ef6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/941 [12:43:54] (03update) 10dcaro: openapi: add the internal server and some description [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/76 (https://phabricator.wikimedia.org/T402032) [12:50:33] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [12:53:02] (03approved) 10dcaro: jobs-api: bump to 0.0.410-20250902120853-d76fc1b5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/945 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:53:06] (03merge) 10dcaro: jobs-api: bump to 0.0.410-20250902120853-d76fc1b5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/945 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:53:17] (03update) 10dcaro: jobs-emailer: bump to 0.0.66-20250902120959-63846eb2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/943 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:53:18] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [12:55:26] (03update) 10raymond-ndibe: Draft: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [13:02:45] (03update) 10raymond-ndibe: Draft: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [13:03:07] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [13:07:25] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [13:11:03] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [13:12:22] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [13:12:51] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/51 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:12:54] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/51 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:13:04] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/130 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:31:50] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [13:34:10] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [13:37:25] 10VPS-project-Codesearch: Move Codesearch behind OAuth login - https://phabricator.wikimedia.org/T403474#11139085 (10A_smart_kitten) One potential concern that comes to mind regarding an OAuth gate is whether it would add a noticeable amount of time to (what is currently able to be) a very quick experience when... [13:37:51] 10VPS-project-Codesearch: Codesearch down/unreachable (2025-08-30) - https://phabricator.wikimedia.org/T403323#11139090 (10A_smart_kitten) >>! In T403323#11134042, @A_smart_kitten wrote: >> If someone is willing to build an anti-AI front for it, by all means. > > random idea, maybe Codesearch could use 10Tool-global-search, 10Discovery-Search (2025.08.15 - 2025.09.05): Global Search displays most search results twice - https://phabricator.wikimedia.org/T391175#11139105 (10Gehel) [13:42:30] 10VPS-project-Codesearch: Move Codesearch behind OAuth login - https://phabricator.wikimedia.org/T403474#11139123 (10Ladsgroup) FWIW a good write up why we shouldn't use Anubis: https://lock.cmpxchg8b.com/anubis.html [13:48:57] (03approved) 10dcaro: jobs-emailer: bump to 0.0.66-20250902120959-63846eb2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/943 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:49:00] (03merge) 10dcaro: jobs-emailer: bump to 0.0.66-20250902120959-63846eb2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/943 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:55:16] 06cloud-services-team, 10Cloud-VPS, 10Toolforge, 10Observability-Alerting, and 2 others: Move WMCS off of Icinga and introduce alertmanager - https://phabricator.wikimedia.org/T328502#11139234 (10tappof) [13:55:45] 06cloud-services-team, 10wikitech.wikimedia.org, 10Observability-Alerting: Move wikitech-static monitoring off Icinga - https://phabricator.wikimedia.org/T362397#11139238 (10tappof) [14:04:03] 10VPS-project-Codesearch: Move Codesearch behind OAuth login - https://phabricator.wikimedia.org/T403474#11139308 (10A_smart_kitten) >>! In T403474#11139123, @Ladsgroup wrote: > FWIW a good write up why we shouldn't use Anubis: https://lock.cmpxchg8b.com/anubis.html To be fair, the idea that AI companies would... [14:09:11] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [14:09:50] (03update) 10dcaro: openapi: add the internal server and some description [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/76 (https://phabricator.wikimedia.org/T402032) [14:13:14] (03update) 10fnegri: openapi: add the internal server and some description [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/76 (https://phabricator.wikimedia.org/T402032) (owner: 10dcaro) [14:13:16] (03approved) 10fnegri: openapi: add the internal server and some description [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/76 (https://phabricator.wikimedia.org/T402032) (owner: 10dcaro) [14:14:53] (03update) 10dcaro: openapi: add the internal server and some description [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/76 (https://phabricator.wikimedia.org/T402032) [14:25:25] (03merge) 10dcaro: openapi: add the internal server and some description [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/76 (https://phabricator.wikimedia.org/T402032) [14:27:54] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.78-20250902142533-4768f899 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/941 (https://phabricator.wikimedia.org/T402032) [14:27:55] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.78-20250902142533-4768f899 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/941 (https://phabricator.wikimedia.org/T402032) [14:30:29] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [14:30:48] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/130 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:30:51] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/130 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:33:51] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.155-20250902143103-54060e84 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/942 [14:34:00] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.155-20250902143103-54060e84 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/942 [14:35:39] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [14:53:27] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/90 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:53:31] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/90 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:55:24] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/90 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:12:20] 06cloud-services-team, 10Toolforge, 07Upstream: Build service tools built with --use-latest-versions are missing /etc/mime.types - https://phabricator.wikimedia.org/T403303#11139776 (10DamianZaremba) You can install the `media-types` package using `project.toml` and then point uwsgi/gunicorn at it (it will e... [15:19:09] 06cloud-services-team, 10Toolforge, 07Documentation, 07Kubernetes: Figure out and document how to call the Kubernetes API as your tool user from inside a pod - https://phabricator.wikimedia.org/T321919#11139841 (10dcaro) > If your code thinks it knows what jobs are running without being able to know where... [15:26:49] (03update) 10dcaro: api-gateway: bump to 0.0.78-20250902142533-4768f899 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/941 (https://phabricator.wikimedia.org/T402032) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:28:49] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [15:29:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-5 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [15:32:34] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [15:33:09] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [15:33:44] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/120 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:34:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-5 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [15:36:47] 06cloud-services-team, 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [api-gateway] https://api.svc.toolforge.org endpoint given in OpenAPI spec returns 403 forbidden errors - https://phabricator.wikimedia.org/T402032#11139948 (10dcaro) [15:37:35] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [15:37:52] (03approved) 10dcaro: api-gateway: bump to 0.0.78-20250902142533-4768f899 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/941 (https://phabricator.wikimedia.org/T402032) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:37:57] (03merge) 10dcaro: api-gateway: bump to 0.0.78-20250902142533-4768f899 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/941 (https://phabricator.wikimedia.org/T402032) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:38:04] 06cloud-services-team, 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [api-gateway] https://api.svc.toolforge.org endpoint given in OpenAPI spec returns 403 forbidden errors - https://phabricator.wikimedia.org/T402032#11139952 (10dcaro) 05In progress→03Resolved [15:38:13] (03update) 10dcaro: components-api: bump to 0.0.155-20250902143103-54060e84 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/942 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:38:19] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [15:39:38] (03update) 10bd808: Makefile: Add targets for format, tidy, and test [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/12 [15:40:47] (03update) 10bd808: Makefile: Add targets for format, tidy, and test [repos/cloud/cloud-vps/tofu-cloudvps] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-cloudvps/-/merge_requests/12 [15:43:35] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [15:47:44] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [15:48:11] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/120 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:48:14] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/120 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:48:57] (03update) 10dcaro: calico: bump to 0.0.16-20250901160128-176866d0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/940 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:52:58] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [16:06:36] (03approved) 10dcaro: components-api: bump to 0.0.155-20250902143103-54060e84 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/942 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:06:39] (03merge) 10dcaro: components-api: bump to 0.0.155-20250902143103-54060e84 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/942 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:08:05] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico [16:16:20] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/73 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:21:11] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/73 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:21:13] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/73 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:21:29] (03update) 10dcaro: calico: bump to 0.0.16-20250901160128-176866d0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/940 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:23:45] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-kubeusers: bump to 0.0.180-20250902162127-0f463217 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/951 [16:23:48] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-kubeusers: bump to 0.0.180-20250902162127-0f463217 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/951 [16:26:01] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico [16:26:26] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component calico [16:33:49] 06cloud-services-team, 10Cloud-VPS, 10Toolforge, 10Observability-Alerting, and 2 others: Move WMCS off of Icinga and introduce alertmanager - https://phabricator.wikimedia.org/T328502#11140193 (10tappof) [16:43:30] 10Tool-paulina: Add autocomplete suggestions to search bar using Wikidata wbsearchentities API - https://phabricator.wikimedia.org/T402458#11140262 (10JosefAnthony) >>! In T402458#11121943, @Pepe_piton wrote: > Great progress! I just changed a little thing in the API call so that the autocomplete is displayed in... [16:46:06] 06Toolforge-standards-committee: Define a process for keeping the committee membership "fresh" - https://phabricator.wikimedia.org/T379844#11140269 (10bd808) I have sent a new message to the TFSC mailing list asking folks to review a strawdog proposal that I drafted last fall based on feedback from the committee... [16:46:56] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico [16:47:05] (03open) 10dcaro: pre-commit: fix golanglint-ci [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/140 [16:47:21] (03approved) 10dcaro: pre-commit: fix golanglint-ci [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/140 [16:51:18] (03open) 10dcaro: pre-commit: fix golanglint-ci [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/34 [16:51:20] (03approved) 10dcaro: pre-commit: fix golanglint-ci [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/34 [16:52:46] (03open) 10dcaro: pre-commit: fix golanglint-ci [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/24 [16:53:00] (03approved) 10dcaro: pre-commit: fix golanglint-ci [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/24 [16:53:11] (03merge) 10dcaro: pre-commit: fix golanglint-ci [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/140 [16:54:09] (03open) 10dcaro: pre-commit: fix golanglint-ci [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/28 [16:54:11] (03approved) 10dcaro: pre-commit: fix golanglint-ci [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/28 [16:54:23] (03merge) 10dcaro: pre-commit: fix golanglint-ci [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/34 [16:55:10] (03approved) 10dcaro: calico: bump to 0.0.16-20250901160128-176866d0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/940 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:55:17] (03merge) 10dcaro: calico: bump to 0.0.16-20250901160128-176866d0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/940 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:55:50] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics [16:55:52] (03update) 10dcaro: wmcs-k8s-metrics: bump to 0.0.22-20250901160038-e95a0711 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/937 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:56:06] (03merge) 10dcaro: pre-commit: fix golanglint-ci [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/24 [16:56:42] 10VPS-project-Codesearch: Move Codesearch behind OAuth login - https://phabricator.wikimedia.org/T403474#11140331 (10LucasWerkmeister) [16:58:45] (03merge) 10dcaro: pre-commit: fix golanglint-ci [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/28 [16:59:51] (03open) 10dcaro: pre-commit: add golangci-lint check [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/63 [16:59:52] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.71-20250902165439-8937b04a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/947 [16:59:57] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.71-20250902165439-8937b04a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/947 [17:00:59] (03update) 10dcaro: api: add `include_unset` parameter to get_job and get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/205 (https://phabricator.wikimedia.org/T402569) [17:01:49] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.64-20250902165629-42bf130b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/948 [17:01:51] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.64-20250902165629-42bf130b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/948 [17:02:10] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.198-20250902165321-2b654726 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/950 [17:02:14] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.198-20250902165321-2b654726 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/950 [17:03:03] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: registry-admission: bump to 0.0.65-20250902165859-35e89a44 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/944 [17:03:06] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: registry-admission: bump to 0.0.65-20250902165859-35e89a44 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/944 [17:12:31] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics [17:12:46] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics [17:19:50] (03update) 10dcaro: gitlab: move to bookworm images [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/86 [17:20:09] (03update) 10dcaro: pre-commit: add golangci-lint check [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/63 [17:20:31] (03approved) 10dcaro: wmcs-k8s-metrics: bump to 0.0.22-20250901160038-e95a0711 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/937 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:20:34] (03merge) 10dcaro: wmcs-k8s-metrics: bump to 0.0.22-20250901160038-e95a0711 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/937 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:22:01] 06cloud-services-team, 10Toolforge, 07Upstream: Build service tools built with --use-latest-versions are missing /etc/mime.types - https://phabricator.wikimedia.org/T403303#11140456 (10LucasWerkmeister) Is `project.toml` something all tools can use? I couldn’t find any documentation about it on Wikitech. (Th... [17:24:30] 06cloud-services-team, 10Toolforge, 07Upstream: Build service tools built with --use-latest-versions are missing /etc/mime.types - https://phabricator.wikimedia.org/T403303#11140466 (10bd808) >>! In T403303#11140456, @LucasWerkmeister wrote: > Is `project.toml` something all tools can use? I couldn’t find an... [17:31:25] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics [17:31:50] 10Tools: Migrate Wikidata Shape Expressions Inference from Aptfile to project.toml - https://phabricator.wikimedia.org/T403509 (10LucasWerkmeister) 03NEW [17:32:02] 10Tools: Migrate Wikidata Shape Expressions Inference from Aptfile to project.toml - https://phabricator.wikimedia.org/T403509#11140530 (10LucasWerkmeister) [17:32:07] 06cloud-services-team, 10Toolforge: [builds-builder,apt] migrate from apt buildpack to Heroku's .deb packages buildpack - https://phabricator.wikimedia.org/T387141#11140531 (10LucasWerkmeister) [17:33:10] 06cloud-services-team, 10Toolforge, 07Upstream: Build service tools built with --use-latest-versions are missing /etc/mime.types - https://phabricator.wikimedia.org/T403303#11140540 (10LucasWerkmeister) Thanks, that’s good to know (⇒ T403509). I take it documentation will appear once this version of the buil... [17:49:01] 10wikitech.wikimedia.org, 07Wikimedia-production-error: Wikitech-static search DBQueryError - https://phabricator.wikimedia.org/T389578#11140623 (10bd808) >>! In T389578#11134604, @TheDJ wrote: > Doesn't wikitech use cirrussearch ? Yes, Wikitech does use CirrusSearch. This task looks to be about wikitech-stat... [18:01:38] 06cloud-services-team, 10Toolforge, 07Upstream: Build service tools built with --use-latest-versions are missing /etc/mime.types - https://phabricator.wikimedia.org/T403303#11140662 (10DamianZaremba) >>! In T403303#11140540, @LucasWerkmeister wrote: > Thanks, that’s good to know (⇒ T403509). I take it docume... [18:09:08] 10Tool-global-search, 10Discovery-Search (2025.08.15 - 2025.09.05): Global Search displays most search results twice - https://phabricator.wikimedia.org/T391175#11140676 (10EBernhardson) No review came in, went ahead with merge and deploy. Problem looks resolved. [18:32:58] 10Tool-global-search, 10Discovery-Search (2025.08.15 - 2025.09.05): Global Search displays most search results twice - https://phabricator.wikimedia.org/T391175#11140742 (10MusikAnimal) 05Open→03Resolved a:03MusikAnimal >>! In T391175#11140675, @EBernhardson wrote: > No review came in, went ahead wit... [18:39:52] 10Toolforge (Toolforge iteration 24): [lima-kilo] fix permission of tool's home dir - https://phabricator.wikimedia.org/T403513 (10Raymond_Ndibe) 03NEW [18:55:48] FIRING: PuppetDisabled: Puppet disabled on cloudcontrol2005-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [18:56:01] 06cloud-services-team: PuppetDisabled Puppet disabled on cloudcontrol2005-dev:9100 - https://phabricator.wikimedia.org/T403515 (10phaultfinder) 03NEW [19:05:48] FIRING: [2x] PuppetDisabled: Puppet disabled on cloudcontrol2005-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [19:05:55] 06cloud-services-team: PuppetDisabled - https://phabricator.wikimedia.org/T403517 (10phaultfinder) 03NEW [19:10:26] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519 (10LucasWerkmeister) 03NEW [19:12:08] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11140880 (10LucasWerkmeister) Some QuickCategories logs (apparentl... [19:16:32] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11140890 (10Mahir256) And a traceback from Mishramilan at 18:01 UT... [19:22:59] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11140917 (10LucasWerkmeister) I added some debug code to QuickCate... [19:25:29] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11140919 (10LucasWerkmeister) Sorry, my debug code was accidentall... [19:27:46] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11140924 (10LucasWerkmeister) ` Runner.__post__init query+tokens+u... [19:28:55] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11140935 (10LucasWerkmeister) ` Runner.__post__init query+tokens+u... [19:31:37] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11140947 (10LucasWerkmeister) Okay, yeah, apparently tokens [end w... [19:32:43] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11140951 (10Mahir256) A similar problem (the same one?) was report... [19:34:38] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11140954 (10LucasWerkmeister) Pinging some people from two recent-... [19:44:37] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11141031 (10LucasWerkmeister) It happens even [in the API sandbox]... [19:48:33] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [19:51:32] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [19:54:05] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11141110 (10LucasWerkmeister) >>! In T403519#11141031, @LucasWerkm... [20:10:12] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11141236 (10LucasWerkmeister) Some helpful comments from @Krinkle... [20:15:56] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11141302 (10LucasWerkmeister) FWIW, I just tested QuickCategories... [20:16:19] RESOLVED: [2x] PuppetDisabled: Puppet disabled on cloudcontrol2005-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [20:23:02] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team, 13Patch-For-Review: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11141341 (10matmarex) We've talked on IRC an... [20:29:28] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team, 13Patch-For-Review: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11141423 (10Krinkle) I can't speak to the wi... [20:49:49] FIRING: ObjectStorageSizeQuotaFull: Object storage quota by 'size' is 100% full for project qrank - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageSizeQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageSizeQuotaFull [20:49:49] FIRING: ObjectStorageObjectQuotaFull: Object storage quota by 'objects' is 80.46% full for project spacemedia - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull - https://grafana.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas - https://alerts.wikimedia.org/?q=alertname%3DObjectStorageObjectQuotaFull [20:58:29] 10wikitech.wikimedia.org: Update https://wikitech.wikimedia.org/wiki/MediaWiki:Gadget-HotCat.js to latest upstream - https://phabricator.wikimedia.org/T403553 (10bd808) 03NEW [21:19:53] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team, 13Patch-For-Review: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11141641 (10LucasWerkmeister) With the above... [21:23:47] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team, 13Patch-For-Review: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11141649 (10LucasWerkmeister) >>! In T403519... [21:36:11] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team, 13Patch-For-Review: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11141687 (10matmarex) Note, affected apps ar... [21:39:50] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team, 13Patch-For-Review: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11141691 (10LucasWerkmeister) >>! In T403519... [22:02:46] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team, 13Patch-For-Review: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11141752 (10LucasWerkmeister) >>! In T403519... [22:16:22] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team, 13Patch-For-Review: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11141834 (10LucasWerkmeister) Should be reso... [23:10:37] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [23:19:08] 10Tool-quickcategories, 10MediaWiki-Core-AuthManager, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team: Several mwapi (Python) based tools are failing to edit: badtoken: Invalid CSRF token. - https://phabricator.wikimedia.org/T403519#11141972 (10matmarex) [23:23:28] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [23:32:42] 10wikitech.wikimedia.org: Update https://wikitech.wikimedia.org/wiki/MediaWiki:Gadget-HotCat.js to latest upstream - https://phabricator.wikimedia.org/T403553#11142002 (10bd808) 05Open→03In progress a:03bd808 [23:43:38] 10wikitech.wikimedia.org: Update https://wikitech.wikimedia.org/wiki/MediaWiki:Gadget-HotCat.js to latest upstream - https://phabricator.wikimedia.org/T403553#11142009 (10bd808) 05In progress→03Resolved I basically followed https://commons.wikimedia.org/wiki/Help:Gadget-HotCat#Using_the_Commons_version_o... [23:55:53] (03update) 10raymond-ndibe: [maintain-harbor.jobs] manage policies and robot accounts [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/47 (https://phabricator.wikimedia.org/T360509)