[00:38:18] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11983608 (10Ottomata) Hm, Refine to Hive failing because the data does not match the schema. The dat... [04:56:21] (03CR) 10Lokal Profil: [C:03+2] Correct coverage scope [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1295764 (owner: 10Lokal Profil) [04:59:11] (03Merged) 10jenkins-bot: Correct coverage scope [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1295764 (owner: 10Lokal Profil) [05:37:05] (03PS1) 10Giuseppe Lavagetto: Add new private info stub for hiddenparma [labs/private] - 10https://gerrit.wikimedia.org/r/1297493 (https://phabricator.wikimedia.org/T428119) [06:10:24] (03open) 10marostegui: switchmaster: Make puppet-merge more explicit [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/20 [06:17:13] (03approved) 10marostegui: switchmaster: Make puppet-merge more explicit [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/20 [06:17:16] (03merge) 10marostegui: switchmaster: Make puppet-merge more explicit [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/20 [08:00:07] (03CR) 10Nikerabbit: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1295894 (owner: 10L10n-bot) [08:30:36] 10Toolforge, 06tools-platform-team: ToolsDB disk space usage growing too fast - https://phabricator.wikimedia.org/T427187#11984173 (10magnusmanske) Update: I rewrote parts of wikidata-terminator to be able to skip the giant update-happy table entirely. the table is already deleted, albeit there is a new, small... [09:24:38] 10Cloud-VPS (Project-requests): Request creation of humaniki-2 VPS project - https://phabricator.wikimedia.org/T428090#11984393 (10Danya) I'll answer your remarks in order: # I don't want to run a MediaWiki instance alongside my tool. Would it be ok to use ToolsDB for a webservice with a persistent connection?... [09:32:02] 10Tools, 06Commons, 07Community-Wishlist: Image Editor for Commons - https://phabricator.wikimedia.org/T426106#11984423 (10Aklapper) [09:49:56] (03PS1) 10Tiziano Fogli: slothslos/report2drive: add dummy drive secret [labs/private] - 10https://gerrit.wikimedia.org/r/1297640 (https://phabricator.wikimedia.org/T425795) [09:50:49] (03PS2) 10Tiziano Fogli: slothslos/report2drive: add dummy drive secret [labs/private] - 10https://gerrit.wikimedia.org/r/1297640 (https://phabricator.wikimedia.org/T425795) [09:51:55] (03CR) 10Tiziano Fogli: [C:03+2] slothslos/report2drive: add dummy drive secret [labs/private] - 10https://gerrit.wikimedia.org/r/1297640 (https://phabricator.wikimedia.org/T425795) (owner: 10Tiziano Fogli) [09:52:09] (03CR) 10Tiziano Fogli: [V:03+2 C:03+2] slothslos/report2drive: add dummy drive secret [labs/private] - 10https://gerrit.wikimedia.org/r/1297640 (https://phabricator.wikimedia.org/T425795) (owner: 10Tiziano Fogli) [10:04:34] 06cloud-services-team, 10Toolforge: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139 (10fnegri) 03NEW [10:05:10] (03PS1) 10Tiziano Fogli: slothslos/report2drive: adjust key name [labs/private] - 10https://gerrit.wikimedia.org/r/1297643 (https://phabricator.wikimedia.org/T425795) [10:05:47] (03CR) 10Tiziano Fogli: [C:03+2] slothslos/report2drive: adjust key name [labs/private] - 10https://gerrit.wikimedia.org/r/1297643 (https://phabricator.wikimedia.org/T425795) (owner: 10Tiziano Fogli) [10:05:49] (03CR) 10Tiziano Fogli: [V:03+2 C:03+2] slothslos/report2drive: adjust key name [labs/private] - 10https://gerrit.wikimedia.org/r/1297643 (https://phabricator.wikimedia.org/T425795) (owner: 10Tiziano Fogli) [10:06:55] 10Toolforge, 06tools-platform-team: ToolsDB disk space usage growing too fast - https://phabricator.wikimedia.org/T427187#11984524 (10fnegri) 05In progress→03Resolved @magnusmanske thank you very much! I added some more visualizations to the [ToolsDB debugging dashboard](https://grafana.wmcloud.org/d/... [10:07:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [10:12:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [10:18:53] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11984549 (10BTullis) [10:21:11] 10Toolforge, 06tools-platform-team: ToolsDB disk space usage growing too fast - https://phabricator.wikimedia.org/T427187#11984551 (10fnegri) [11:06:35] 10Tool-inteGraality: Qualifier columns: count entities as covered only if all relevant statements have the qualifier - https://phabricator.wikimedia.org/T428144 (10JeanFred) 03NEW [11:07:04] 10Tool-inteGraality: Qualifier columns: count entities as covered only if all relevant statements have the qualifier - https://phabricator.wikimedia.org/T428144#11984663 (10JeanFred) [11:09:47] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11984668 (10Osuji_pius) **Weekly Internship Report Week 1: May 25 – May 31** **Task progress** - First introductory call with Nokib and Tiven. We spoke about the project and expect... [11:59:09] (03PS1) 10Siebrand: gitignore: Add frontend/dist/ build output [labs/tools/bookreader] - 10https://gerrit.wikimedia.org/r/1297659 [11:59:09] (03PS1) 10Siebrand: Use @wikimedia/language-data for language autonyms [labs/tools/bookreader] - 10https://gerrit.wikimedia.org/r/1297660 [11:59:09] (03PS1) 10Siebrand: messages: Remove obsolete lang-* translation keys [labs/tools/bookreader] - 10https://gerrit.wikimedia.org/r/1297661 [12:01:09] (03PS1) 10Majavah: toolforge: kyverno: Remove default versions [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1297662 [12:01:09] (03PS1) 10Majavah: toolforge: kyberno: Update Kyverno images to match 1.16 chart [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1297663 (https://phabricator.wikimedia.org/T427928) [12:01:49] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry [12:01:49] !log taavi@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.16.4 [12:02:22] !log taavi@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.16.4 [12:02:39] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) [12:03:35] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry [12:03:35] !log taavi@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.16.4 [12:03:41] !log taavi@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.16.4 [12:04:23] !log taavi@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.16.4 [12:04:35] (03CR) 10Siebrand: "I wrote this patch to improve the out of the box i18n and to reduce the translation effort needed for each newly supported language." [labs/tools/bookreader] - 10https://gerrit.wikimedia.org/r/1297660 (owner: 10Siebrand) [12:04:45] !log taavi@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.16.4 [12:05:15] !log taavi@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.16.4 [12:05:37] !log taavi@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.16.4 [12:06:01] !log taavi@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/kubectl:v1.32.7 [12:06:17] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) [12:13:12] (03open) 10taavi: istio-gateway: Do not use strict host affinity on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1280 [12:13:17] (03update) 10taavi: istio-gateway: Do not use strict host affinity on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1280 [12:14:09] (03update) 10taavi: istio-gateway: Do not use strict host affinity on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1280 [12:15:22] (03update) 10taavi: istio-gateway: Do not use strict host affinity on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1280 [12:15:26] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [12:15:32] !log taavi@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component istio-gateway [12:15:36] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [12:15:48] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway [12:20:27] (03update) 10taavi: istio-gateway: Do not use strict host affinity on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1280 [12:23:34] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/42 [12:24:34] (03update) 10taavi: istio-gateway: Do not use strict host affinity on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1280 [12:24:36] (03update) 10taavi: istio-gateway: Disable X-Envoy-Peer-Metadata headers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356) [12:24:56] (03update) 10taavi: istio-gateway: Disable X-Envoy-Peer-Metadata headers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356) [12:27:03] (03update) 10taavi: istio-gateway: Reduce Istio header verbosity [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356) [12:27:11] (03update) 10taavi: istio-gateway: Reduce Istio header verbosity [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356) [12:27:14] (03update) 10taavi: istio-gateway: Do not use strict host affinity on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1280 [12:28:46] 10Toolforge, 06tools-infrastructure-team: Istio X-Envoy-Peer-Metadata causes buffer size issues for uWSGI based tools - https://phabricator.wikimedia.org/T428159 (10taavi) 03NEW [12:29:18] (03update) 10taavi: istio-gateway: Reduce Istio header verbosity [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356 https://phabricator.wikimedia.org/T428159) [12:29:22] (03update) 10taavi: istio-gateway: Reduce Istio header verbosity [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356 https://phabricator.wikimedia.org/T428159) [12:29:46] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1297676 (owner: 10L10n-bot) [12:34:30] (03approved) 10fnegri: istio-gateway: Do not use strict host affinity on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1280 (owner: 10taavi) [12:35:35] (03approved) 10fnegri: istio-gateway: Reduce Istio header verbosity [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356 https://phabricator.wikimedia.org/T428159) (owner: 10taavi) [12:35:37] (03update) 10taavi: istio-gateway: Do not use strict host affinity on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1280 [12:35:37] (03update) 10taavi: istio-gateway: Reduce Istio header verbosity [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356 https://phabricator.wikimedia.org/T428159) [12:35:38] (03update) 10taavi: istio-gateway: Disable X-Envoy-Peer-Metadata header, for real [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1281 (https://phabricator.wikimedia.org/T428159) [12:35:41] (03open) 10taavi: istio-gateway: Disable X-Envoy-Peer-Metadata header, for real [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1281 (https://phabricator.wikimedia.org/T428159) [12:35:44] (03update) 10taavi: istio-gateway: Disable X-Envoy-Peer-Metadata header, for real [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1281 (https://phabricator.wikimedia.org/T428159) [12:35:48] (03update) 10fnegri: istio-gateway: Do not use strict host affinity on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1280 (owner: 10taavi) [12:37:43] (03update) 10taavi: istio-gateway: Disable X-Envoy-Peer-Metadata header, for real [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1281 (https://phabricator.wikimedia.org/T428159) [12:39:15] (03approved) 10fnegri: istio-gateway: Disable X-Envoy-Peer-Metadata header, for real [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1281 (https://phabricator.wikimedia.org/T428159) (owner: 10taavi) [12:39:16] (03update) 10fnegri: istio-gateway: Disable X-Envoy-Peer-Metadata header, for real [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1281 (https://phabricator.wikimedia.org/T428159) (owner: 10taavi) [12:40:19] (03open) 10taavi: kyverno: Bump to v1.16 series [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1282 [12:40:22] (03update) 10taavi: kyverno: Bump to v1.16 series [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1282 [12:40:31] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [12:41:21] (03update) 10taavi: istio-gateway: Disable X-Envoy-Peer-Metadata header, for real [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1281 (https://phabricator.wikimedia.org/T428159) [12:41:25] (03update) 10taavi: istio-gateway: Reduce Istio header verbosity [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356 https://phabricator.wikimedia.org/T428159) [12:55:27] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway [12:56:00] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [12:56:44] (03update) 10taavi: istio-gateway: Do not use strict host affinity on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1280 [12:56:45] (03merge) 10taavi: istio-gateway: Do not use strict host affinity on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1280 [12:56:47] (03update) 10taavi: istio-gateway: Reduce Istio header verbosity [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356 https://phabricator.wikimedia.org/T428159) [12:56:58] (03update) 10taavi: istio-gateway: Disable X-Envoy-Peer-Metadata header, for real [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1281 (https://phabricator.wikimedia.org/T428159) [12:56:59] (03merge) 10taavi: istio-gateway: Reduce Istio header verbosity [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356 https://phabricator.wikimedia.org/T428159) [12:57:07] (03merge) 10taavi: istio-gateway: Disable X-Envoy-Peer-Metadata header, for real [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1281 (https://phabricator.wikimedia.org/T428159) [12:57:37] (03PS2) 10Majavah: toolforge: kyverno: Update Kyverno images to match 1.16 chart [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1297663 (https://phabricator.wikimedia.org/T427928) [12:58:12] (03update) 10taavi: kyverno: Bump to v1.16 series [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1282 (https://phabricator.wikimedia.org/T427928) [12:58:16] (03update) 10taavi: kyverno: Bump to v1.16 series [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1282 (https://phabricator.wikimedia.org/T427928) [13:02:10] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway [13:02:58] 10Toolforge, 06tools-infrastructure-team, 13Patch-For-Review: Istio X-Envoy-Peer-Metadata causes buffer size issues for uWSGI based tools - https://phabricator.wikimedia.org/T428159#11985124 (10taavi) 05Open→03Resolved [13:14:22] 10Cloud-VPS (Project-requests): Request creation of tiff compression VPS project - https://phabricator.wikimedia.org/T428102#11985217 (10Andrew) +1 [13:17:49] 10Cloud-VPS (Project-requests): Request creation of tiff compression VPS project - https://phabricator.wikimedia.org/T428102#11985246 (10taavi) +1 > One floating IP would be amazing too so we can carve out an exception in rate limits. Or just carve out the rate limit for the v6 address? [13:18:46] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component kyverno [13:22:14] FIRING: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [13:22:26] !log andrew@cloudcumin1001 tiff-compression START - Cookbook wmcs.vps.create_project for project tiff-compression in eqiad1 [13:22:26] andrew@cloudcumin1001: Unknown project "tiff-compression" [13:23:20] (03open) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project tiff-compression [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/315 [13:23:20] (03update) 10group_199_bot_f98be072172e323ae6d1441939d3e461: projects: added project tiff-compression [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/315 [13:25:11] (03merge) 10andrew: projects: added project tiff-compression [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/315 (owner: 10group_199_bot_f98be072172e323ae6d1441939d3e461) [13:26:25] !log andrew@cloudcumin1001 tiff-compression END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project tiff-compression in eqiad1 [13:26:27] andrew@cloudcumin1001: Unknown project "tiff-compression" [13:27:14] RESOLVED: [2x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDo [13:30:52] !log andrew@cloudcumin1001 tiff-compression START - Cookbook wmcs.openstack.quota_increase by 16384 ram [13:30:52] andrew@cloudcumin1001: Unknown project "tiff-compression" [13:30:53] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Quota increase request for project tiff-compression - https://phabricator.wikimedia.org/T428169 (10Andrew) 03NEW [13:31:00] !log andrew@cloudcumin1001 tiff-compression END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 16384 ram [13:31:00] andrew@cloudcumin1001: Unknown project "tiff-compression" [13:31:24] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Quota increase request for project tiff-compression - https://phabricator.wikimedia.org/T428169#11985317 (10Andrew) 05Open→03Resolved [13:32:18] (03PS1) 10Tiziano Fogli: slothslos/report2drive: move secrets under grafana role [labs/private] - 10https://gerrit.wikimedia.org/r/1297696 (https://phabricator.wikimedia.org/T425795) [13:32:20] 10Cloud-VPS (Project-requests): Request creation of tiff compression VPS project - https://phabricator.wikimedia.org/T428102#11985321 (10Andrew) 05Open→03Resolved a:03Andrew I created this project and boosted the RAM quota (T428169). Let me know if you can't do the rate limit exception via ipv6 and we... [13:32:40] (03CR) 10Tiziano Fogli: [C:03+2] slothslos/report2drive: move secrets under grafana role [labs/private] - 10https://gerrit.wikimedia.org/r/1297696 (https://phabricator.wikimedia.org/T425795) (owner: 10Tiziano Fogli) [13:32:43] (03CR) 10Tiziano Fogli: [V:03+2 C:03+2] slothslos/report2drive: move secrets under grafana role [labs/private] - 10https://gerrit.wikimedia.org/r/1297696 (https://phabricator.wikimedia.org/T425795) (owner: 10Tiziano Fogli) [13:40:09] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno [13:51:07] 10Toolforge, 06tools-infrastructure-team, 13Patch-For-Review: toolforge VMs should use nftables - https://phabricator.wikimedia.org/T427799#11985442 (10taavi) [14:05:42] 10Cloud-VPS (Project-requests): Request creation of tiff compression VPS project - https://phabricator.wikimedia.org/T428102#11985527 (10Ladsgroup) Sounds good. Thanks! [15:06:00] 06cloud-services-team, 10Data-Services, 06DBA, 06Privacy Engineering, 07SecTeam-Processed: filerevision view should not filter out deleted file revisions - https://phabricator.wikimedia.org/T426804#11985909 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views run by ladsgroup: Started... [15:13:48] 06cloud-services-team, 10Data-Services, 06DBA, 06Privacy Engineering, 07SecTeam-Processed: filerevision view should not filter out deleted file revisions - https://phabricator.wikimedia.org/T426804#11985962 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views started by ladsgroup compl... [15:29:11] 06cloud-services-team, 10Data-Services, 06DBA, 06Privacy Engineering, 07SecTeam-Processed: filerevision view should not filter out deleted file revisions - https://phabricator.wikimedia.org/T426804#11986005 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views run by ladsgroup: Started... [15:40:07] 06cloud-services-team, 10Data-Services, 06DBA, 06Privacy Engineering, 07SecTeam-Processed: filerevision view should not filter out deleted file revisions - https://phabricator.wikimedia.org/T426804#11986017 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views started by ladsgroup compl... [15:46:42] 06cloud-services-team, 10Data-Services, 06DBA, 06Privacy Engineering, 07SecTeam-Processed: filerevision view should not filter out deleted file revisions - https://phabricator.wikimedia.org/T426804#11986032 (10Ladsgroup) 05Open→03Resolved [17:03:54] 06cloud-services-team, 10Toolforge: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#11986301 (10fnegri) There is something that is compounding the effect of queries from s52323 (dimasitkbot), and it's queries from s51138 (heritage): `lang=sql ---TRANSACTION (0... [17:15:25] 06cloud-services-team, 10Toolforge: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#11986338 (10fnegri) > I can see similar queries in the slow query log since last year (2025-11-18) so I'm not sure why the "Transaction History Length" has only started increasi... [18:01:20] 06tools-platform-team: Reach out to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T428196 (10komla) 03NEW [18:01:36] 06tools-platform-team: Reach out to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T428196#11986472 (10komla) a:03komla [18:15:31] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11986522 (10Ottomata) You have data in Hive! `event.webrequest_dumps_dev0`! [18:58:46] (03open) 10countcount: Migrate to sqlx 0.9 and adopt replica-pools 0.1.1 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/15 [19:04:22] (03update) 10countcount: Migrate to sqlx 0.9 and adopt replica-pools 0.1.1 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/15 [19:08:20] (03update) 10countcount: Migrate to sqlx 0.9 and adopt replica-pools 0.1.1 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/15 [19:12:45] (03merge) 10countcount: Migrate to sqlx 0.9 and adopt replica-pools 0.1.1 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/15 [19:14:09] (03close) 10renovatebot: Draft: Update Rust crate sqlx to 0.9.0 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/6 [19:14:10] (03update) 10renovatebot: Draft: Update Rust crate sqlx to 0.9.0 - autoclosed [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/6 [19:14:12] (03update) 10countcount: Draft: Update Rust crate sqlx to 0.9.0 - autoclosed [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/6 (owner: 10renovatebot) [19:42:24] (03open) 10countcount: Draft: ci: add report-only test coverage (shared node-ci templates) [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/24 [19:48:55] (03open) 10countcount: Draft: ci: add report-only test coverage (shared python-ci templates) [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/29 [19:52:50] (03open) 10countcount: Draft: Add report-only test coverage to CI [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/9 [19:53:12] (03open) 10countcount: Draft: Add report-only test coverage to CI [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/16 [19:55:26] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/42 (owner: 10l10n-bot) [19:55:38] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/42 (owner: 10l10n-bot) [20:06:34] (03open) 10countcount: Draft: Add report-only test coverage to CI [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/18 [20:06:57] (03update) 10countcount: Draft: Add report-only test coverage to CI [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/18 [20:06:59] (03open) 10countcount: Draft: ci: add report-only coverage for wasm crate and front-end [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/8 [20:19:28] 06tools-platform-team: Reach out to Cloud VPS project maintainers about Debian Bullseye deprecation - https://phabricator.wikimedia.org/T428196#11986827 (10komla) [20:23:07] (03CR) 10MusikAnimal: Add proofreadpage quality support (034 comments) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1257331 (https://phabricator.wikimedia.org/T173012) (owner: 10Alien4444) [20:23:13] 06cloud-services-team, 10Toolforge: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#11986835 (10fnegri) I have temporarily stopped the dimastbkbot tool. I sent an email to the maintainer and also posted a message to their user page linking to this Phabricator t... [20:27:14] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#11986843 (10fnegri) 05Open→03In progress p:05Triage→03High a:03fnegri [20:39:30] 10Tool-humaniki-2: Migrate site to Cloud VPS? - https://phabricator.wikimedia.org/T425489#11986859 (10Danya) Cloud VPS request can be found here: T428090 [20:46:18] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#11986863 (10fnegri) I also lowered `idle_transaction_timeout` to 60: `lang=mysql SET GLOBAL idle_transaction_timeout=60; ` But that's not enough to av... [20:56:34] (03PS1) 10MusikAnimal: AdminStatsTest: fix issue with static caching [labs/xtools] - 10https://gerrit.wikimedia.org/r/1297778 [20:57:06] (03CR) 10CI reject: [V:04-1] AdminStatsTest: fix issue with static caching [labs/xtools] - 10https://gerrit.wikimedia.org/r/1297778 (owner: 10MusikAnimal) [20:57:57] (03CR) 10MusikAnimal: "Well there you have it. CI refuses to run it in a separate process, so I guess the easiest thing to do is change the test to not mock the " [labs/xtools] - 10https://gerrit.wikimedia.org/r/1297778 (owner: 10MusikAnimal) [21:05:50] (03PS2) 10MusikAnimal: AdminStatsTest: fix issue with static caching [labs/xtools] - 10https://gerrit.wikimedia.org/r/1297778 [21:07:27] (03CR) 10MusikAnimal: [C:03+2] AdminStats: +footnote on (re)protect and (re)block [labs/xtools] - 10https://gerrit.wikimedia.org/r/1275236 (https://phabricator.wikimedia.org/T354024) (owner: 10Alien4444) [21:07:45] (03CR) 10MusikAnimal: "Done" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1297778 (owner: 10MusikAnimal) [21:08:05] (03Merged) 10jenkins-bot: AdminStats: +footnote on (re)protect and (re)block [labs/xtools] - 10https://gerrit.wikimedia.org/r/1275236 (https://phabricator.wikimedia.org/T354024) (owner: 10Alien4444) [21:11:38] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#11986979 (10fnegri) I suspect there is also an underlying issue that I haven't discovered yet, and the stuck queries from `heritage` and `dimastbkbot` m... [21:19:17] FIRING: JobUnavailable: Reduced availability for job blackbox_http in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:24:17] RESOLVED: JobUnavailable: Reduced availability for job blackbox_http in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:52:39] (03open) 10siebrand: Replace manually translated language names with i18n_data (CLDR) [toolforge-repos/luthor] - 10https://gitlab.wikimedia.org/toolforge-repos/luthor/-/merge_requests/4 [22:12:30] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26982622098 (https://github.com/cluebotng/component-configs/commits/eb9cf1341e6e78387424cb9070a3aec87971e54d) [22:12:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [22:16:13] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/26982622098 (https://github.com/cluebotng/component-configs/commits/eb9cf1341e6e78387424cb9070a3aec87971e54d) [22:16:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [22:35:57] (03open) 10komla: add script(backup) to send email via gmail smtp [repos/cloud/wmcs/cloud-survey-ops] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/cloud-survey-ops/-/merge_requests/6 [22:35:57] (03approved) 10komla: add script(backup) to send email via gmail smtp [repos/cloud/wmcs/cloud-survey-ops] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/cloud-survey-ops/-/merge_requests/6 [22:36:02] (03merge) 10komla: add script(backup) to send email via gmail smtp [repos/cloud/wmcs/cloud-survey-ops] - 10https://gitlab.wikimedia.org/repos/cloud/wmcs/cloud-survey-ops/-/merge_requests/6 [22:36:16] 06cloud-services-team, 10Toolforge: components-api failing to connect to internal api - https://phabricator.wikimedia.org/T428214 (10DamianZaremba) 03NEW [22:56:45] 10Cloud-VPS (Project-requests): Request creation of humaniki-2 VPS project - https://phabricator.wikimedia.org/T428090#11987153 (10bd808) >>! In T428090#11984393, @Danya wrote: > I'll answer your remarks in order: > > 1. I don't want to run a MediaWiki instance alongside my tool. Sorry I misunderstood that par... [22:59:27] 10Cloud-VPS (Project-requests): Request creation of humaniki-2 VPS project - https://phabricator.wikimedia.org/T428090#11987154 (10bd808) A different path would be to talk with the folks who run the [[https://openstack-browser.toolforge.org/project/wikidumpparse|wikidumpparse Cloud VPS project]] which is the cur... [23:02:45] 06cloud-services-team, 10Toolforge, 06tools-platform-team: components-api failing to connect to internal api - https://phabricator.wikimedia.org/T428214#11987187 (10BLiviero-WMF) [23:50:13] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11987261 (10Ahoelzl) @KCVelaga_WMF we have data, are you unblocked? [23:52:11] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-04-24 - 2026-05-15), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#11987267 (10Ahoelzl)