[01:50:08] (03PS2) 10Wandji collins: Bug:Remove wrong import call [labs/tools/wdaudiolex-fe] - 10https://gerrit.wikimedia.org/r/1286279 [02:16:05] (03CR) 10Wandji collins: "recheck" [labs/tools/wdaudiolex-fe] - 10https://gerrit.wikimedia.org/r/1286279 (owner: 10Wandji collins) [03:02:52] (03PS1) 10MusikAnimal: Authorship: Add support for eleven more languages [labs/xtools] - 10https://gerrit.wikimedia.org/r/1287050 (https://phabricator.wikimedia.org/T372340) [03:04:27] (03CR) 10MusikAnimal: [C:03+2] Authorship: Add support for eleven more languages [labs/xtools] - 10https://gerrit.wikimedia.org/r/1287050 (https://phabricator.wikimedia.org/T372340) (owner: 10MusikAnimal) [03:05:06] (03Merged) 10jenkins-bot: Authorship: Add support for eleven more languages [labs/xtools] - 10https://gerrit.wikimedia.org/r/1287050 (https://phabricator.wikimedia.org/T372340) (owner: 10MusikAnimal) [03:13:28] (03PS1) 10MusikAnimal: 3.24.7 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1287051 [03:17:15] (03CR) 10MusikAnimal: [C:03+2] 3.24.7 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1287051 (owner: 10MusikAnimal) [03:17:53] (03Merged) 10jenkins-bot: 3.24.7 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1287051 (owner: 10MusikAnimal) [03:57:00] (03merge) 10wikigit: Migrate to ruleset repo package [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/22 [06:50:40] !log komla@cloudcumin1001 language START - Cookbook wmcs.openstack.quota_increase by 176 gigabytes (T425905) [06:50:44] T425905: Quota increase request for project language - https://phabricator.wikimedia.org/T425905 [06:50:48] !log komla@cloudcumin1001 language END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 176 gigabytes (T425905) [06:51:58] 06cloud-services-team, 10Toolforge, 06tools-platform-team: Audit tools memory requests vs actual usage - https://phabricator.wikimedia.org/T420565#11920604 (10fgiunchedi) The first reduction is default memory requests has been deployed, as expected we're now under the alerting threshold for memory requests (... [06:56:27] 10Cloud-VPS (Quota-requests): Quota increase request for project language - https://phabricator.wikimedia.org/T425905#11920606 (10komla) Done [06:56:43] 10Cloud-VPS (Quota-requests): Quota increase request for project language - https://phabricator.wikimedia.org/T425905#11920607 (10komla) a:03komla [08:10:19] 06cloud-services-team, 10Toolforge, 06SRE Observability: Support deploying paging alerts to non-production environments (was: make toolsbeta paging alerts less confusing) - https://phabricator.wikimedia.org/T424814#11920654 (10tappof) The proposed solutions seem fine to the o11y team. If you want to go ahead... [08:38:16] 10Tool-humaniki-2: Use dumps - https://phabricator.wikimedia.org/T423012#11920686 (10Danya) 05Open→03In progress [08:39:05] 10Tool-humaniki-2: Migrate site to Cloud VPS - https://phabricator.wikimedia.org/T425489#11920694 (10Danya) [08:39:06] 10Tool-humaniki-2: Investigate best solution between ToolsDB and self-hosted database - https://phabricator.wikimedia.org/T425991#11920693 (10Danya) [08:39:12] 10Tool-humaniki-2: Investigate best solution between ToolsDB and self-hosted database - https://phabricator.wikimedia.org/T425991#11920695 (10Danya) [08:39:13] 10Tool-humaniki-2: Migrate site to Cloud VPS - https://phabricator.wikimedia.org/T425489#11920696 (10Danya) [08:39:38] 10Tool-humaniki-2: Use dumps - https://phabricator.wikimedia.org/T423012#11920701 (10Danya) [08:39:40] 10Tool-humaniki-2: Investigate best solution between ToolsDB and self-hosted database - https://phabricator.wikimedia.org/T425991#11920700 (10Danya) [08:42:14] 10Tool-humaniki-2: Migrate site to Cloud VPS - https://phabricator.wikimedia.org/T425489#11920706 (10Danya) [08:42:15] 10Tool-humaniki-2: Link to the Cloud Services Terms of Use - https://phabricator.wikimedia.org/T426264#11920705 (10Danya) [08:44:04] 10Tool-humaniki-2: Aggregate gender data in the database instead of client - https://phabricator.wikimedia.org/T425219#11920711 (10Danya) [08:45:01] 10Tool-humaniki-2: Migrate site to Cloud VPS - https://phabricator.wikimedia.org/T425489#11920713 (10Danya) [08:45:12] 10Tool-humaniki-2: Investigate best solution between ToolsDB and self-hosted database - https://phabricator.wikimedia.org/T425991#11920715 (10Danya) [08:45:16] 10Tool-humaniki-2: Use dumps - https://phabricator.wikimedia.org/T423012#11920716 (10Danya) [08:45:32] 10Tool-humaniki-2, 13Patch-For-Review: Integrate Denelezh and merge stats tables - https://phabricator.wikimedia.org/T422343#11920717 (10Danya) [08:45:39] 10Tool-humaniki-2: Admin page - https://phabricator.wikimedia.org/T425293#11920718 (10Danya) [08:45:42] 10Tool-humaniki-2: Scheduler does not take full snapshots - https://phabricator.wikimedia.org/T423025#11920719 (10Danya) [08:45:55] 10Tool-humaniki-2: Add page to query data - https://phabricator.wikimedia.org/T425023#11920720 (10Danya) [08:46:16] 10Tool-humaniki-2: Investigate best solution between ToolsDB and self-hosted database - https://phabricator.wikimedia.org/T425991#11920721 (10Danya) [08:46:41] 10Tool-humaniki-2, 13Patch-For-Review: Integrate Denelezh and merge stats tables - https://phabricator.wikimedia.org/T422343#11920723 (10Danya) [08:46:43] 10Tool-humaniki-2: Admin page - https://phabricator.wikimedia.org/T425293#11920724 (10Danya) [08:47:21] 10Tool-humaniki-2: Retro-active statistics - https://phabricator.wikimedia.org/T422342#11920725 (10Danya) [08:47:23] 10Tool-humaniki-2: Setup push to deploy - https://phabricator.wikimedia.org/T422347#11920726 (10Danya) [08:47:34] 10Tool-humaniki-2: Migrate site to Cloud VPS? - https://phabricator.wikimedia.org/T425489#11920727 (10Danya) [10:24:24] 10Toolforge, 06tools-platform-team: toolsbeta tools are not reachable - https://phabricator.wikimedia.org/T426304 (10fnegri) 03NEW [10:24:40] 10Toolforge, 06tools-platform-team: toolsbeta tools are not reachable - https://phabricator.wikimedia.org/T426304#11921063 (10fnegri) 05Open→03In progress p:05Triage→03High a:03fnegri [10:32:22] 10Toolforge, 06tools-platform-team: toolsbeta tools are not reachable - https://phabricator.wikimedia.org/T426304#11921077 (10Raymond_Ndibe) What do you think could be the reason @fnegri ? I noticed this towards the end of my workday yesterday, but thought it was just a proxy server thing. And I also tried dep... [10:38:44] RESOLVED: [3x] IstioGatewayPodMisplaced: istio-gateway pod misplaced - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IstioGatewayPodMisplaced - https://prometheus-alerts.wmcloud.org/?q=alertname%3DIstioGatewayPodMisplaced [10:39:45] 10Toolforge, 06tools-platform-team: toolsbeta tools are not reachable - https://phabricator.wikimedia.org/T426304#11921101 (10fnegri) Redeploying istio-system seems to have fixed it, there was an unexpected Helm difference: `lang=shell-session fnegri@toolsbeta-test-k8s-control-10:~/toolforge-deploy$ sudo -i /... [10:40:14] RESOLVED: [6x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-gateway-1.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDow [10:40:34] RESOLVED: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [10:40:37] RESOLVED: [8x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:49:37] !log fnegri@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway (T425479) [10:49:42] T425479: [lima-kilo] istio not starting correctly - https://phabricator.wikimedia.org/T425479 [10:51:43] (03update) 10fnegri: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) (owner: 10raymond-ndibe) [10:52:13] 10Tool-humaniki-2: Link to the Cloud Services End User Terms of Use - https://phabricator.wikimedia.org/T426264#11921129 (10Danya) [11:04:07] !log fnegri@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway (T425479) [11:04:13] T425479: [lima-kilo] istio not starting correctly - https://phabricator.wikimedia.org/T425479 [11:06:35] 10Toolforge, 06tools-platform-team: toolsbeta tools are not reachable - https://phabricator.wikimedia.org/T426304#11921209 (10fnegri) The Helm diff above is because this branch was deployed in toolsbeta a few days ago: https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235/di... [11:08:44] 10Toolforge, 06tools-platform-team: toolsbeta tools are not reachable - https://phabricator.wikimedia.org/T426304#11921221 (10taavi) Removing that diff did not cause things to start working again. This alert recovering thanks to the re-deployment did: `lang=irc <+wmcs-alerts> RESOLVED: [3x] IstioGatewayPodMisp... [11:12:24] 10Toolforge, 06tools-platform-team: toolsbeta tools are not reachable - https://phabricator.wikimedia.org/T426304#11921228 (10fnegri) > This alert recovering thanks to the re-deployment did So we just needed a redeploy of istio-system, with or without that diff? And why did a redeployment of istio-system fix... [11:30:03] 10Toolforge, 06tools-platform-team: toolsbeta tools are not reachable - https://phabricator.wikimedia.org/T426304#11921294 (10taavi) The re-deployment caused the pods to be re-created and they got scheduled on the correct nodes. [11:51:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:51:53] (03update) 10raymond-ndibe: use Istio port instead of ingress-nginx [repos/cloud/toolforge/toolforge-deploy] (add_tmp_mount) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1246 (https://phabricator.wikimedia.org/T425479) [11:56:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [12:01:33] 10Tool-humaniki-2: Migrate site to Cloud VPS? - https://phabricator.wikimedia.org/T425489#11921339 (10Danya) [12:01:34] 10Tool-humaniki-2: Link to the Cloud Services End User Terms of Use - https://phabricator.wikimedia.org/T426264#11921338 (10Danya) [12:07:01] 10Cloud-VPS (Quota-requests): Quota increase request for project language - https://phabricator.wikimedia.org/T425905#11921353 (10santhosh) 05Open→03Resolved Thanks! [12:12:29] 10Tool-humaniki-2: Add a page to describe statistical methodology - https://phabricator.wikimedia.org/T426314 (10Danya) 03NEW [12:14:21] 10Tool-humaniki-2: Add a page to describe statistical methodology - https://phabricator.wikimedia.org/T426314#11921376 (10Danya) [12:14:23] 10Tool-humaniki-2: Add a page to describe statistical methodology - https://phabricator.wikimedia.org/T426314#11921378 (10Danya) p:05Triage→03Medium [12:20:51] 10Toolforge, 06tools-platform-team: toolsbeta tools are not reachable - https://phabricator.wikimedia.org/T426304#11921394 (10Raymond_Ndibe) @fnegri I'm trying to test the theory locally. Will avoid merging and deploying until then. [12:21:16] (03update) 10raymond-ndibe: istio-system: Disable X-Envoy-Peer-Metadata headers [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235 (https://phabricator.wikimedia.org/T392356) (owner: 10taavi) [12:24:38] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/35 [12:24:39] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/70 [12:24:39] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/40 [12:26:16] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1287399 (owner: 10L10n-bot) [12:26:19] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1287404 (owner: 10L10n-bot) [12:26:20] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1287405 (owner: 10L10n-bot) [12:35:36] 06cloud-services-team, 10Toolforge, 06SRE Observability: Support deploying paging alerts to non-production environments (was: make toolsbeta paging alerts less confusing) - https://phabricator.wikimedia.org/T424814#11921418 (10fgiunchedi) Thank you for following up @tappof ! I'll give the implementation a go... [12:48:38] 10Tool-centralnotice-banner-editor: Add support for codex design color tokens - https://phabricator.wikimedia.org/T426315 (10Oyelola_Victoria) 03NEW [12:49:33] 10Tool-humaniki-2: Link to the Cloud Services End User Terms of Use - https://phabricator.wikimedia.org/T426264#11921444 (10Danya) 05Open→03Declined Not needed. [12:50:28] (03open) 10vriaa: feat: add Codex color token support to all color inputs [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/64 (https://phabricator.wikimedia.org/T420941 https://phabricator.wikimedia.org/T426315) [12:54:47] (03merge) 10vriaa: feat: add support for templates with animated image galleries [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/62 (https://phabricator.wikimedia.org/T420918) [13:03:20] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Publish the Wikimedia Spectral ruleset to NPM - https://phabricator.wikimedia.org/T425627#11921506 (10HCoplin-WMF) p:05Triage→03Medium [13:05:34] 10Toolforge, 06tools-platform-team: toolsbeta tools are not reachable - https://phabricator.wikimedia.org/T426304#11921517 (10fnegri) @taavi after reading https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IstioGatewayPodMisplaced, my current understanding is: * whenever we deploy the "istio-g... [13:10:43] (03update) 10vriaa: feat: commons image lookup [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/60 (https://phabricator.wikimedia.org/T424078) [13:10:50] (03merge) 10vriaa: feat: commons image lookup [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/60 (https://phabricator.wikimedia.org/T424078) [13:19:11] 10Toolforge, 06tools-platform-team: toolsbeta tools are not reachable - https://phabricator.wikimedia.org/T426304#11921559 (10taavi) Re-deploying istio-system usually does not help, since that is a no-op. Today it did, because it made changes to the pod definitions, which triggered the deployment to get re-cre... [13:22:35] (03update) 10vriaa: feat: add support for localisable messages [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/63 (https://phabricator.wikimedia.org/T423927) [13:22:38] 10Toolforge, 06tools-platform-team: toolsbeta tools are not reachable - https://phabricator.wikimedia.org/T426304#11921575 (10Raymond_Ndibe) >>! In T426304#11921394, @Raymond_Ndibe wrote: > @fnegri I'm trying to test the theory locally. Will avoid merging and deploying until then. What I was testing was unrel... [13:27:34] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [13:30:15] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway [13:31:26] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [13:49:33] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway [13:51:21] (03merge) 10raymond-ndibe: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) [13:51:22] (03update) 10raymond-ndibe: use Istio port instead of ingress-nginx [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1246 (https://phabricator.wikimedia.org/T425479) [13:51:38] (03approved) 10raymond-ndibe: use Istio port instead of ingress-nginx [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1246 (https://phabricator.wikimedia.org/T425479) [13:53:14] (03merge) 10raymond-ndibe: use Istio port instead of ingress-nginx [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1246 (https://phabricator.wikimedia.org/T425479) [13:56:49] 06cloud-services-team, 10Toolforge: [istio-gateway] Deploying the component can cause an outage - https://phabricator.wikimedia.org/T426321 (10fnegri) 03NEW [13:57:52] 10Toolforge, 06tools-platform-team: toolsbeta tools are not reachable - https://phabricator.wikimedia.org/T426304#11921758 (10fnegri) [13:57:54] 06cloud-services-team, 10Toolforge: [istio-gateway] Deploying the component can cause an outage - https://phabricator.wikimedia.org/T426321#11921757 (10fnegri) [13:58:57] 10Toolforge, 06tools-platform-team: toolsbeta tools are not reachable - https://phabricator.wikimedia.org/T426304#11921760 (10fnegri) 05In progress→03Resolved > Re-deploying istio-system usually does not help, since that is a no-op. Today it did, because it made changes to the pod definitions, which tr... [14:16:36] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [lima-kilo] istio not starting correctly - https://phabricator.wikimedia.org/T425479#11921830 (10Raymond_Ndibe) 05Open→03Resolved [14:21:42] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/35 (owner: 10l10n-bot) [14:21:45] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/35 (owner: 10l10n-bot) [14:22:32] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/70 (owner: 10l10n-bot) [14:22:39] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/70 (owner: 10l10n-bot) [14:31:56] (03update) 10raymond-ndibe: rename harbor_ip to harbor_hostname [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) [14:31:56] (03update) 10raymond-ndibe: rename harbor_ip to harbor_hostname [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) [14:32:01] (03update) 10raymond-ndibe: rename harbor_ip to harbor_hostname [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) [14:34:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:35:14] (03update) 10fnegri: rename harbor_ip to harbor_hostname [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [14:35:15] (03approved) 10fnegri: rename harbor_ip to harbor_hostname [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [14:35:17] (03update) 10fnegri: rename harbor_ip to harbor_hostname [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [14:36:22] (03update) 10raymond-ndibe: allow harbor.local host [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/39 (https://phabricator.wikimedia.org/T426016) [14:38:19] (03update) 10raymond-ndibe: allow harbor.local host [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/39 (https://phabricator.wikimedia.org/T426016) [14:38:44] (03update) 10raymond-ndibe: deployment: use dummy value for allowedRegistries [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/39 [14:39:34] (03update) 10raymond-ndibe: deployment: use dummy value for allowedRegistries [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/39 [14:43:05] (03update) 10fnegri: deployment: use dummy value for allowedRegistries [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/39 (owner: 10raymond-ndibe) [14:43:21] (03update) 10fnegri: deployment: use dummy value for allowedRegistries [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/39 (owner: 10raymond-ndibe) [14:43:22] (03approved) 10fnegri: deployment: use dummy value for allowedRegistries [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/39 (owner: 10raymond-ndibe) [14:43:54] (03update) 10raymond-ndibe: rename harbor_ip to harbor_hostname [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) [14:43:55] (03approved) 10raymond-ndibe: rename harbor_ip to harbor_hostname [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) [14:44:01] (03merge) 10raymond-ndibe: rename harbor_ip to harbor_hostname [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) [14:44:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:45:36] (03update) 10raymond-ndibe: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) [14:45:55] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-builder: bump to 0.0.150-20260514144411-83a0f413 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1255 (https://phabricator.wikimedia.org/T424362 https://phabricator.wikimedia.org/T426016) [14:45:57] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-builder: bump to 0.0.150-20260514144411-83a0f413 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1255 (https://phabricator.wikimedia.org/T424362 https://phabricator.wikimedia.org/T426016) [14:45:57] (03update) 10raymond-ndibe: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) [14:46:02] (03update) 10raymond-ndibe: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) [14:46:29] (03update) 10fnegri: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [14:46:31] (03approved) 10fnegri: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [14:47:37] (03merge) 10raymond-ndibe: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/159 (https://phabricator.wikimedia.org/T426016) [14:48:10] (03approved) 10raymond-ndibe: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/298 (https://phabricator.wikimedia.org/T426016) [14:48:20] (03merge) 10raymond-ndibe: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/298 (https://phabricator.wikimedia.org/T426016) [14:48:35] (03update) 10raymond-ndibe: deployment: use dummy value for allowedRegistries [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/39 [14:48:36] (03update) 10raymond-ndibe: deployment: use dummy value for allowedRegistries [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/39 [14:48:37] (03approved) 10raymond-ndibe: deployment: use dummy value for allowedRegistries [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/39 [14:48:43] (03merge) 10raymond-ndibe: deployment: use dummy value for allowedRegistries [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/39 [14:49:19] (03approved) 10raymond-ndibe: components: use harbor.local hostname for harbor in local env [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1254 (https://phabricator.wikimedia.org/T426016) [14:49:24] (03update) 10raymond-ndibe: components: use harbor.local hostname for harbor in local env [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1254 (https://phabricator.wikimedia.org/T426016) [14:49:43] (03update) 10raymond-ndibe: components: use harbor.local hostname for harbor in local env [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1254 (https://phabricator.wikimedia.org/T426016) [14:49:57] (03update) 10raymond-ndibe: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) [14:50:14] (03merge) 10raymond-ndibe: components: use harbor.local hostname for harbor in local env [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1254 (https://phabricator.wikimedia.org/T426016) [14:50:15] (03update) 10raymond-ndibe: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417 https://phabricator.wikimedia.org/T424362) (owner: 10fnegri) [14:50:21] (03update) 10raymond-ndibe: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) [14:50:23] (03approved) 10raymond-ndibe: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) [14:50:32] (03merge) 10raymond-ndibe: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) [14:51:31] (03update) 10raymond-ndibe: builds-builder: bump to 0.0.150-20260514144411-83a0f413 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1255 (https://phabricator.wikimedia.org/T424362 https://phabricator.wikimedia.org/T426016) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:52:06] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.493-20260514144834-29d5671c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1256 (https://phabricator.wikimedia.org/T426016) [14:52:11] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.493-20260514144834-29d5671c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1256 (https://phabricator.wikimedia.org/T426016) [14:52:32] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [14:53:33] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: registry-admission: bump to 0.0.75-20260514144856-ee2268f5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1257 [14:56:41] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [14:57:03] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [14:58:11] 10Tool-wikimedia-attribution, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 13Patch-For-Review: Attribution API: Include wprov parameters in response URLs - https://phabricator.wikimedia.org/T425576#11922017 (10pmiazga) @Sarai-WMF @HCoplin-WMF I added also `wprov` param info to schema documentation: {F81432... [15:02:16] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [15:02:58] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:03:17] (03update) 10raymond-ndibe: builds-builder: bump to 0.0.150-20260514144411-83a0f413 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1255 (https://phabricator.wikimedia.org/T424362 https://phabricator.wikimedia.org/T426016) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:03:19] (03approved) 10raymond-ndibe: builds-builder: bump to 0.0.150-20260514144411-83a0f413 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1255 (https://phabricator.wikimedia.org/T424362 https://phabricator.wikimedia.org/T426016) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:03:26] (03merge) 10raymond-ndibe: builds-builder: bump to 0.0.150-20260514144411-83a0f413 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1255 (https://phabricator.wikimedia.org/T424362 https://phabricator.wikimedia.org/T426016) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:03:48] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.493-20260514144834-29d5671c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1256 (https://phabricator.wikimedia.org/T426016) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:14:12] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:16:07] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:16:19] 10Toolforge, 06tools-infrastructure-team: Stand up an etcd cluster for testing requestctl in toolsbeta - https://phabricator.wikimedia.org/T410721#11922148 (10taavi) a:05taavi→03None [15:18:31] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07OKR-Work: [SPIKE] Identify OAD properties not supported by MediaWiki REST Framework - https://phabricator.wikimedia.org/T425942#11922154 (10HCoplin-WMF) Notes from estimation: - https://docs.google.com/spreadsheets/d/1nxGsWvpu_GXD28F... [15:28:17] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:28:39] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.493-20260514144834-29d5671c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1256 (https://phabricator.wikimedia.org/T426016) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:28:51] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.493-20260514144834-29d5671c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1256 (https://phabricator.wikimedia.org/T426016) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:29:10] (03update) 10raymond-ndibe: registry-admission: bump to 0.0.75-20260514144856-ee2268f5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1257 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:30:05] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [15:31:07] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team: Exclude link objects from wikimedia-paths-parameter-example-exists - https://phabricator.wikimedia.org/T425920#11922222 (10HCoplin-WMF) [15:33:26] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Fix issue line highlighting in the linter - https://phabricator.wikimedia.org/T425935#11922235 (10HCoplin-WMF) [15:37:00] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Publish the Wikimedia Spectral ruleset to NPM - https://phabricator.wikimedia.org/T425627#11922249 (10HCoplin-WMF) Notes from Estimation: - Changelog is done - We probably don't want to automate the pipeline right now; it will be safer to have a m... [15:37:35] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Publish the Wikimedia Spectral ruleset to NPM - https://phabricator.wikimedia.org/T425627#11922254 (10HCoplin-WMF) [15:38:13] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [15:38:47] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Publish the Wikimedia Spectral ruleset to NPM - https://phabricator.wikimedia.org/T425627#11922263 (10Mooeypoo) [15:39:05] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [15:39:37] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Publish the Wikimedia Spectral ruleset to NPM - https://phabricator.wikimedia.org/T425627#11922269 (10HCoplin-WMF) [15:40:26] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-mx04 with a new OS and newer MX service stack - https://phabricator.wikimedia.org/T426326 (10bd808) 03NEW [15:40:56] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-mx04 with a new OS and newer MX service stack - https://phabricator.wikimedia.org/T426326#11922295 (10bd808) p:05Triage→03High [15:41:42] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-mx04 with a newer OS and MX service stack - https://phabricator.wikimedia.org/T426326#11922299 (10bd808) [15:42:33] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-mx04 with a newer OS and MX service stack - https://phabricator.wikimedia.org/T426326#11922305 (10bd808) [15:46:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:48:19] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [15:51:48] FIRING: [2x] PuppetFailure: Puppet has failed on cloudcontrol2005-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:57:45] FIRING: WidespreadPuppetFailure: Puppet has failed on wmcs cluster - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=3&var-cluster=wmcs - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [15:58:32] 06cloud-services-team, 10Toolforge, 10Wikibugs: istio-envoy enforcing strict RFC 7231 compliance for 204 status responses - https://phabricator.wikimedia.org/T425229#11922365 (10bd808) [15:58:52] 06cloud-services-team, 10Toolforge, 10Wikibugs: istio-envoy enforcing strict RFC 7231 compliance for 204 status responses - https://phabricator.wikimedia.org/T425229#11922369 (10bd808) [15:58:54] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 13Patch-For-Review: Replace ingress-nginx before upstream EOL date - https://phabricator.wikimedia.org/T392356#11922368 (10bd808) [16:01:48] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [16:03:28] !log taavi@cloudcumin1001 swift START - Cookbook wmcs.openstack.quota_increase by 6 cores, 12288 ram [16:03:35] !log taavi@cloudcumin1001 swift END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 6 cores, 12288 ram [16:03:59] 10Cloud-VPS (Quota-requests), 10SRE-swift-storage: Quota increase request for project swift - https://phabricator.wikimedia.org/T425975#11922386 (10taavi) 05Open→03Resolved a:03taavi [16:06:40] !log taavi@cloudcumin1001 development-metrics START - Cookbook wmcs.openstack.quota_increase by 2 instances [16:06:48] !log taavi@cloudcumin1001 development-metrics END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 2 instances [16:07:29] 10Cloud-VPS (Quota-requests): Quota increase request for project development-metrics - https://phabricator.wikimedia.org/T426070#11922414 (10taavi) 05Open→03Resolved a:03taavi Bumped by 2 since that doesn't give access to any other resources in practice but prevents us from having to do this again once... [16:11:17] (03open) 10taavi: maintain-kubeusers: Increase memory for wiki-tts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1258 (https://phabricator.wikimedia.org/T425804) [16:11:19] (03update) 10taavi: maintain-kubeusers: Increase memory for wiki-tts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1258 (https://phabricator.wikimedia.org/T425804) [16:11:26] (03approved) 10raymond-ndibe: registry-admission: bump to 0.0.75-20260514144856-ee2268f5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1257 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:11:39] (03merge) 10raymond-ndibe: registry-admission: bump to 0.0.75-20260514144856-ee2268f5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1257 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:13:29] (03update) 10raymond-ndibe: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417 https://phabricator.wikimedia.org/T424362) (owner: 10fnegri) [16:15:21] (03approved) 10volans: maintain-kubeusers: Increase memory for wiki-tts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1258 (https://phabricator.wikimedia.org/T425804) (owner: 10taavi) [16:16:47] (03update) 10taavi: maintain-kubeusers: Increase memory for wiki-tts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1258 (https://phabricator.wikimedia.org/T425804) [16:18:44] (03merge) 10taavi: maintain-kubeusers: Increase memory for wiki-tts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1258 (https://phabricator.wikimedia.org/T425804) [16:18:50] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [16:19:03] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [16:20:38] 10Toolforge (Quota-requests), 06Machine-Learning-Team (Q4 FY2025-26), 13Patch-For-Review: Request increased quota for wiki-tts Toolforge tool - https://phabricator.wikimedia.org/T425804#11922494 (10taavi) 05Open→03Resolved a:03taavi [16:20:53] 10Cloud-VPS (Project-requests), 06Machine-Learning-Team (Q4 FY2025-26): Request creation of wikitts VPS project - https://phabricator.wikimedia.org/T425909#11922514 (10taavi) 05Open→03Declined Per {T425804} happening instead. [16:27:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:27:38] 10Cloud-VPS (Project-requests): Request creation of wiki-polis-backend VPS project - https://phabricator.wikimedia.org/T425892#11922569 (10taavi) Could you please describe what exactly is Polis? Why can this not be run in Toolforge? [16:32:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:36:08] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 3 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11922645 (10Dzahn) I left a a bunch of comments I should have put here on t... [16:47:15] (03open) 10raymond-ndibe: add ip to /etc/hosts in lima-kilo [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/323 (https://phabricator.wikimedia.org/T426016) [16:47:37] (03update) 10fnegri: add ip to /etc/hosts in lima-kilo [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/323 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [16:48:25] (03approved) 10fnegri: add ip to /etc/hosts in lima-kilo [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/323 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [16:48:59] (03approved) 10raymond-ndibe: add ip to /etc/hosts in lima-kilo [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/323 (https://phabricator.wikimedia.org/T426016) [16:49:10] (03merge) 10raymond-ndibe: add ip to /etc/hosts in lima-kilo [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/323 (https://phabricator.wikimedia.org/T426016) [16:51:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:56:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:00:25] 10Cloud-VPS (Project-requests): Request creation of wiki-polis-backend VPS project - https://phabricator.wikimedia.org/T425892#11922786 (10Effeietsanders) Polis is an established opinion collection tool (https://pol.is ) that I have built on top of, and want to see if it would help establish more constructive co... [17:02:17] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25873443162 (https://github.com/cluebotng/component-configs/commits/815c3361fd913b83b92ecd2a0d61ae298b921dda) [17:02:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [17:10:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:37:16] 10Cloud-VPS (Project-requests): Request creation of wiki-polis-backend VPS project - https://phabricator.wikimedia.org/T425892#11922933 (10Aklapper) I'm generally always worried about increasing fragmentation by having more conversation venues in our communities (which is not a new problem though). Which use cas... [17:55:14] 10Cloud-VPS (Project-requests): Request creation of wiki-polis-backend VPS project - https://phabricator.wikimedia.org/T425892#11922964 (10Effeietsanders) I find it hard to discuss these concepts with community members without a working prototype. I would honestly love to discuss your concerns more in a synchron... [18:00:59] 10Cloud-VPS (Project-requests): Request creation of wiki-polis-backend VPS project - https://phabricator.wikimedia.org/T425892#11922970 (10Effeietsanders) Another use I'm trying to gather momentum for, would eventually be to better understand the concerns and possibilities around effective AI use in the Wikimedi... [18:29:35] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25877799217 (https://github.com/cluebotng/component-configs/commits/df1444522160198ee3ab31965936679052be618b) [18:29:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [18:36:31] 10Cloud-VPS (Project-requests): Request creation of wiki-polis-backend VPS project - https://phabricator.wikimedia.org/T425892#11923055 (10Sj) +1 for experimenting with this. The outputs of a completed Polis question can be migrated onto Meta for archiving and defragmentation; it's a different tool that we don'... [18:48:47] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25878752565 (https://github.com/cluebotng/component-configs/commits/21e928fa1870ddaf5fae15afc6f92aa3cb3fb970) [18:48:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [18:49:27] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25878752531 (https://github.com/cluebotng/component-configs/commits/21e928fa1870ddaf5fae15afc6f92aa3cb3fb970) [18:49:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [19:06:45] (03update) 10raymond-ndibe: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417 https://phabricator.wikimedia.org/T424362) (owner: 10fnegri) [19:12:23] 10Tool-humaniki-2: Migrate site to Cloud VPS? - https://phabricator.wikimedia.org/T425489#11923138 (10Danya) [19:14:39] 10Tool-upload-workbench: Include tool URL and version in every edit summary written to Commons (!38) - https://phabricator.wikimedia.org/T425978#11923139 (10Daanvr) Something broke stashDupesById is not defined this was shown on page load. plase test yoursef first locally [19:21:33] (03PS1) 10Wandji collins: Edit Repo Config [labs/tools/WdTmCollab] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1286275 [19:26:16] (03Abandoned) 10Wandji collins: Edit Repo Config [labs/tools/WdTmCollab] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1286275 (owner: 10Wandji collins) [20:11:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:26:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:38:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:43:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:01:39] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885319112 (https://github.com/cluebotng/component-configs/commits/8369d11ffe6281d9c21f2e63e23bd03ac30ca98a) [21:01:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [21:10:35] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25885710134 (https://github.com/cluebotng/component-configs/commits/c2d1834d313a784edca516e9f9d4e29fd90a3e5f) [21:10:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [21:12:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:17:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:46:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:47:01] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887423820 (https://github.com/cluebotng/component-configs/commits/55401ec37eaad5c0bffbfbdac85eda0a67eed5c0) [21:47:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [21:52:03] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25887641793 (https://github.com/cluebotng/component-configs/commits/a533fb138a23d96ee62535463d21714a7e24fe18) [21:52:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [21:56:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:12:13] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547#11923594 (10jrbs) [22:17:47] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547#11923609 (10Novem_Linguae) Someone should probably check the times those accounts became qualified to vote. I... [22:19:51] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547#11923647 (10jrbs) >>! In T423547#11923609, @Novem_Linguae wrote: > Someone should probably check the times th... [22:26:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:34:42] 10VPS-project-voterlists, 10MediaWiki-extensions-SecurePoll, 06Product Safety and Integrity: Switch global election workflow to use voterlists.wmcloud.org? - https://phabricator.wikimedia.org/T423547#11923739 (10jrbs) Unfortunately I can't seem to use the file with `importGlobalVoterList.php` because it's to... [22:36:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:40:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:45:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [22:52:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:02:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:16:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:29:53] 10Tool-humaniki-2: Aggregate gender data in the database instead of client - https://phabricator.wikimedia.org/T425219#11923936 (10Danya) 05Open→03In progress [23:29:58] 10Tool-humaniki-2: Use transactions for snapshots - https://phabricator.wikimedia.org/T424903#11923938 (10Danya) 05Open→03In progress [23:35:40] 10Tool-humaniki-2, 13Patch-For-Review: Integrate Denelezh and merge stats tables - https://phabricator.wikimedia.org/T422343#11923969 (10Danya) 05Stalled→03In progress [23:36:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown