[00:01:38] (03open) 10raymond-ndibe: pyproject.toml: move uvicorn to tool.poetry.dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/170 (https://phabricator.wikimedia.org/T422184) [00:03:15] (03update) 10raymond-ndibe: pyproject.toml: move uvicorn to tool.poetry.dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/170 (https://phabricator.wikimedia.org/T422184) [00:03:17] (03approved) 10raymond-ndibe: pyproject.toml: move uvicorn to tool.poetry.dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/170 (https://phabricator.wikimedia.org/T422184) [00:14:17] (03merge) 10raymond-ndibe: pyproject.toml: move uvicorn to tool.poetry.dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/170 (https://phabricator.wikimedia.org/T422184) [00:17:08] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.198-20260513001428-d6234be8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1252 (https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T424368) [00:17:12] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.198-20260513001428-d6234be8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1252 (https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T424368) [00:19:52] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [00:24:46] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [00:25:02] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [00:30:27] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [00:32:33] (03update) 10raymond-ndibe: components-api: bump to 0.0.198-20260513001428-d6234be8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1252 (https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T424368) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [00:32:34] (03approved) 10raymond-ndibe: components-api: bump to 0.0.198-20260513001428-d6234be8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1252 (https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T424368) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [00:32:39] (03merge) 10raymond-ndibe: components-api: bump to 0.0.198-20260513001428-d6234be8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1252 (https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T424368) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [00:33:30] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [general] upgrade all python repos to python >=3.13 - https://phabricator.wikimedia.org/T422184#11915455 (10Raymond_Ndibe) 05Open→03Resolved [00:53:37] (03update) 10raymond-ndibe: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417 https://phabricator.wikimedia.org/T424362) (owner: 10fnegri) [00:58:57] 06cloud-services-team, 10Cloud-VPS: Bastion immediately dropping SSH connection (LDAP sync issue?) - https://phabricator.wikimedia.org/T425815#11915464 (10Jony) 05Resolved→03Open [01:14:03] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771823348 (https://github.com/cluebotng/component-configs/commits/d3130e631e8e51f86a65880414bcb85c6e35b979) [01:14:04] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771881181 (https://github.com/cluebotng/component-configs/commits/d180b98d98a8b5c89115a2080375145455ab4730) [01:14:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [01:14:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [01:17:24] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25771924892 (https://github.com/cluebotng/component-configs/commits/a8bb5b7a82d6f9fbfe284e624bd8ec03b24b5e0e) [01:17:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [01:27:49] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772343768 (https://github.com/cluebotng/component-configs/commits/6b6570012dd25ddae243b54aa3376d2fb9485ec1) [01:27:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [01:33:44] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25772564857 (https://github.com/cluebotng/component-configs/commits/a1494790dc79d0332e0f7d6bdc8d2cf36635de8a) [01:33:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [01:38:32] FIRING: InstanceDown: Project tools instance tools-puppetserver-01 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:57:47] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773326566 (https://github.com/cluebotng/component-configs/commits/7112f7cc4f34aa90e0c7d7db1ee89c2104da8c46) [01:57:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [02:04:59] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773616046 (https://github.com/cluebotng/component-configs/commits/0fd601991775a24b437113d09438e74b996c991b) [02:05:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [02:06:42] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/25773616037 (https://github.com/cluebotng/component-configs/commits/0fd601991775a24b437113d09438e74b996c991b) [02:06:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [02:22:32] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-k8s-worker-102 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [02:27:32] FIRING: [12x] PuppetAgentNoResources: No Puppet resources found on instance tools-docker-registry-8 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [02:28:32] RESOLVED: InstanceDown: Project tools instance tools-puppetserver-01 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:32:32] FIRING: [31x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [02:37:32] FIRING: [52x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [02:42:32] FIRING: [75x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [02:47:32] FIRING: [80x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [02:52:32] FIRING: [80x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [02:57:32] FIRING: [73x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [03:02:32] RESOLVED: [58x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-4 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [04:11:26] (03open) 10raymond-ndibe: use hostname instead of ip for harbor setup [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) [05:12:05] (03open) 10raymond-ndibe: allow lima-kilo harbor host [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/39 (https://phabricator.wikimedia.org/T426016) [05:17:34] (03open) 10raymond-ndibe: deployment: use hostname for harbor not ip in local env [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/298 (https://phabricator.wikimedia.org/T426016) [05:23:38] (03open) 10raymond-ndibe: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) [05:32:04] (03open) 10raymond-ndibe: deployment: use hostname for harbor not ip [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/159 (https://phabricator.wikimedia.org/T426016) [05:33:29] (03update) 10raymond-ndibe: allow lima-kilo harbor host [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/39 (https://phabricator.wikimedia.org/T426016) [05:33:33] (03update) 10raymond-ndibe: allow harbor.local host [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/39 (https://phabricator.wikimedia.org/T426016) [05:35:49] (03update) 10raymond-ndibe: deployment: use hostname for harbor not ip [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/159 (https://phabricator.wikimedia.org/T426016) [05:36:56] (03update) 10raymond-ndibe: deployment: use hostname for harbor not ip [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/159 (https://phabricator.wikimedia.org/T426016) [05:37:24] (03update) 10raymond-ndibe: deployment: use hostname for harbor not ip [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/159 (https://phabricator.wikimedia.org/T426016) [05:37:40] (03update) 10raymond-ndibe: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/159 (https://phabricator.wikimedia.org/T426016) [05:43:43] (03update) 10raymond-ndibe: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) [05:45:25] (03update) 10raymond-ndibe: deployment: use hostname for harbor not ip in local env [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/298 (https://phabricator.wikimedia.org/T426016) [05:46:34] (03update) 10raymond-ndibe: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/298 (https://phabricator.wikimedia.org/T426016) [05:49:57] (03update) 10raymond-ndibe: use hostname instead of ip for harbor setup [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) [05:50:24] (03update) 10raymond-ndibe: rename harbor_ip to harbor_hostname [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) [05:50:30] (03update) 10raymond-ndibe: rename harbor_ip to harbor_hostname [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) [05:56:18] (03open) 10raymond-ndibe: components: use harbor.local hostname for harbor in local env [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1254 (https://phabricator.wikimedia.org/T426016) [06:05:48] (03update) 10raymond-ndibe: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/159 (https://phabricator.wikimedia.org/T426016) [06:07:32] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07OKR-Work: Validate the MediaWiki REST API OpenAPI description - https://phabricator.wikimedia.org/T422917#11915696 (10KBach) 05Stalled→03In progress [06:07:36] (03update) 10raymond-ndibe: components: use harbor.local hostname for harbor in local env [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1254 (https://phabricator.wikimedia.org/T426016) [06:09:33] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07OKR-Work: [SPIKE] Identify OAD properties not supported by MediaWiki REST Framework - https://phabricator.wikimedia.org/T425942#11915698 (10KBach) [06:10:14] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07OKR-Work: Validate the MediaWiki REST API OpenAPI description - https://phabricator.wikimedia.org/T422917#11915701 (10KBach) 05In progress→03Open [06:13:44] (03update) 10raymond-ndibe: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/159 (https://phabricator.wikimedia.org/T426016) [06:15:11] (03update) 10raymond-ndibe: components: use harbor.local hostname for harbor in local env [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1254 (https://phabricator.wikimedia.org/T426016) [06:15:58] (03update) 10raymond-ndibe: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) [06:16:57] (03update) 10raymond-ndibe: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/159 (https://phabricator.wikimedia.org/T426016) [06:17:21] (03update) 10raymond-ndibe: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/298 (https://phabricator.wikimedia.org/T426016) [06:17:54] (03update) 10raymond-ndibe: rename harbor_ip to harbor_hostname [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) [06:26:27] (03update) 10raymond-ndibe: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) [06:35:52] (03update) 10raymond-ndibe: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) [06:36:13] (03update) 10raymond-ndibe: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) [06:36:13] (03update) 10raymond-ndibe: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) [07:13:04] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-33 (2026-05-05 to 2026-05-19)), 07OKR-Work: [SPIKE] Add security scheme information to MediaWiki REST API description - https://phabricator.wikimedia.org/T423552#11915779 (10KineticPelagic) 05Open→03In progress [07:54:22] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team, 07OKR-Work: Document linter rules - https://phabricator.wikimedia.org/T422930#11915872 (10KBach) p:05Triage→03Medium [07:59:44] (03merge) 10filippo: k8s: adjust default memory requests [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/102 (https://phabricator.wikimedia.org/T420565) [08:13:34] (03open) 10filippo: d/changelog: bump to 0.103.21 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/105 (https://phabricator.wikimedia.org/T420565) [08:18:43] !log filippo@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [08:19:56] !log filippo@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli [08:35:21] !log filippo@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli [08:36:48] !log filippo@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli [08:49:38] 10Tool-wikimedia-attribution: Update data links to the Attribution API Beta - https://phabricator.wikimedia.org/T421385#11916145 (10Sarai-WMF) 05Open→03Resolved [08:49:52] 10Tool-wikimedia-attribution: [WAF·V1] Update site with reviewed content - https://phabricator.wikimedia.org/T416982#11916148 (10Sarai-WMF) 05Open→03Resolved [08:50:31] 10Tool-wikimedia-attribution: [WAF·V1] Add New Attribution Configurator to Search scenario page - https://phabricator.wikimedia.org/T416984#11916150 (10Sarai-WMF) a:03AlexHollender-WMF [08:50:48] 10Tool-wikimedia-attribution: [WAF·V1] Add New Attribution Configurator to Search scenario page - https://phabricator.wikimedia.org/T416984#11916151 (10Sarai-WMF) [08:50:53] 10Tool-wikimedia-attribution: [WAF·V1] Add New Attribution Configurator to Search scenario page - https://phabricator.wikimedia.org/T416984#11916153 (10Sarai-WMF) 05Open→03Resolved [08:51:23] 10Tool-wikimedia-attribution: [WAF·V1] Add New Attribution Configurator to AI assistants page - https://phabricator.wikimedia.org/T416988#11916154 (10Sarai-WMF) a:03AlexHollender-WMF [08:51:33] 10Tool-wikimedia-attribution: [WAF·V1] Add New Attribution Configurator to AI assistants page - https://phabricator.wikimedia.org/T416988#11916156 (10Sarai-WMF) 05Open→03Resolved [09:08:39] 10Tool-curator: Curator: Entering single image keys generated errors - https://phabricator.wikimedia.org/T426158 (10PantheraLeo1359531) 03NEW [09:10:08] 10Cloud-VPS (Quota-requests), 10SRE-swift-storage: Quota increase request for project swift - https://phabricator.wikimedia.org/T425975#11916231 (10fnegri) +1 [09:10:21] 10Cloud-VPS (Quota-requests): Quota increase request for project development-metrics - https://phabricator.wikimedia.org/T426070#11916233 (10fnegri) +1 [09:11:10] 10Tool-wikimedia-attribution: [WE5.3.1c] Publish attribution guidelines for Games and Rich media experiences - https://phabricator.wikimedia.org/T416992#11916236 (10Sarai-WMF) [09:13:11] 10Tool-wikimedia-attribution: [WE5.3.1c] Publish attribution guidelines for the Social media reuse scenario - https://phabricator.wikimedia.org/T416993#11916242 (10Sarai-WMF) [09:13:15] 10Cloud-VPS (Quota-requests): Quota increase request for project language - https://phabricator.wikimedia.org/T425905#11916245 (10fgiunchedi) +1 [09:15:23] 10Tool-wikimedia-attribution: [WE5.3.1c] Publish attribution guidelines for the Media and publications scenario - https://phabricator.wikimedia.org/T426160 (10Sarai-WMF) 03NEW [09:22:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:27:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:26:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:31:07] 10Cloud-VPS (Quota-requests), 10SRE-swift-storage: Quota increase request for project swift - https://phabricator.wikimedia.org/T425975#11916688 (10KOfori) Preemptively adding my approval here as the Data Persistence manager in case that's needed for this quota increase [10:31:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:43:37] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:52:10] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07OKR-Work: [SPIKE] Identify OAD properties not supported by MediaWiki REST Framework - https://phabricator.wikimedia.org/T425942#11916781 (10KBach) [11:03:37] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [11:06:16] 10Cloud-VPS (Project-requests): Request creation of wiki-polis-backend VPS project - https://phabricator.wikimedia.org/T425892#11916835 (10Effeietsanders) [11:44:37] 10Tool-wikimedia-attribution: Create curated project brand mark category - https://phabricator.wikimedia.org/T426184 (10Sarai-WMF) 03NEW [11:45:09] 10Tool-wikimedia-attribution: Create curated brand mark collection in Wikimedia Commons - https://phabricator.wikimedia.org/T426184#11916985 (10Sarai-WMF) [12:55:10] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07OKR-Work: [SPIKE] Identify OAD properties not supported by MediaWiki REST Framework - https://phabricator.wikimedia.org/T425942#11917196 (10KBach) [13:23:51] (03update) 10fnegri: rename harbor_ip to harbor_hostname [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [13:25:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [13:26:55] (03open) 10ahecht: Support "Merging" and "Deleting" in nomination statements, show "(Nom)" text for merge nominations. [toolforge-repos/afdstats] - 10https://gitlab.wikimedia.org/toolforge-repos/afdstats/-/merge_requests/5 [13:27:07] (03approved) 10ahecht: Support "Merging" and "Deleting" in nomination statements, show "(Nom)" text for merge nominations. [toolforge-repos/afdstats] - 10https://gitlab.wikimedia.org/toolforge-repos/afdstats/-/merge_requests/5 [13:27:15] (03merge) 10ahecht: Support "Merging" and "Deleting" in nomination statements, show "(Nom)" text for merge nominations. [toolforge-repos/afdstats] - 10https://gitlab.wikimedia.org/toolforge-repos/afdstats/-/merge_requests/5 [13:35:10] (03update) 10raymond-ndibe: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) [13:36:03] (03update) 10raymond-ndibe: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) [13:36:20] (03update) 10raymond-ndibe: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) [14:04:28] 10Toolforge (Quota-requests), 06Machine-Learning-Team (Q4 FY2025-26): Request increased quota for wiki-tts Toolforge tool - https://phabricator.wikimedia.org/T425804#11917642 (10Andrew) +1 approved [14:10:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [14:12:35] 06cloud-services-team, 10Toolforge: restarted pod failed to schedule due to resource constraints - https://phabricator.wikimedia.org/T425696#11917686 (10aputhin) This might be fixed by T420565, which is being deployed today / next week. @DamianZaremba you might want to try reverting your change above and see i... [14:15:10] 06cloud-services-team, 10Cloud-VPS: Bastion immediately dropping SSH connection (LDAP sync issue?) - https://phabricator.wikimedia.org/T425815#11917693 (10Andrew) ` However, it looks like the setup bug also failed to scaffold the ou=sudoers directory ` Indeed, this was pretty broken. I've added the necessary... [14:15:32] 06cloud-services-team, 10Cloud-VPS: Bastion immediately dropping SSH connection (LDAP sync issue?) - https://phabricator.wikimedia.org/T425815#11917695 (10Andrew) 05Open→03Resolved [14:16:09] 06cloud-services-team, 10Toolforge, 06Release-Engineering-Team: [toolforge_deploy_mr] Branches can only be tested for 24 hours, otherwise fail with CI too old. - https://phabricator.wikimedia.org/T408999#11917699 (10aputhin) p:05Triage→03Low [14:16:45] (03CR) 10Peterxy12: [C:03+1] Localisation updates from https://translatewiki.net. [labs/tools/wikiinfo] - 10https://gerrit.wikimedia.org/r/1285777 (owner: 10L10n-bot) [14:18:55] 10Toolforge, 06tools-platform-team: Toolforge CNF build pack jobs are not terminated gracefully. - https://phabricator.wikimedia.org/T372879#11917722 (10aputhin) p:05Triage→03Medium We need to check if this is still happening - @Count_Count if you have evidence of this still being a problem, please let us... [14:21:00] 06cloud-services-team, 10Toolforge: Buildservice "network is unreachable" error - https://phabricator.wikimedia.org/T362958#11917740 (10aputhin) p:05Triage→03Low Let's check if this happens still and/or often. Might be a problem of the past. [14:21:15] 10Toolforge, 06tools-platform-team: Buildservice "network is unreachable" error - https://phabricator.wikimedia.org/T362958#11917746 (10aputhin) [14:21:20] 10Tool-global-search: "500: Internal Server Error" when global-searching with a regex that includes a less-than symbol (<) - https://phabricator.wikimedia.org/T426204 (10A_smart_kitten) 03NEW [14:26:54] 06cloud-services-team, 10Toolforge: reset kube config for montage-dev - https://phabricator.wikimedia.org/T425289#11917814 (10aputhin) a:03komla [14:31:26] 06cloud-services-team, 10Toolforge, 06tools-platform-team: toolforge webservice restart does not wait for pod to be ready, only running - https://phabricator.wikimedia.org/T425215#11917870 (10aputhin) p:05Triage→03Medium We plan to replace the `webservice-cli` with `jobs-cli` (T348755), so we might end u... [14:38:14] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [components-api] build quota failures leave state unclear - https://phabricator.wikimedia.org/T422951#11917920 (10aputhin) > that does not cover builds that already finished and would actually break my workflow (which relies on partial builds to work... [14:42:15] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [builds-api] expose supported versions - https://phabricator.wikimedia.org/T422046#11917935 (10aputhin) [14:43:34] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: add more logs tests to toolforge-deploy - https://phabricator.wikimedia.org/T418326#11917942 (10aputhin) [14:55:54] (03open) 10vriaa: feat: add support for localisable messages [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/63 (https://phabricator.wikimedia.org/T423927) [15:01:14] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team, 07OKR-Work: [SPIKE] Identify OAD properties not supported by MediaWiki REST Framework - https://phabricator.wikimedia.org/T425942#11918038 (10KBach) [15:05:18] 06cloud-services-team, 10Data-Services, 06Data-Persistence: imagelinks table missing in pplwiki, several other tables missing in testcommonswiki - https://phabricator.wikimedia.org/T425384#11918045 (10BTullis) I have now run the following: ` btullis@an-redacteddb1001:~$ sudo maintain-views --replace --all-da... [15:14:01] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/wikiinfo] - 10https://gerrit.wikimedia.org/r/1285777 (owner: 10L10n-bot) [15:14:09] (03CR) 10Abijeet Patro: [V:03+2] "recheck" [labs/tools/wikiinfo] - 10https://gerrit.wikimedia.org/r/1285777 (owner: 10L10n-bot) [15:48:04] 10Toolforge (Quota-requests), 06Machine-Learning-Team (Q4 FY2025-26): Request increased quota for wiki-tts Toolforge tool - https://phabricator.wikimedia.org/T425804#11918262 (10komla) > Since each replica can use 1CPU and 2Gi RAM, we may not need changes to the per-job LimitRange. We request an increase to o... [15:53:20] (03update) 10raymond-ndibe: [webservice-smoke-test.bats]: fix functional tests [repos/cloud/toolforge/toolforge-deploy] (add_tmp_mount) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1246 (https://phabricator.wikimedia.org/T425479) [15:53:53] (03update) 10raymond-ndibe: use Istio port instead of ingress-nginx [repos/cloud/toolforge/toolforge-deploy] (add_tmp_mount) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1246 (https://phabricator.wikimedia.org/T425479) [15:54:03] (03update) 10raymond-ndibe: use Istio port instead of ingress-nginx [repos/cloud/toolforge/toolforge-deploy] (add_tmp_mount) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1246 (https://phabricator.wikimedia.org/T425479) [15:54:23] (03update) 10raymond-ndibe: use Istio port instead of ingress-nginx [repos/cloud/toolforge/toolforge-deploy] (add_tmp_mount) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1246 (https://phabricator.wikimedia.org/T425479) [16:03:33] 06cloud-services-team, 10Toolforge: reset kube config for montage-dev - https://phabricator.wikimedia.org/T425289#11918352 (10komla) 05Open→03Resolved [16:11:36] (03update) 10raymond-ndibe: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) [16:11:41] (03update) 10raymond-ndibe: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) [16:11:43] (03approved) 10raymond-ndibe: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) [16:13:14] (03update) 10raymond-ndibe: use Istio port instead of ingress-nginx [repos/cloud/toolforge/toolforge-deploy] (add_tmp_mount) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1246 (https://phabricator.wikimedia.org/T425479) [16:14:14] (03update) 10raymond-ndibe: use Istio port instead of ingress-nginx [repos/cloud/toolforge/toolforge-deploy] (add_tmp_mount) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1246 (https://phabricator.wikimedia.org/T425479) [16:14:37] (03update) 10fnegri: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) (owner: 10raymond-ndibe) [16:14:38] (03approved) 10fnegri: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) (owner: 10raymond-ndibe) [16:15:11] (03update) 10raymond-ndibe: use Istio port instead of ingress-nginx [repos/cloud/toolforge/toolforge-deploy] (add_tmp_mount) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1246 (https://phabricator.wikimedia.org/T425479) [16:15:14] (03update) 10raymond-ndibe: use Istio port instead of ingress-nginx [repos/cloud/toolforge/toolforge-deploy] (add_tmp_mount) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1246 (https://phabricator.wikimedia.org/T425479) [16:16:21] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [16:16:23] !log raymond-ndibe@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component istio-gateway [16:16:41] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [16:16:49] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway [16:17:52] 10Toolforge (Quota-requests), 06Machine-Learning-Team (Q4 FY2025-26): Request increased quota for wiki-tts Toolforge tool - https://phabricator.wikimedia.org/T425804#11918415 (10kevinbazira) [16:19:14] 10Toolforge (Quota-requests), 06Machine-Learning-Team (Q4 FY2025-26): Request increased quota for wiki-tts Toolforge tool - https://phabricator.wikimedia.org/T425804#11918417 (10kevinbazira) >>! In T425804#11918262, @komla wrote: > @kevinbazira please go ahead and update the request with the latest updates done [16:22:31] (03update) 10raymond-ndibe: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) [16:22:46] (03update) 10raymond-ndibe: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) [16:25:37] (03update) 10raymond-ndibe: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) [16:25:43] (03update) 10raymond-ndibe: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) [16:27:00] (03update) 10raymond-ndibe: [istio-gateway] add tmp mount [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1247 (https://phabricator.wikimedia.org/T425479) [16:27:32] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [16:28:23] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [lima-kilo] istio not starting correctly - https://phabricator.wikimedia.org/T425479#11918467 (10fnegri) >> istio-gateway pods failing with Failed to create temporary file > Still investigating this one, to understand if we need a patch like https://git... [16:28:40] (03merge) 10filippo: d/changelog: bump to 0.103.21 [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/105 (https://phabricator.wikimedia.org/T420565) [16:30:14] FIRING: [4x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-gateway-1.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [16:30:37] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:33:44] FIRING: [3x] IstioGatewayPodMisplaced: istio-gateway pod misplaced - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/IstioGatewayPodMisplaced - https://prometheus-alerts.wmcloud.org/?q=alertname%3DIstioGatewayPodMisplaced [16:35:14] FIRING: [6x] ToolforgeKubernetesHAproxyServerDown: Toolforge HAProxy server toolsbeta-test-k8s-gateway-1.toolsbeta.eqiad1.wikimedia.cloud is DOWN - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [16:35:37] FIRING: [8x] ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [16:38:43] (03update) 10raymond-ndibe: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] (use_hostname_for_harbor_not_in_local) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417 https://phabricator.wikimedia.org/T424362) (owner: 10fnegri) [16:38:57] (03update) 10raymond-ndibe: builds-api: update heroku builder and runner images [repos/cloud/toolforge/toolforge-deploy] (use_hostname_for_harbor_not_in_local) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225 (https://phabricator.wikimedia.org/T423417 https://phabricator.wikimedia.org/T424362) (owner: 10fnegri) [16:40:33] FIRING: ToolforgeWebHighErrorRate: High 5xx rate on Toolforge web services #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeWebHighErrorRate - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/infra-k8s-haproxy?var-frontend=k8s-ingress-https&var-backend=k8s-ingress-http&var-cluster=prometheus-toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeWebHighErrorRate [16:41:58] (03update) 10fnegri: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [16:44:21] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway [16:46:07] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11918551 (10Arnoldokoth) There is an issue running `scap` from the deploy h... [16:46:38] (03update) 10raymond-ndibe: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) [16:47:50] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway [16:57:14] (03update) 10fnegri: components: use harbor.local hostname for harbor in local env [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1254 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [16:58:06] (03update) 10fnegri: components: use harbor.local hostname for harbor in local env [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1254 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [16:58:09] (03update) 10fnegri: components: use harbor.local hostname for harbor in local env [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1254 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [16:58:12] (03update) 10fnegri: components: use harbor.local hostname for harbor in local env [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1254 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [16:58:14] (03approved) 10fnegri: components: use harbor.local hostname for harbor in local env [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1254 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [17:03:58] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway [17:11:54] (03PS1) 10Herron: add dummy token for dashboard reporter [labs/private] - 10https://gerrit.wikimedia.org/r/1286988 [17:13:04] (03update) 10fnegri: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/159 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [17:22:14] (03update) 10fnegri: allow harbor.local host [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/39 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [17:40:30] (03update) 10fnegri: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/298 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [17:40:35] (03approved) 10fnegri: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/298 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [17:40:52] (03update) 10fnegri: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [18:17:29] (03update) 10fnegri: rename harbor_ip to harbor_hostname [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [18:27:58] (03update) 10fnegri: allow harbor.local host [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/39 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [18:28:04] (03update) 10fnegri: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/298 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [18:30:55] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#11919020 (10Arnoldokoth) P92512 I've copied the error output here. [18:33:33] (03update) 10fnegri: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [18:34:13] (03update) 10fnegri: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/159 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [18:34:28] (03approved) 10fnegri: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/159 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [18:34:40] (03update) 10fnegri: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/159 (https://phabricator.wikimedia.org/T426016) (owner: 10raymond-ndibe) [18:43:29] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: heroku builder and runner 24_0.21.8 rejects harbor ip host - https://phabricator.wikimedia.org/T426016#11919050 (10fnegri) [19:08:35] 10Tool-humaniki-2: Use dumps? - https://phabricator.wikimedia.org/T423012#11919145 (10Danya) [19:08:38] 10Tool-humaniki-2, 13Patch-For-Review: Integrate Denelezh and merge stats tables - https://phabricator.wikimedia.org/T422343#11919144 (10Danya) [19:09:09] 10Tool-humaniki-2: Use dumps? - https://phabricator.wikimedia.org/T423012#11919149 (10Danya) p:05Triage→03High [19:11:19] 10Tool-humaniki-2: Use dumps - https://phabricator.wikimedia.org/T423012#11919150 (10Danya) [19:11:51] 10Tool-humaniki-2: Use dumps - https://phabricator.wikimedia.org/T423012#11919153 (10Danya) [19:12:10] 10Tool-humaniki-2, 13Patch-For-Review: Integrate Denelezh and merge stats tables - https://phabricator.wikimedia.org/T422343#11919154 (10Danya) [19:12:17] 10Tool-humaniki-2: Migrate site to Cloud VPS? - https://phabricator.wikimedia.org/T425489#11919155 (10Danya) [19:12:39] 10Tool-humaniki-2: Scheduler does not take full snapshots - https://phabricator.wikimedia.org/T423025#11919156 (10Danya) 05In progress→03Stalled [19:13:38] 10Tool-humaniki-2, 13Patch-For-Review: Integrate Denelezh and merge stats tables - https://phabricator.wikimedia.org/T422343#11919166 (10Danya) 05In progress→03Stalled Stalled until server is migrated to Cloud VPS (T425489). [19:13:57] 10Tool-humaniki-2: Migrate site to Cloud VPS? - https://phabricator.wikimedia.org/T425489#11919172 (10Danya) p:05Triage→03High [19:14:44] 10Tool-humaniki-2: Use ToolsDB instead of SQLite - https://phabricator.wikimedia.org/T425991#11919185 (10Danya) [19:14:45] 10Tool-humaniki-2: Migrate site to Cloud VPS? - https://phabricator.wikimedia.org/T425489#11919186 (10Danya) [19:15:01] 10Tool-humaniki-2: Use dumps - https://phabricator.wikimedia.org/T423012#11919188 (10Danya) [19:15:03] 10Tool-humaniki-2: Migrate site to Cloud VPS? - https://phabricator.wikimedia.org/T425489#11919189 (10Danya) [19:15:34] 10Tool-humaniki-2: Migrate site to Cloud VPS - https://phabricator.wikimedia.org/T425489#11919192 (10Danya) [19:17:34] 06cloud-services-team, 10Cloud-VPS: Bastion immediately dropping SSH connection (LDAP sync issue?) - https://phabricator.wikimedia.org/T425815#11919196 (10Jony) Hi Andrew, Great! https://chat.wmcloud.org is now officially live! The migration is completed, and the service is running smoothly on Wikimedia Cl... [19:17:56] 10Tool-humaniki-2: Investigate best solution between ToolsDB and self-hosted database - https://phabricator.wikimedia.org/T425991#11919197 (10Danya) [19:18:58] 10Tool-humaniki-2: Investigate best solution between ToolsDB and self-hosted database - https://phabricator.wikimedia.org/T425991#11919199 (10Danya) [19:18:59] 10Tool-humaniki-2: Use dumps - https://phabricator.wikimedia.org/T423012#11919198 (10Danya) [19:19:00] 10Tool-humaniki-2: Migrate site to Cloud VPS - https://phabricator.wikimedia.org/T425489#11919200 (10Danya) [19:20:03] 10Tool-humaniki-2: Scheduler does not take full snapshots - https://phabricator.wikimedia.org/T423025#11919215 (10Danya) 05Stalled→03In progress [19:20:04] 10Tool-humaniki-2, 13Patch-For-Review: Integrate Denelezh and merge stats tables - https://phabricator.wikimedia.org/T422343#11919213 (10Danya) 05Stalled→03In progress [19:20:09] 10Tool-humaniki-2, 13Patch-For-Review: Integrate Denelezh and merge stats tables - https://phabricator.wikimedia.org/T422343#11919218 (10Danya) 05In progress→03Stalled [19:20:14] 10Tool-humaniki-2: Scheduler does not take full snapshots - https://phabricator.wikimedia.org/T423025#11919219 (10Danya) 05In progress→03Stalled [19:54:14] (03update) 10raymond-ndibe: deployment: set harbor hostname for local to harbor.local [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/159 (https://phabricator.wikimedia.org/T426016) [20:04:16] (03update) 10raymond-ndibe: kubernetes.py: handle 409 errors [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/104 (https://phabricator.wikimedia.org/T423005) [20:17:47] (03open) 10wikigit: Migrate to ruleset repo package [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/22 [20:19:18] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-33 (2026-05-05 to 2026-05-19)), 07OKR-Work: Connect the new ruleset repository to the linter tool - https://phabricator.wikimedia.org/T425626#11919533 (10Mooeypoo) a:03Mooeypoo [20:20:53] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-33 (2026-05-05 to 2026-05-19)), 07OKR-Work: Connect the new ruleset repository to the linter tool - https://phabricator.wikimedia.org/T425626#11919544 (10Mooeypoo) Merge request: https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merg... [20:29:09] 06cloud-services-team, 10Cloud-VPS: upgrade WMCS ceph nodes to Debian Trixie - https://phabricator.wikimedia.org/T413726#11919576 (10Andrew) Surprisingly this is still blocked on the same issue. A ceph slack response: "Most of the work on this area has fallen into the bucket of 'rocky10' support. Because rocky... [20:35:51] (03update) 10raymond-ndibe: use hostname for harbor not ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/322 (https://phabricator.wikimedia.org/T426016) [20:38:28] (03update) 10raymond-ndibe: rename harbor_ip to harbor_hostname [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/90 (https://phabricator.wikimedia.org/T426016) [20:40:19] 10Tool-humaniki-2: Clearly state that this is not an official WMF project - https://phabricator.wikimedia.org/T426259 (10Danya) 03NEW [20:40:23] 10Tool-humaniki-2: Clearly state that this is not an official WMF project - https://phabricator.wikimedia.org/T426259#11919702 (10Danya) [20:40:32] 10Tool-humaniki-2: Clearly state that this is not an official WMF project - https://phabricator.wikimedia.org/T426259#11919704 (10Danya) p:05Triage→03High [20:48:46] 10Tool-humaniki-2: Add Privacy Statement - https://phabricator.wikimedia.org/T426262 (10Danya) 03NEW [20:48:56] 10Tool-humaniki-2: Add Privacy Statement - https://phabricator.wikimedia.org/T426262#11919784 (10Danya) [20:49:00] 10Tool-humaniki-2: Add Privacy Statement - https://phabricator.wikimedia.org/T426262#11919786 (10Danya) p:05Triage→03Medium [20:51:36] 10Tool-humaniki-2: Add Privacy Statement - https://phabricator.wikimedia.org/T426262#11919817 (10JJMC89) [20:51:58] 10Tool-humaniki-2: Link to the Cloud Services Terms of Use - https://phabricator.wikimedia.org/T426264 (10Danya) 03NEW [20:52:03] 10Tool-humaniki-2: Link to the Cloud Services Terms of Use - https://phabricator.wikimedia.org/T426264#11919834 (10Danya) [20:52:08] 10Tool-humaniki-2: Link to the Cloud Services Terms of Use - https://phabricator.wikimedia.org/T426264#11919836 (10Danya) p:05Triage→03Medium [22:44:51] (03approved) 10kineticpelagic: Migrate to ruleset repo package [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/22 (owner: 10wikigit) [23:08:49] (03approved) 10milazg: Migrate to ruleset repo package [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/22 (owner: 10wikigit)