[00:19:02] (03CR) 10CI reject: [V:04-1] build: Updating mediawiki/mediawiki-codesniffer to 47.0.0 [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1143223 (owner: 10Libraryupgrader) [00:40:55] 10Tool-gitlab-content, 07User-notice, 03Wikimedia-Hackathon-2025: Implement a reverse proxy for gitlab.wikimedia.org raw content that supports mime-type specification - https://phabricator.wikimedia.org/T392431#10802966 (10UOzurumba) Hello @bd808, for tech news - What wording would you suggest as the content... [02:05:41] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:52:23] 10Tool-inteGraality, 10VerySmallGLAM, 10Wikibase (3rd party installations), 06Wikibase Suite Team, 07Product-Feature: integraality for Wikibases? - https://phabricator.wikimedia.org/T294892#10803277 (10Olea) > Perhaps we need a centeralized tool to enable other tools to easily interact with other Wikiba... [07:04:35] (03PS1) 10Muehlenhoff: Remove obsolete videoscaler stub certs [labs/private] - 10https://gerrit.wikimedia.org/r/1143392 (https://phabricator.wikimedia.org/T360636) [07:07:41] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove obsolete videoscaler stub certs [labs/private] - 10https://gerrit.wikimedia.org/r/1143392 (https://phabricator.wikimedia.org/T360636) (owner: 10Muehlenhoff) [08:05:03] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [08:09:29] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10803355 (10taavi) [08:17:19] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [08:19:05] (03approved) 10dcaro: api-gateway: bump to 0.0.67-20250507163032-f26d5cd1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/764 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:19:09] (03merge) 10dcaro: api-gateway: bump to 0.0.67-20250507163032-f26d5cd1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/764 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:21:31] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate Puppet CA: project-proxy-puppetmaster-01.project-proxy.eqiad.wmflabs is about to expire in 17d 18h 14m 54s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [08:21:48] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/131 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:21:51] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/131 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:27:07] (03update) 10dcaro: api-gateway: bump to 0.0.68-20250507164636-2fdb8c02 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/764 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:27:47] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10803449 (10taavi) [08:28:42] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.189-20250508082204-c72b2ef2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/765 [08:29:17] (03close) 10dcaro: create_precommit_update_mrs: add the `deploy: skip` header [repos/cloud/cicd/gitlab-ci] (add_skip_deploy_detection) - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/54 [08:29:28] (03close) 10dcaro: create_toolforge_deploy_mr: skip deployment if flagged [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/53 [08:30:13] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [08:30:15] (03approved) 10taavi: create_toolforge_deploy_mr: update MR title/desc if exists [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/55 (owner: 10dcaro) [08:31:38] (03merge) 10dcaro: create_toolforge_deploy_mr: update MR title/desc if exists [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/55 [08:34:51] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [08:35:36] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [08:39:29] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [08:40:10] FIRING: ProjectProxyMainProxyInstanceDown: Proxy on proxy-03 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyInstanceDown [08:43:39] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [08:44:10] FIRING: ProjectProxyMainProxyDown: Proxy service address is unreachable - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyDown [08:45:10] FIRING: ProjectProxyMainProxyInstanceDown: Proxy on proxy-04 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyInstanceDown [08:46:28] FIRING: InstanceDown: Project project-proxy instance proxy-03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [08:48:39] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [08:50:10] RESOLVED: ProjectProxyMainProxyInstanceDown: Proxy on proxy-04 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/MainProxyInstanceDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProjectProxyMainProxyInstanceDown [08:51:11] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [08:51:18] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [08:51:28] RESOLVED: InstanceDown: Project project-proxy instance proxy-03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [08:53:26] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [08:53:32] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [08:57:56] FIRING: HarborDown: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [08:57:57] FIRING: HarborDown: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [09:04:17] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [09:05:04] (03open) 10dcaro: openapi: retry a few times when accessing a backend [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/66 [09:05:16] (03update) 10dcaro: openapi: retry a few times when accessing a backend [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/66 [09:07:56] RESOLVED: HarborDown: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [09:07:57] RESOLVED: HarborDown: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [09:11:28] FIRING: InstanceDown: Project project-proxy instance proxy-03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:14:09] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [09:16:28] RESOLVED: InstanceDown: Project project-proxy instance proxy-03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:17:45] 06cloud-services-team, 10Cloud-VPS, 06SRE: [cloudsw] enable 25G network - https://phabricator.wikimedia.org/T393676 (10dcaro) 03NEW [09:17:55] 06cloud-services-team, 10Cloud-VPS, 06SRE: [cloudsw] enable 25G network - https://phabricator.wikimedia.org/T393676#10803598 (10dcaro) [09:18:08] 06cloud-services-team, 10Cloud-VPS, 06SRE: [cloudsw] enable 25G network - https://phabricator.wikimedia.org/T393676#10803602 (10dcaro) [09:18:19] 06cloud-services-team, 10Cloud-VPS, 06SRE: [cloudsw] enable 25G network - https://phabricator.wikimedia.org/T393676#10803604 (10dcaro) [09:18:27] 06cloud-services-team, 10Cloud-VPS, 06SRE: [cloudsw] enable 25G network - https://phabricator.wikimedia.org/T393676#10803606 (10dcaro) [09:20:02] 06cloud-services-team, 10Cloud-VPS, 06SRE: [cloudsw] enable 25G network - https://phabricator.wikimedia.org/T393676#10803608 (10dcaro) @ayounsi @cmooney feel free to use this task for this work, or link the one you are using for it, thanks! [09:23:50] 06cloud-services-team, 10Cloud-VPS, 06SRE: [cloudsw] enable 25G network - https://phabricator.wikimedia.org/T393676#10803615 (10cmooney) @dcaro is there another task for those servers to be installed/provisioned? We have support for 25G in Eqiad racks E4 and F4, and codfw B1. Not in eqiad C8/D5. It's main... [09:25:48] 06cloud-services-team, 10Cloud-VPS, 06SRE: [cloudsw] enable 25G network - https://phabricator.wikimedia.org/T393676#10803628 (10dcaro) >>! In T393676#10803615, @cmooney wrote: > @dcaro is there another task for those servers to be installed/provisioned? It's linked as parent {T389851}, not yet bought, but w... [09:25:54] 06cloud-services-team, 10Cloud-VPS, 06SRE: [cloudsw] enable 25G network - https://phabricator.wikimedia.org/T393676#10803630 (10cmooney) Sorry I found it. I added a note about C8/D5 in eqiad. [09:33:08] 06cloud-services-team, 10Cloud-VPS, 06SRE: [cloudsw] enable 25G network - https://phabricator.wikimedia.org/T393676#10803636 (10dcaro) > We will need to support 25G on all of the racks, as we have to spread the nodes for high availability (specially critical if the hosts are that big) This is a blocker to b... [09:34:00] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [09:46:15] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [09:51:24] 06cloud-services-team, 10Cloud-VPS, 06SRE: [cloudsw] enable 25G network - https://phabricator.wikimedia.org/T393676#10803654 (10dcaro) [09:52:01] 06cloud-services-team, 10Cloud-VPS, 06SRE: [cloudsw] enable 25G network - https://phabricator.wikimedia.org/T393676#10803656 (10dcaro) p:05Triage→03High [09:52:02] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: add proper dry-run/diff mode to maintain-views - https://phabricator.wikimedia.org/T351637#10803657 (10fnegri) 05Open→03In progress a:03fnegri [09:52:58] (03approved) 10dcaro: builds-api: bump to 0.0.189-20250508082204-c72b2ef2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/765 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:53:01] (03merge) 10dcaro: builds-api: bump to 0.0.189-20250508082204-c72b2ef2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/765 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:53:18] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/162 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [09:53:21] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/162 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:08:44] (03open) 10dcaro: create_toolforge_deploy_mr: don't push bare newlines [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/56 [10:09:20] (03update) 10dcaro: create_toolforge_deploy_mr: don't push bare newlines [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/56 [10:09:29] (03update) 10dcaro: create_toolforge_deploy_mr: don't push bare newlines [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/56 [10:10:29] (03approved) 10aborrero: create_toolforge_deploy_mr: don't push bare newlines [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/56 (owner: 10dcaro) [10:10:52] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: tofuinfratest fails to create web proxy - https://phabricator.wikimedia.org/T393679 (10fnegri) 03NEW [10:12:51] (03merge) 10dcaro: create_toolforge_deploy_mr: don't push bare newlines [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/56 [10:12:51] (03update) 10dcaro: create_toolforge_deploy_mr: don't push bare newlines [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/56 [10:14:22] (03PS4) 10Raymond Ndibe: [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [10:17:40] (03CR) 10CI reject: [V:04-1] [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [10:21:50] 06cloud-services-team, 10Cloud-VPS: Cloud-hosts connected at 1G - https://phabricator.wikimedia.org/T349735#10803751 (10taavi) 05Open→03Resolved a:03taavi None of those hosts are around anymore, so closing. [10:25:08] (03PS5) 10Raymond Ndibe: [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [10:28:08] (03CR) 10CI reject: [V:04-1] [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [10:28:25] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.371-20250508101549-e00444d9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/766 [10:31:14] (03update) 10dcaro: openapi: retry a few times when accessing a backend [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/66 [10:33:58] (03PS6) 10Raymond Ndibe: [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [10:34:23] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/163 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:35:07] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/163 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:36:26] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/163 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:37:02] (03CR) 10CI reject: [V:04-1] [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [10:38:56] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.372-20250508103639-0dd28f2a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/766 [10:38:59] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.372-20250508103639-0dd28f2a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/766 [10:39:29] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [10:41:30] (03PS7) 10Raymond Ndibe: [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [10:43:40] 10Striker, 10CAS-SSO, 06Infrastructure-Foundations: Use IDP for authentication in Striker - https://phabricator.wikimedia.org/T359554#10803809 (10Arendpieter) [10:43:45] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/80 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:43:48] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/80 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:43:54] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [10:44:08] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/42 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:44:10] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/42 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:44:18] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/104 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:44:22] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/104 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:44:27] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/30 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:44:29] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/30 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:45:18] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [10:45:24] (03CR) 10CI reject: [V:04-1] [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [10:54:57] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [10:56:05] 10Tool-yearinreview: Auto-Deploy on Merge – Year in Review Tool - https://phabricator.wikimedia.org/T393683 (10Gopavasanth) 03NEW [10:56:34] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10803841 (10taavi) [10:56:47] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10803842 (10taavi) [10:57:41] (03update) 10raymond-ndibe: [toolforge-deploy] add test tags to component charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/755 (https://phabricator.wikimedia.org/T381011) [11:01:32] 10Tool-yearinreview: Auto-Deploy on Merge – Year in Review Tool - https://phabricator.wikimedia.org/T393683#10803844 (10Gopavasanth) a:05Gopavasanth→03None [11:01:34] 10Tool-yearinreview: Auto-Deploy on Merge – Year in Review Tool - https://phabricator.wikimedia.org/T393683#10803845 (10Gopavasanth) Refs: https://github.com/indictechcom/translatable-wikitext-converter/blob/main/.github/workflows/deploy.yml [11:01:47] (03PS2) 10Wandji collins: T386331: Remove not required packages from requirements [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133577 (owner: 10Bovimacoco) [11:02:42] 10Tool-toolwatch: Auto-Deploy on Merge – ToolWatch - https://phabricator.wikimedia.org/T393685 (10Gopavasanth) 03NEW [11:03:59] (03Abandoned) 10Wandji collins: Remove unrequired packages from requirements Bug: T386331 [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132115 (https://phabricator.wikimedia.org/T386331) (owner: 10Juniorbesong) [11:04:33] (03CR) 10Wandji collins: [C:03+2] T386331: Remove not required packages from requirements [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133577 (owner: 10Bovimacoco) [11:05:00] (03CR) 10Wandji collins: [V:03+2 C:03+2] T386331: Remove not required packages from requirements [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133577 (owner: 10Bovimacoco) [11:07:15] (03PS2) 10Wandji collins: T386329:Rearranging the Readme files [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132161 (owner: 10Bovimacoco) [11:08:04] (03CR) 10Wandji collins: [C:03+2] T386329:Rearranging the Readme files [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132161 (owner: 10Bovimacoco) [11:08:09] (03CR) 10Wandji collins: [V:03+2 C:03+2] T386329:Rearranging the Readme files [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132161 (owner: 10Bovimacoco) [11:09:44] (03PS2) 10Wandji collins: T390368: Format ReadMe file to display properly [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132164 (owner: 10Bovimacoco) [11:10:07] (03CR) 10Wandji collins: [C:03+2] T390368: Format ReadMe file to display properly [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132164 (owner: 10Bovimacoco) [11:10:14] (03CR) 10Wandji collins: [V:03+2 C:03+2] T390368: Format ReadMe file to display properly [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132164 (owner: 10Bovimacoco) [11:12:29] (03PS8) 10Raymond Ndibe: [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [11:14:29] 06cloud-services-team, 10Toolforge: tofu-provisioning: factorize gitlab pipeline logic - https://phabricator.wikimedia.org/T393686 (10aborrero) 03NEW [11:14:43] 06cloud-services-team, 10Toolforge: tofu-provisioning: factorize gitlab pipeline logic - https://phabricator.wikimedia.org/T393686#10803888 (10aborrero) 05Open→03In progress [11:15:32] (03CR) 10CI reject: [V:04-1] [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [11:15:53] 06cloud-services-team, 10Toolforge: tofu-provisioning: factorize gitlab pipeline logic - https://phabricator.wikimedia.org/T393686#10803891 (10aborrero) p:05Triage→03Medium [11:21:12] (03PS2) 10Wandji collins: T386329 = Remove app pycache files from git [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132165 (owner: 10Bovimacoco) [11:21:30] (03CR) 10Wandji collins: T386329 = Remove app pycache files from git (031 comment) [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132165 (owner: 10Bovimacoco) [11:22:09] (03PS3) 10Wandji collins: T386329: Remove app pycache files from git [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132165 (owner: 10Bovimacoco) [11:22:23] (03CR) 10Wandji collins: [C:03+2] T386329: Remove app pycache files from git [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132165 (owner: 10Bovimacoco) [11:22:26] (03CR) 10Wandji collins: [V:03+2 C:03+2] T386329: Remove app pycache files from git [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132165 (owner: 10Bovimacoco) [11:28:01] (03PS9) 10Raymond Ndibe: [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [11:28:28] (03Abandoned) 10Wandji collins: T386329:Rearranging the Readme files [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132161 (owner: 10Bovimacoco) [11:31:01] (03CR) 10CI reject: [V:04-1] [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [11:32:24] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10803962 (10taavi) [11:33:23] (03PS10) 10Raymond Ndibe: [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [11:34:24] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) [11:34:46] (03update) 10aborrero: gitlab-ci: replace local logic with included one [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/23 (https://phabricator.wikimedia.org/T393686) [11:36:23] (03CR) 10CI reject: [V:04-1] [wmcs-cookbooks] run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [11:38:47] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) [11:38:48] (03PS3) 10Wandji collins: T390368: Format ReadMe file to display properly [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132164 (owner: 10Bovimacoco) [11:40:43] (03Abandoned) 10Wandji collins: T386326 Remove duplicate routes copy file [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132166 (owner: 10Bovimacoco) [11:42:03] (03CR) 10Majavah: [wmcs-cookbooks] run specific tests on deploy (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [11:43:50] (03Abandoned) 10Wandji collins: Removing the duplicate app.__pycache__ from gitignore leaving only 1 [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132593 (owner: 10Bovimacoco) [11:45:44] (03CR) 10Wandji collins: [C:03+2] T390368: Format ReadMe file to display properly [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132164 (owner: 10Bovimacoco) [11:45:49] (03CR) 10Wandji collins: [V:03+2 C:03+2] T390368: Format ReadMe file to display properly [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132164 (owner: 10Bovimacoco) [11:46:23] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) [11:47:38] (03update) 10aborrero: gitlab-ci: replace local logic with included one [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/23 (https://phabricator.wikimedia.org/T393686) [11:48:17] (03CR) 10Wandji collins: [C:03+2] Remove init.py file from application - Added venv to gitignore [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1125263 (https://phabricator.wikimedia.org/T386325) (owner: 10Eugene233) [11:48:24] (03CR) 10Wandji collins: [V:03+2 C:03+2] Remove init.py file from application - Added venv to gitignore [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1125263 (https://phabricator.wikimedia.org/T386325) (owner: 10Eugene233) [11:50:45] (03PS2) 10Wandji collins: BE-Curate accepted language codes. [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133584 (owner: 10Bovimacoco) [11:51:03] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) [11:51:07] (03update) 10aborrero: gitlab-ci: replace local logic with included one [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/23 (https://phabricator.wikimedia.org/T393686) [11:51:20] 10Tool-wdactle, 10Wikidata: Add API (option) to format Wikidata entities as plain text in bulk - https://phabricator.wikimedia.org/T393691 (10LucasWerkmeister) 03NEW [11:53:21] (03PS3) 10Wandji collins: BE-Curate accepted language codes. [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133584 (owner: 10Bovimacoco) [11:53:56] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) [11:54:20] (03CR) 10Wandji collins: [C:03+2] BE-Curate accepted language codes. [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133584 (owner: 10Bovimacoco) [11:54:25] (03CR) 10Wandji collins: [V:03+2 C:03+2] BE-Curate accepted language codes. [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133584 (owner: 10Bovimacoco) [11:55:05] 06cloud-services-team, 10Horizon: Allow encrypting connection from the shared Cloud VPS web proxy to the backend - https://phabricator.wikimedia.org/T274386#10804014 (10taavi) a:03taavi [12:02:54] (03PS1) 10Majavah: Support specifying a protocol [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1143541 (https://phabricator.wikimedia.org/T274386) [12:03:30] (03PS4) 10Wandji collins: BE-Curate accepted language codes. [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133584 (owner: 10Bovimacoco) [12:04:11] (03CR) 10Wandji collins: [C:03+2] BE-Curate accepted language codes. [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133584 (owner: 10Bovimacoco) [12:04:14] (03CR) 10Wandji collins: [V:03+2 C:03+2] BE-Curate accepted language codes. [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133584 (owner: 10Bovimacoco) [12:04:27] (03update) 10dcaro: openapi: retry a few times when accessing a backend [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/66 [12:04:37] (03update) 10dcaro: openapi: retry a few times when accessing a backend [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/66 [12:06:05] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:07:28] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) [12:13:18] (03update) 10raymond-ndibe: openapi: retry a few times when accessing a backend [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/66 (owner: 10dcaro) [12:13:24] (03update) 10raymond-ndibe: openapi: retry a few times when accessing a backend [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/66 (owner: 10dcaro) [12:13:25] (03approved) 10raymond-ndibe: openapi: retry a few times when accessing a backend [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/66 (owner: 10dcaro) [12:16:11] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) [12:16:13] (03update) 10aborrero: gitlab-ci: replace local logic with included one [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/23 (https://phabricator.wikimedia.org/T393686) [12:17:36] 06cloud-services-team, 10Toolforge: tofu-provisioning: factorize gitlab pipeline logic - https://phabricator.wikimedia.org/T393686#10804138 (10taavi) +1. Please also migrate the tofu container image to somewhere where it doesn't need to be rebuilt for every single pipeline run. [12:18:46] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [12:20:12] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) [12:21:53] 06cloud-services-team, 10Toolforge: tofu-provisioning: factorize gitlab pipeline logic - https://phabricator.wikimedia.org/T393686#10804158 (10aborrero) working on this: https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 example: https://gitlab.wikimedia.org/repos/cloud/cloud-vps/netwo... [12:24:07] (03merge) 10dcaro: openapi: retry a few times when accessing a backend [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/66 [12:24:39] (03approved) 10dcaro: jobs-api: bump to 0.0.372-20250508103639-0dd28f2a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/766 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:24:42] (03merge) 10dcaro: jobs-api: bump to 0.0.372-20250508103639-0dd28f2a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/766 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:26:18] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.69-20250508122418-92f906ce [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/767 [12:26:20] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.69-20250508122418-92f906ce [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/767 [12:27:32] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/20 [12:27:33] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/36 [12:31:34] 10Tool-wdactle, 10Wikibase Action API (WPP), 10Wikidata, 13Patch-For-Review: Add API (option) to format Wikidata entities as plain text in bulk - https://phabricator.wikimedia.org/T393691#10804183 (10LucasWerkmeister) [12:31:56] FIRING: HarborDown: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [12:31:57] FIRING: HarborDown: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [12:39:14] (03PS3) 10Wandji collins: BE - search for lexemes forms matching word [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133586 (https://phabricator.wikimedia.org/T388191) (owner: 10Bovimacoco) [12:39:37] 06cloud-services-team, 10Toolforge, 07IPv6: Rebuild Toolforge Prometheus nodes in v6-dualstack network - https://phabricator.wikimedia.org/T393697 (10taavi) 03NEW [12:41:56] RESOLVED: HarborDown: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [12:41:57] RESOLVED: HarborDown: Harbor is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborDown [12:42:31] (03CR) 10Wandji collins: [C:03+2] BE - search for lexemes forms matching word [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133586 (https://phabricator.wikimedia.org/T388191) (owner: 10Bovimacoco) [12:42:33] (03CR) 10Wandji collins: [V:03+2 C:03+2] BE - search for lexemes forms matching word [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133586 (https://phabricator.wikimedia.org/T388191) (owner: 10Bovimacoco) [12:42:43] (03CR) 10Raymond Ndibe: [wmcs-cookbooks] run specific tests on deploy (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [12:43:08] (03PS11) 10Raymond Ndibe: toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [12:47:14] (03CR) 10CI reject: [V:04-1] toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [12:53:08] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [12:54:07] 14Toolforge (Toolforge iteration 14): [toolforge deploy] direct-api tests fail intermittently on toolsbeta - https://phabricator.wikimedia.org/T369891#10804229 (10dcaro) 05Resolved→03Open Re-opening, as this is still hapenning, some new logs from the api-gateway service when the issue happens: ` [pod/api-gat... [12:54:26] 10Toolforge (Toolforge iteration 19): [toolforge deploy] direct-api tests fail intermittently on toolsbeta - https://phabricator.wikimedia.org/T369891#10804232 (10dcaro) a:05dcaro→03None [12:56:38] (03PS2) 10Wandji collins: Remove duplicate templates dir. [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132113 (https://phabricator.wikimedia.org/T386327) (owner: 10Juniorbesong) [12:58:56] (03PS3) 10Wandji collins: Remove duplicate templates dir. [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132113 (https://phabricator.wikimedia.org/T386327) (owner: 10Juniorbesong) [13:00:27] (03Abandoned) 10Wandji collins: Remove duplicate templates dir [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132114 (https://phabricator.wikimedia.org/T386327) (owner: 10Juniorbesong) [13:00:44] (03CR) 10Wandji collins: [C:03+2] Remove duplicate templates dir. [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132113 (https://phabricator.wikimedia.org/T386327) (owner: 10Juniorbesong) [13:00:48] (03CR) 10Wandji collins: [V:03+2 C:03+2] Remove duplicate templates dir. [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132113 (https://phabricator.wikimedia.org/T386327) (owner: 10Juniorbesong) [13:02:34] 10cloud-services-team (Hardware): [ceph] Figure out hosts placement in the racks - https://phabricator.wikimedia.org/T393699 (10dcaro) 03NEW [13:03:07] (03PS12) 10Raymond Ndibe: toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [13:03:49] 06cloud-services-team, 10Toolforge, 07IPv6: Rebuild Toolforge Prometheus nodes in v6-dualstack network - https://phabricator.wikimedia.org/T393697#10804278 (10aborrero) Maybe we can use the opportunity to rebuild them via https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning [13:04:11] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10804279 (10taavi) [13:04:51] (03CR) 10Raymond Ndibe: toolforge: run specific tests on deploy (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [13:05:09] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/20 (owner: 10l10n-bot) [13:05:11] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/20 (owner: 10l10n-bot) [13:05:16] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [13:05:56] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/36 (owner: 10l10n-bot) [13:06:00] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/36 (owner: 10l10n-bot) [13:07:27] (03update) 10raymond-ndibe: [toolforge-deploy] add test tags to component charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/755 (https://phabricator.wikimedia.org/T381011) [13:08:37] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) [13:09:01] 10cloud-services-team (Hardware): [ceph] Figure out hosts placement in the racks - https://phabricator.wikimedia.org/T393699#10804309 (10cmooney) I think the first thing in terms of any potential 2x10G is to work out if it will do the trick. The hosts are already connected at 2x10G, to the normal network and th... [13:09:26] (03approved) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/22 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:09:28] (03update) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/22 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:09:31] (03merge) 10raymond-ndibe: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/22 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:13:29] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-admission: bump to 0.0.29-20250508130943-6c306f5d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/768 [13:13:47] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [13:16:05] (03update) 10raymond-ndibe: envvars-admission: bump to 0.0.29-20250508130943-6c306f5d [repos/cloud/toolforge/toolforge-deploy] (run_specific_tests_on_deploy) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/768 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:16:15] (03update) 10raymond-ndibe: envvars-admission: bump to 0.0.29-20250508130943-6c306f5d [repos/cloud/toolforge/toolforge-deploy] (run_specific_tests_on_deploy) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/768 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:17:09] 10cloud-services-team (Hardware): [ceph] Figure out hosts placement in the racks - https://phabricator.wikimedia.org/T393699#10804343 (10dcaro) [13:20:03] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [13:20:41] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [13:26:39] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [13:35:26] (03PS4) 10Wandji collins: Format ReadMe file to display properly [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132164 (owner: 10Bovimacoco) [13:36:36] (03CR) 10Wandji collins: [C:03+2] Format ReadMe file to display properly [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132164 (owner: 10Bovimacoco) [13:36:40] (03CR) 10Wandji collins: [V:03+2 C:03+2] Format ReadMe file to display properly [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132164 (owner: 10Bovimacoco) [13:40:56] 10Tool-wdactle, 10Wikibase Action API (WPP), 10Wikidata, 13Patch-For-Review: Add API (option) to format Wikidata entities as plain text in bulk - https://phabricator.wikimedia.org/T393691#10804441 (10LucasWerkmeister) [13:47:56] (03PS13) 10Raymond Ndibe: toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [13:52:06] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [13:52:21] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [13:52:37] (03CR) 10Wandji collins: [C:03+2] T386329: Remove app pycache files from git [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132165 (owner: 10Bovimacoco) [13:52:44] (03CR) 10Wandji collins: [V:03+2 C:03+2] T386329: Remove app pycache files from git [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132165 (owner: 10Bovimacoco) [13:52:56] 10cloud-services-team (Hardware): [ceph] Figure out hosts placement in the racks - https://phabricator.wikimedia.org/T393699#10804518 (10Andrew) It's also an option to just not order giant osd servers until we have 25G networking everywhere. I don't love that option but it will be a lot easier on netops if we ha... [13:59:04] (03PS4) 10Wandji collins: T386329: Remove app pycache files from git [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132165 (owner: 10Bovimacoco) [14:00:21] (03PS5) 10Wandji collins: T386329: Remove app pycache files from git [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132165 (owner: 10Bovimacoco) [14:00:32] (03CR) 10Wandji collins: [C:03+2] T386329: Remove app pycache files from git [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132165 (owner: 10Bovimacoco) [14:00:35] (03CR) 10Wandji collins: [V:03+2 C:03+2] T386329: Remove app pycache files from git [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1132165 (owner: 10Bovimacoco) [14:15:06] (03PS14) 10Raymond Ndibe: toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [14:17:09] (03approved) 10dcaro: api-gateway: bump to 0.0.69-20250508122418-92f906ce [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/767 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:17:12] (03merge) 10dcaro: api-gateway: bump to 0.0.69-20250508122418-92f906ce [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/767 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:22:20] 10cloud-services-team (Hardware): [ceph] Figure out hosts placement in the racks - https://phabricator.wikimedia.org/T393699#10804764 (10dcaro) [14:22:47] 10cloud-services-team (Hardware): [ceph] Figure out hosts placement in the racks - https://phabricator.wikimedia.org/T393699#10804780 (10dcaro) [14:24:58] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [14:25:14] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [14:41:05] (03PS15) 10Raymond Ndibe: toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [14:45:09] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [14:45:25] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [14:54:25] 10Quarry: Find somewhere else (not NFS) to store Quarry's resultsets - https://phabricator.wikimedia.org/T178520#10804962 (10taavi) Relating to this: does anyone happen to know if there's a specific reason why Quarry stores results as sqlite files and not, say, CSVs or some other format that the frontend could p... [15:04:06] 06cloud-services-team, 10Cloud-VPS, 10Data-Engineering (Q4 2025 April 1st - June 30th), 07IPv6, 13Patch-For-Review: Add new WMCS IP ranges to analytics - https://phabricator.wikimedia.org/T392468#10805008 (10xcollazo) Looks like we need to do this soon as it is essential work. [15:07:50] 10Cloud Services Proposals, 06cloud-services-team, 10Cloud-VPS: Decision Request - How openstack projects relate to tofu-infra - https://phabricator.wikimedia.org/T385604#10805026 (10dcaro) a:03dcaro [15:14:35] (03PS16) 10Raymond Ndibe: toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [15:19:36] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [15:19:51] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [15:20:47] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: tofuinfratest fails to create web proxy - https://phabricator.wikimedia.org/T393679#10805063 (10taavi) Looks like a Designate policy issue coming from the recent refactor: ` May 08 11:58:59 proxy-5 uwsgi-invisible-unicorn[858025]: 2025-05-08 11:58:59.863... [15:21:54] (03PS17) 10Raymond Ndibe: toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [15:30:00] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [15:30:28] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [15:30:39] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [15:31:19] 10VPS-project-Codesearch: Codesearch fails with "TypeError: Failed to fetch" - https://phabricator.wikimedia.org/T393720 (10Tgr) 03NEW [15:37:31] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) [15:38:40] (03PS18) 10Raymond Ndibe: toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [15:41:46] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [15:45:58] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [15:46:10] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [15:49:12] (03update) 10eliza189: Eliza data update [toolforge-repos/miss-search] (linkhere_branch) - 10https://gitlab.wikimedia.org/toolforge-repos/miss-search/-/merge_requests/3 [15:50:55] (03update) 10eliza189: Eliza data update [toolforge-repos/miss-search] (linkhere_branch) - 10https://gitlab.wikimedia.org/toolforge-repos/miss-search/-/merge_requests/3 [16:00:09] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) [16:02:04] (03PS19) 10Raymond Ndibe: toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [16:03:05] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30) - https://phabricator.wikimedia.org/T362869#10805181 (10dcaro) [16:05:18] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [16:05:33] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [16:07:17] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [16:07:28] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [16:09:41] (03PS20) 10Raymond Ndibe: toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [16:10:52] (03CR) 10Raymond Ndibe: "Sorry about the patch spam I'm just experimenting to find out to best way to extract the string I need from the command execution output. " [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [16:11:09] 10cloud-services-team (Hardware): [ceph] Figure out hosts placement in the racks - https://phabricator.wikimedia.org/T393699#10805194 (10dcaro) [16:11:24] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [16:11:39] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [16:11:42] 10cloud-services-team (Hardware): [ceph] Figure out hosts placement in the racks - https://phabricator.wikimedia.org/T393699#10805199 (10dcaro) The proposal is the same if we get the 2x10G setup or not, just better if we get the setup :) [16:13:16] (03CR) 10CI reject: [V:04-1] toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [16:13:29] (03PS21) 10Raymond Ndibe: toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [16:13:39] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [16:13:51] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [16:15:51] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [16:16:03] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [16:17:29] (03CR) 10CI reject: [V:04-1] toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [16:17:46] (03PS22) 10Raymond Ndibe: toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [16:19:40] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [16:21:28] 10cloud-services-team (Hardware): [ceph] Figure out hosts placement in the racks - https://phabricator.wikimedia.org/T393699#10805235 (10dcaro) 05Open→03Resolved a:03dcaro [16:22:02] (03CR) 10CI reject: [V:04-1] toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [16:22:30] 06cloud-services-team, 10Cloud-VPS, 06SRE: [cloudsw] enable 25G network - https://phabricator.wikimedia.org/T393676#10805238 (10dcaro) This is not a blocker anymore in general, will still be needed the more big hosts we get, but can wait for the general 25G everywhere some time. [16:22:49] (03PS23) 10Raymond Ndibe: toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [16:25:18] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [16:26:53] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [16:31:31] (03CR) 10Raymond Ndibe: "faced some challenges figuring out how to get the exact chart string I needed from the output (valid yaml). In the end I used a separator " [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [16:31:57] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Cloud-VPS: tofuinfratest fails to create web proxy - https://phabricator.wikimedia.org/T393679#10805274 (10Andrew) 05Open→03Resolved a:03Andrew [16:33:06] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [16:34:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-puppetdb-03 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:36:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance tools-puppetdb-2 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:38:03] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [16:46:55] !log raymond-ndibe@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-admission [16:46:59] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [16:47:14] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [16:47:52] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [16:48:04] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [16:48:25] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [16:53:10] 10cloud-services-team (Hardware): [ceph] Figure out hosts placement in the racks - https://phabricator.wikimedia.org/T393699#10805322 (10dcaro) [16:54:04] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [16:54:42] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [17:04:00] 10Toolforge (Toolforge iteration 19): [toolforge deploy] direct-api tests fail intermittently on toolsbeta - https://phabricator.wikimedia.org/T369891#10805347 (10dcaro) It might be relieved with the latest changes to the api-gateway: https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/commit/92f906... [17:04:26] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/21 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:04:28] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/21 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:05:07] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [17:07:19] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30) - https://phabricator.wikimedia.org/T362869#10805355 (10dcaro) [17:07:33] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.56-20250508170443-1a7a8bea [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/769 [17:07:43] (03approved) 10dcaro: envvars-admission: bump to 0.0.29-20250508130943-6c306f5d [repos/cloud/toolforge/toolforge-deploy] (run_specific_tests_on_deploy) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/768 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:07:46] (03merge) 10dcaro: envvars-admission: bump to 0.0.29-20250508130943-6c306f5d [repos/cloud/toolforge/toolforge-deploy] (run_specific_tests_on_deploy) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/768 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:07:48] (03update) 10dcaro: [toolforge-deploy] add test tags to component charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/755 (https://phabricator.wikimedia.org/T381011) (owner: 10raymond-ndibe) [17:08:22] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [17:10:04] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer [17:13:32] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [17:24:42] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [17:26:28] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance tools-puppetdb-2 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [17:29:28] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [17:39:23] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [17:39:29] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [17:40:11] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [17:40:15] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [17:40:23] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [17:40:28] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [17:41:50] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [17:41:58] (03PS24) 10Raymond Ndibe: toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [17:42:16] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [17:42:20] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [17:45:22] (03CR) 10CI reject: [V:04-1] toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) (owner: 10Raymond Ndibe) [17:45:27] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [17:45:31] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [17:46:50] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [17:46:54] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [17:48:38] 06cloud-services-team, 10Cloud-VPS: Horizon proxy tab is showing "Gateway Timeout" - https://phabricator.wikimedia.org/T393725 (10Mhurd) 03NEW [17:49:28] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance toolsbeta-puppetdb-03 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [17:49:43] 06cloud-services-team, 10Horizon: Horizon proxy tab is showing "Gateway Timeout" - https://phabricator.wikimedia.org/T393725#10805485 (10JJMC89) [17:57:44] (03PS25) 10Raymond Ndibe: toolforge: run specific tests on deploy [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1140246 (https://phabricator.wikimedia.org/T381011) [17:58:13] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [17:58:17] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission [18:02:07] 06cloud-services-team, 10Horizon: Horizon proxy tab is showing "Gateway Timeout" - https://phabricator.wikimedia.org/T393725#10805529 (10Andrew) 05Open→03Resolved a:03Andrew We were missing a firewall rule for ipv6 and the proxy API just now started using v6. I added: ` Ingress IPv6 TCP 5668 ::... [19:18:17] 06cloud-services-team, 10Cloud-VPS: sssd/LDAP flakiness (May 2025) - https://phabricator.wikimedia.org/T393732 (10taavi) 03NEW [19:22:13] 06cloud-services-team, 10Cloud-VPS: sssd/LDAP flakiness (May 2025) - https://phabricator.wikimedia.org/T393732#10805750 (10taavi) [19:25:26] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad: Q4:rack/setup/install clouddb102[1-4] - https://phabricator.wikimedia.org/T393733 (10RobH) 03NEW [19:25:47] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad: Q4:rack/setup/install clouddb102[1-4] - https://phabricator.wikimedia.org/T393733#10805769 (10RobH) [19:26:57] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad: Q4:rack/setup/install clouddb102[1-4] - https://phabricator.wikimedia.org/T393733#10805770 (10RobH) a:03fnegri Please note the workflow for racking tasks has changed this fiscal year, and we now require the puppet updates from the sub-team receiving... [19:57:35] 06cloud-services-team, 10Cloud-VPS: Upgrade metricsinfra Grafana hosts (grafana.wmcloud.org) to Bookworm / Grafana v11 - https://phabricator.wikimedia.org/T393735 (10taavi) 03NEW [22:22:31] FIRING: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-5 is lagging behind the primary, the current lag is 3673 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh