[00:08:57] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:18:42] RESOLVED: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:05:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [03:35:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [04:58:54] 06cloud-services-team, 07Surveys: 2025 Cloud Services Survey - https://phabricator.wikimedia.org/T411421#11780759 (10komla) [05:03:54] 06cloud-services-team, 07Surveys: Review survey summary/analyses for publication on wiki - https://phabricator.wikimedia.org/T422102 (10komla) 03NEW [05:04:16] 06cloud-services-team, 07Surveys: Review survey summary/analysis for publication on wiki - https://phabricator.wikimedia.org/T422102#11780773 (10komla) [05:04:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [05:34:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [06:52:16] 06Toolforge-standards-committee, 10Tools, 10Wikidata, 07Privacy: Tool "wd-vectordb" loads assets from Google gstatic.com - https://phabricator.wikimedia.org/T421889#11780884 (10valerio.bozzolan) 05Open→03In progress https://github.com/wmde/WikidataSearch/pull/7 [07:57:56] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 06tools-platform-team: [jobs-api] Create storage layer, and save business models in persistent storage - https://phabricator.wikimedia.org/T359650#11780981 (10dcaro) a:05Raymond_Ndibe→03dcaro [08:10:19] (03open) 10dcaro: jobs-api: make storage the source of truth [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1190 [08:27:29] (03update) 10dcaro: [cli.py] explain what --since and --until defaults to [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/149 (owner: 10raymond-ndibe) [08:28:09] (03approved) 10dcaro: [cli.py] explain what --since and --until defaults to [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/149 (owner: 10raymond-ndibe) [08:28:22] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/147 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:28:26] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/147 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:29:43] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/147 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:35:01] (03update) 10dcaro: [cli.py] explain what --since and --until defaults to [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/149 (owner: 10raymond-ndibe) [08:52:59] (03update) 10dcaro: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [08:59:44] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:12:42] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [jobs-api] Allow customizing time to request Loki logs for - https://phabricator.wikimedia.org/T400917#11781184 (10dcaro) This is available now on toolforge :) [09:28:16] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [09:37:07] (03PS1) 10David Caro: toolforge.component.deploy: improve readability of mr comment [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1266954 [09:37:43] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:37:48] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [09:39:27] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:41:00] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: improve readability of mr comment [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1266954 (owner: 10David Caro) [09:42:17] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 07Epic, 13Patch-For-Review: [jobs-api] allow exposing jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#11781294 (10dcaro) [09:42:44] 10Cloud-VPS (Quota-requests), 10Browser Test Platform, 10Continuous-Integration-Infrastructure, 10Continuous-Integration-Config, 07Jenkins: New flavor for the integration project with more vCPU and ephemeral disk space - https://phabricator.wikimedia.org/T421242#11781307 (10hashar) @taavi: @Peter and I h... [09:49:56] (03update) 10taavi: eqiad1: flavors: Add g4.cores10.ram24.disk20.ephemeral100.4xiops [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/304 (https://phabricator.wikimedia.org/T421242) [09:50:05] (03update) 10taavi: eqiad1: flavors: Add g4.cores10.ram24.disk20.ephemeral100.4xiops [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/304 (https://phabricator.wikimedia.org/T421242) [09:50:09] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [09:57:34] (03update) 10raymond-ndibe: loki.py: handle query time range error in do_follow [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/17 [09:58:46] (03PS2) 10David Caro: toolforge.component.deploy: improve readability of mr comment [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1266954 [10:00:37] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [10:03:10] (03update) 10raymond-ndibe: [cli.py] explain what --since and --until defaults to [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/149 [10:03:25] (03update) 10raymond-ndibe: [cli.py] explain what --since and --until defaults to [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/149 [10:04:02] (03update) 10raymond-ndibe: [cli.py] explain what --since and --until defaults to [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/149 [10:04:03] (03approved) 10raymond-ndibe: [cli.py] explain what --since and --until defaults to [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/149 [10:07:54] (03merge) 10raymond-ndibe: [cli.py] explain what --since and --until defaults to [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/149 [10:09:17] 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [jobs-api] Use the same images as webservice - https://phabricator.wikimedia.org/T415322#11781381 (10dcaro) [10:11:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [10:13:35] (03PS3) 10David Caro: toolforge.component.deploy: improve readability of mr comment [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1266954 [10:13:41] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [10:17:11] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/75 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:17:14] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/75 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:17:20] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/76 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:17:23] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/76 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:17:49] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/131 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:17:52] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/131 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:19:20] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/76 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [10:20:06] (03PS4) 10David Caro: toolforge.component.deploy: improve readability of mr comment [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1266954 [10:27:21] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 07Epic, 13Patch-For-Review: [jobs-api] allow exposing continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#11781533 (10dcaro) [10:27:31] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [10:27:44] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api] allow exposing continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#11781534 (10dcaro) [10:30:47] (03open) 10raymond-ndibe: d/changelog: bump to 16.1.27 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/150 [10:33:22] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics [10:33:31] !log dcaro@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component wmcs-k8s-metrics [10:33:37] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics [10:34:14] !log dcaro@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component wmcs-k8s-metrics [10:34:40] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [10:35:23] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [10:35:35] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission [10:35:59] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [10:36:12] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission [10:36:47] (03approved) 10dcaro: jobs-api: make storage the source of truth [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1190 [10:36:50] (03merge) 10dcaro: jobs-api: make storage the source of truth [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1190 [10:37:05] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [10:37:18] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission [10:38:09] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [10:38:22] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission [10:39:02] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 06tools-platform-team: [jobs-api] Create storage layer, and save business models in persistent storage - https://phabricator.wikimedia.org/T359650#11781666 (10dcaro) [10:40:13] (03open) 10dcaro: DONOTMERGE: for testing the deployment only [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1191 [10:40:32] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [10:40:48] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission [10:41:47] !log raymond-ndibe@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli [10:42:48] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [10:44:13] 10Toolforge (Toolforge iteration 26), 13Patch-For-Review: [jobs-api] Use the same images as webservice - https://phabricator.wikimedia.org/T415322#11781699 (10dcaro) [10:44:20] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [10:44:33] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission [10:47:19] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [10:47:34] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission [10:51:09] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [10:56:24] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [11:01:03] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/304 [11:01:22] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/304 [11:02:18] (03merge) 10taavi: eqiad1: flavors: Add g4.cores10.ram24.disk20.ephemeral100.4xiops [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/304 (https://phabricator.wikimedia.org/T421242) [11:02:22] !log taavi@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [11:03:21] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [11:03:59] 10Cloud-VPS (Quota-requests), 10Browser Test Platform, 10Continuous-Integration-Infrastructure, 10Continuous-Integration-Config, 07Jenkins: New flavor for the integration project with more vCPU and ephemeral disk space - https://phabricator.wikimedia.org/T421242#11781837 (10taavi) 05Open→03Resolve... [11:04:27] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [11:07:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [11:09:48] (03approved) 10raymond-ndibe: d/changelog: bump to 16.1.27 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/150 [11:10:00] (03merge) 10raymond-ndibe: d/changelog: bump to 16.1.27 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/150 [11:16:00] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [11:18:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [11:19:04] (03CR) 10David Caro: [V:03+1] "This was wrickier than expected xd" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1266954 (owner: 10David Caro) [11:26:14] (03update) 10raymond-ndibe: logs-api: add more logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [11:30:13] (03update) 10raymond-ndibe: logs-api: add more logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [11:30:46] (03update) 10raymond-ndibe: logs-api: add more logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [11:37:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [11:50:50] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [11:51:37] (03open) 10raymond-ndibe: pypi: prevent pypi deploy if commit not both tagged and merged to main [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/82 [12:01:11] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [12:03:28] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:08:18] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11782057 (10Yogesh_U_G) Hi, I have submitted my GSoC 2026 proposal for CampWiz NxT frontend migration. I would really appreciate any feedback or suggestions. Thank you! [12:09:06] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11782058 (10Yogesh_U_G) Additionally, I would be happy to start working on small frontend issues or microtasks to better understand the codebase. Please let me know if there are any b... [12:11:40] (03close) 10dcaro: DONOTMERGE: for testing the deployment only [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1191 [12:14:29] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/159 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:14:32] (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/159 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:16:30] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [12:16:50] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/159 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:17:15] 10Toolforge (Toolforge iteration 26), 06tools-platform-team, 13Patch-For-Review: [jobs-api] Use the same images as webservice - https://phabricator.wikimedia.org/T415322#11782093 (10dcaro) [12:17:42] (03approved) 10raymond-ndibe: logs-api: add more logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [12:18:24] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#11782106 (10dcaro) [12:19:11] (03update) 10andrew: Add ptr record for mail.wikimedia.az [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/303 (https://phabricator.wikimedia.org/T421025) [12:19:34] (03update) 10raymond-ndibe: logs-api: add more logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [12:19:52] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.189-20260402121659-82dca570 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1192 [12:19:57] (03update) 10raymond-ndibe: logs-api: add more logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [12:20:02] (03merge) 10raymond-ndibe: logs-api: add more logs tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1140 (https://phabricator.wikimedia.org/T418326) [12:20:04] (03update) 10raymond-ndibe: logs: test since, until params for jobs-api logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [12:20:54] (03open) 10vriaa: feat: add confirmation dialog when selecting a template with existing content [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/46 [12:21:55] (03open) 10vriaa: feat: auto-strip protocol from image URL input [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/47 [12:22:53] (03open) 10vriaa: feat: add confirmation modal before deleting a banner [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/48 [12:23:46] (03open) 10vriaa: feat: add fixed CSS support to generated banner code [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/49 [12:24:26] (03approved) 10dcaro: Add ptr record for mail.wikimedia.az [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/303 (https://phabricator.wikimedia.org/T421025) (owner: 10andrew) [12:24:47] (03open) 10vriaa: fix: use banner ID instead of template ID when generating CSS selectors [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/50 [12:31:05] (03update) 10andrew: Add ptr record for mail.wikimedia.az [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/303 (https://phabricator.wikimedia.org/T421025) [12:32:51] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/160 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:34:26] (03merge) 10andrew: Add ptr record for mail.wikimedia.az [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/303 (https://phabricator.wikimedia.org/T421025) [12:34:27] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [12:34:57] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [12:35:10] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [12:35:40] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [12:35:48] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:36:16] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [12:41:47] (03open) 10andrew: mail.wikimedia.az. -- corrected to be in parent ptr zone [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/305 (https://phabricator.wikimedia.org/T421025) [12:42:53] (03merge) 10andrew: mail.wikimedia.az. -- corrected to be in parent ptr zone [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/305 (https://phabricator.wikimedia.org/T421025) [12:43:14] (03open) 10andrew: codfw1dev testlabs quota: increased floating IP from 2 to 3 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/306 [12:43:31] (03merge) 10andrew: codfw1dev testlabs quota: increased floating IP from 2 to 3 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/306 [12:43:37] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [12:44:07] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [12:44:14] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:44:36] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [12:45:11] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:45:47] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:47:11] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 13Patch-For-Review: Add PTR record for azwikimedia - https://phabricator.wikimedia.org/T421025#11782209 (10Andrew) I'm pretty sure the issue was that tofu was removing the by-hand record and then the ip-updater adding the instance- record. I've added this... [12:48:57] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#11782210 (10dcaro) [12:50:34] 10Tools, 06LPL Onboarding and Development, 10LPL Projects (Starter kit): Host and deploy starter kit tool on Toolforge - https://phabricator.wikimedia.org/T420074#11782218 (10MaryMunyoki) [12:50:47] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 26), 06tools-platform-team, 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#11782219 (10dcaro) The `--use-deprecated-version... [12:52:28] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/160 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:52:32] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/160 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:54:48] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.190-20260402125243-4d887b53 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1192 [12:54:52] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.190-20260402125243-4d887b53 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1192 [13:00:18] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [13:04:47] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [13:06:16] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [13:11:17] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [13:35:34] (03approved) 10dcaro: components-api: bump to 0.0.190-20260402125243-4d887b53 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1192 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:35:37] (03merge) 10dcaro: components-api: bump to 0.0.190-20260402125243-4d887b53 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1192 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:38:41] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:40:27] (03CR) 10Raymond Ndibe: [C:03+2] toolforge.component.deploy: improve readability of mr comment [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1266954 (owner: 10David Caro) [13:41:51] (03update) 10raymond-ndibe: loki.py: handle query time range error in do_follow [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/17 [13:42:31] (03update) 10vriaa: feat: auto-strip protocol from image URL input [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/47 (https://phabricator.wikimedia.org/T421070) [13:43:21] (03update) 10vriaa: feat: add fixed CSS support to generated banner code [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/49 (https://phabricator.wikimedia.org/T420950) [13:43:57] (03Merged) 10jenkins-bot: toolforge.component.deploy: improve readability of mr comment [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1266954 (owner: 10David Caro) [13:44:46] (03update) 10vriaa: feat: add confirmation modal before deleting a banner [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/48 (https://phabricator.wikimedia.org/T420958) [13:45:50] (03update) 10vriaa: feat: add confirmation dialog when selecting a template with existing content [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/46 (https://phabricator.wikimedia.org/T420955) [13:47:17] 10Tool-wikimonitor, 07good first task, 07patch-welcome: Replace Dotenv with Spring @Value and update environment setup docs - https://phabricator.wikimedia.org/T421587#11782460 (10Gerges) 05Open→03Resolved @Praffq, Thanks :) [13:47:19] (03update) 10dcaro: istio-gateway: allow customizing the resources [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1189 [13:47:23] (03update) 10dcaro: istio-gateway: allow customizing the resources [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1189 [13:58:11] 06cloud-services-team, 06tools-platform-team, 07Surveys: Review survey summary/analysis for publication on wiki - https://phabricator.wikimedia.org/T422102#11782512 (10komla) [14:29:46] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/62 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:29:49] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/62 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:30:03] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/317 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:30:06] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/317 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:30:15] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/64 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:30:19] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/64 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:30:32] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/85 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:30:35] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/85 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:31:04] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/46 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:31:10] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/46 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:31:14] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/72 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:31:20] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/72 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:31:52] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/156 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:31:56] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/156 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:32:02] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/89 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:32:13] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/89 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:33:20] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/42 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:33:28] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/39 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:33:29] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-harbor: bump to 0.0.69-20260402143031-966c6168 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1193 [14:33:30] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-harbor: bump to 0.0.69-20260402143031-966c6168 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1193 [14:33:35] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/42 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:33:38] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/39 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:33:39] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/99 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:33:52] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/99 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:33:59] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/280 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:34:11] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/105 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:34:14] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/280 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:34:25] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/105 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:34:36] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/32 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:34:42] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-builder: bump to 0.0.144-20260402143052-dcd3d2fa [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1194 [14:34:44] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/32 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:34:50] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-builder: bump to 0.0.144-20260402143052-dcd3d2fa [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1194 [14:35:01] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/15 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:35:05] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/14 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:35:11] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/43 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:35:18] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/15 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:35:23] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/85 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:35:23] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/14 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:35:30] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/38 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:35:33] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/85 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:35:40] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/38 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:36:17] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [14:38:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [14:39:15] (03update) 10raymond-ndibe: loki.py: handle query time range error in do_follow [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/17 [14:39:21] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.78-20260402143429-a834ddf3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1195 [14:41:30] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [14:41:35] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-api: bump to 0.0.83-20260402143135-c3f2dcc8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1196 [14:41:43] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-api: bump to 0.0.83-20260402143135-c3f2dcc8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1196 [14:44:46] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.86-20260402143548-e87f6446 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1197 [14:44:47] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: registry-admission: bump to 0.0.74-20260402143720-ac017c44 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1198 [14:44:48] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.86-20260402143548-e87f6446 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1197 [14:44:49] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: registry-admission: bump to 0.0.74-20260402143720-ac017c44 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1198 [14:44:55] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.211-20260402143212-0de993af [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1199 [14:44:59] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.211-20260402143212-0de993af [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1199 [14:45:25] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-kubeusers: bump to 0.0.193-20260402143819-02b2a899 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1200 [14:45:26] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-kubeusers: bump to 0.0.193-20260402143819-02b2a899 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1200 [14:45:27] (03update) 10raymond-ndibe: loki.py: handle query time range error in do_follow [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/17 [14:45:40] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [14:45:58] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.83-20260402143124-e97e4504 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1201 [14:45:59] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.83-20260402143124-e97e4504 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1201 [14:46:49] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.90-20260402143226-2e084eb0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1202 [14:46:50] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.90-20260402143226-2e084eb0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1202 [14:48:28] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: logs-api: bump to 0.0.17-20260402143549-d3a41687 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1203 [14:48:30] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: logs-api: bump to 0.0.17-20260402143549-d3a41687 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1203 [14:49:41] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.480-20260402143448-7ac6018a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1204 [14:49:42] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.480-20260402143448-7ac6018a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1204 [14:49:45] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-admission: bump to 0.0.39-20260402143521-2183085c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1205 [14:49:47] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-admission: bump to 0.0.39-20260402143521-2183085c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1205 [14:54:13] (03update) 10raymond-ndibe: loki.py: handle query time range error in do_follow [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/17 [15:03:11] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/43 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:03:22] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/43 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:03:41] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/16 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:04:14] (03approved) 10dcaro: maintain-harbor: bump to 0.0.69-20260402143031-966c6168 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1193 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:04:19] (03merge) 10dcaro: maintain-harbor: bump to 0.0.69-20260402143031-966c6168 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1193 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:04:28] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [15:04:33] (03update) 10dcaro: envvars-admission: bump to 0.0.39-20260402143521-2183085c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1205 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:05:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [15:07:11] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.79-20260402150412-71b0f8df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1195 [15:07:18] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.79-20260402150412-71b0f8df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1195 [15:08:25] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#11782911 (10Priyanjali05) Thank you for sharing the subtask! I will start exploring it and try to understand the current implementation and requirements. I will share my progress and... [15:11:03] 10Tools, 06LPL Onboarding and Development, 10LPL Projects (Starter kit): Host and deploy starter kit tool on Toolforge - https://phabricator.wikimedia.org/T420074#11782924 (10MaryMunyoki) [15:12:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [15:16:18] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [15:25:13] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [15:31:56] (03approved) 10dcaro: envvars-admission: bump to 0.0.39-20260402143521-2183085c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1205 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:31:59] (03merge) 10dcaro: envvars-admission: bump to 0.0.39-20260402143521-2183085c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1205 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:32:20] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:34:14] (03approved) 10dcaro: Add --diff-mode and remove --dry-run [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/10 (owner: 10fnegri) [15:35:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [15:43:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:45:08] 10Toolforge (Toolforge iteration 26), 06tools-platform-team: [ci,builds-cli] job to update dependencies breaks when trying builds-cli - https://phabricator.wikimedia.org/T422018#11783069 (10dcaro) p:05Triage→03Medium [15:46:26] (03open) 10dcaro: build: upgrading deps manually [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/132 [15:46:46] (03update) 10dcaro: build: upgrading deps manually [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/132 [15:46:55] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [16:00:52] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [16:02:43] (03update) 10raymond-ndibe: logs: test since, until params for jobs-api logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [16:05:02] (03approved) 10dcaro: jobs-api: bump to 0.0.480-20260402143448-7ac6018a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1204 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:05:05] (03update) 10dcaro: jobs-api: bump to 0.0.480-20260402143448-7ac6018a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1204 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:05:40] (03merge) 10dcaro: jobs-api: bump to 0.0.480-20260402143448-7ac6018a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1204 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:06:32] (03update) 10dcaro: logs-api: bump to 0.0.17-20260402143549-d3a41687 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1203 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:06:33] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [16:07:51] (03update) 10raymond-ndibe: logs: test since, until params for jobs-api logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [16:13:34] (03update) 10dcaro: build: upgrading deps manually [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/132 [16:15:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [16:15:18] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [16:16:43] (03update) 10dcaro: build: upgrading deps manually [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/132 [16:17:08] (03update) 10dcaro: build: upgrading deps manually [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/132 [16:17:09] (03approved) 10dcaro: build: upgrading deps manually [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/132 [16:17:12] (03merge) 10dcaro: build: upgrading deps manually [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/132 [16:18:56] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/16 [16:18:58] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/16 [16:19:12] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/77 [16:19:15] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/77 [16:19:20] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/162 [16:19:26] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/162 [16:19:34] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/44 [16:19:42] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/44 [16:19:49] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/281 [16:20:02] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/281 [16:20:04] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/106 [16:20:08] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/106 [16:20:27] 06cloud-services-team, 06tools-platform-team, 07Surveys: Review survey summary/analysis for publication on wiki - https://phabricator.wikimedia.org/T422102#11783216 (10komla) 05Open→03Resolved [16:20:27] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/90 [16:20:28] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/90 [16:20:28] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/63 [16:20:30] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/63 [16:24:25] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, 13Patch-For-Review: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11783238 (10Dzahn) Thanks @A_smart_kitten I merged and deployed the change and have confirmed there... [16:24:29] 10Toolforge (Toolforge iteration 26), 06tools-platform-team: [ci,builds-cli] job to update dependencies breaks when trying builds-cli - https://phabricator.wikimedia.org/T422018#11783240 (10dcaro) a:03dcaro [16:25:02] 10Toolforge (Toolforge iteration 26), 06tools-platform-team: [general] upgrade all python repos to python >=3.13 - https://phabricator.wikimedia.org/T422184 (10dcaro) 03NEW [16:25:08] 10Toolforge (Toolforge iteration 26), 06tools-platform-team: [general] upgrade all python repos to python >=3.13 - https://phabricator.wikimedia.org/T422184#11783258 (10dcaro) [16:26:08] 10Toolforge (Toolforge iteration 26), 06tools-platform-team: [ci,builds-cli] job to update dependencies breaks when trying builds-cli - https://phabricator.wikimedia.org/T422018#11783280 (10dcaro) I had to manually upgrade cffi by unpinning the deps to python 3.9, so it would get a new enough cffi. [16:26:14] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [16:26:39] 10Toolforge (Toolforge iteration 26), 06tools-platform-team: [ci,builds-cli] job to update dependencies breaks when trying builds-cli - https://phabricator.wikimedia.org/T422018#11783285 (10dcaro) 05Open→03Resolved [16:26:44] (03approved) 10dcaro: logs-api: bump to 0.0.17-20260402143549-d3a41687 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1203 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:27:19] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [16:30:51] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [16:32:01] (03update) 10dcaro: loki.py: handle query time range error in do_follow [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/17 (owner: 10raymond-ndibe) [16:32:12] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [16:36:17] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [16:36:54] (03approved) 10dcaro: api-gateway: bump to 0.0.90-20260402143226-2e084eb0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1202 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:36:54] (03merge) 10dcaro: api-gateway: bump to 0.0.90-20260402143226-2e084eb0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1202 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:38:54] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [16:41:45] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [16:47:46] 10Toolforge (Toolforge iteration 26), 06tools-platform-team: [global] First of the month automatic dependency upgrade - https://phabricator.wikimedia.org/T422191 (10dcaro) 03NEW [16:47:55] 10Toolforge (Toolforge iteration 26), 06tools-platform-team: [global] First of the month automatic dependency upgrade - https://phabricator.wikimedia.org/T422191#11783456 (10dcaro) p:05Triage→03Medium [16:47:56] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [16:50:54] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [16:51:00] (03update) 10dcaro: istio-gateway: allow customizing the resources [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1189 [16:51:06] (03update) 10dcaro: test for jobs-api backend in webservice-cli [repos/cloud/toolforge/toolforge-deploy] (test_continuous_job_webservice) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1134 (https://phabricator.wikimedia.org/T348755) (owner: 10raymond-ndibe) [16:51:45] (03update) 10dcaro: jobs-api: use webservice image variants in one-off job tests [repos/cloud/toolforge/toolforge-deploy] (test_for_image_argument_handling_in_jobs) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1115 (https://phabricator.wikimedia.org/T415322) (owner: 10raymond-ndibe) [16:51:49] 10Tool-wikimonitor: 30-Minute SseEmitter Timeout with Event Resumption - https://phabricator.wikimedia.org/T422194 (10Gerges) 03NEW [16:51:58] (03update) 10dcaro: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] (use_webservice_image_variants_in_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) (owner: 10raymond-ndibe) [16:52:14] (03approved) 10dcaro: volume-admission: bump to 0.0.83-20260402143124-e97e4504 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1201 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:52:18] (03update) 10dcaro: volume-admission: bump to 0.0.83-20260402143124-e97e4504 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1201 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:52:39] (03update) 10dcaro: maintain-kubeusers: bump to 0.0.193-20260402143819-02b2a899 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1200 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:52:40] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [16:52:45] (03merge) 10dcaro: volume-admission: bump to 0.0.83-20260402143124-e97e4504 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1201 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:53:17] (03close) 10dcaro: logging: alloy: Allow running on the entire cluster [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/836 (https://phabricator.wikimedia.org/T386480) (owner: 10taavi) [16:53:20] 10Tool-wikimonitor: 30-Minute SseEmitter Timeout with Event Resumption - https://phabricator.wikimedia.org/T422194#11783522 (10Gerges) p:05Triage→03High [16:53:39] (03close) 10dcaro: logging: alloy: Add routing for infrastructure logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/835 (https://phabricator.wikimedia.org/T386480) (owner: 10taavi) [16:53:46] (03close) 10dcaro: logging: loki: Add second Loki instance for infrastructure logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/834 (https://phabricator.wikimedia.org/T386480) (owner: 10taavi) [17:00:01] 10Tool-wikimonitor: 30-Minute SseEmitter Timeout with Event Resumption - https://phabricator.wikimedia.org/T422194#11783582 (10Gerges) [17:00:22] (03update) 10dcaro: maintain-kubeusers: bump to 0.0.193-20260402143819-02b2a899 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1200 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:05:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [17:08:38] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [17:10:22] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [17:19:33] (03close) 10raymond-ndibe: logs: test since, until params for jobs-api logs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1141 (https://phabricator.wikimedia.org/T400917) [17:29:32] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [17:34:17] (03approved) 10dcaro: maintain-kubeusers: bump to 0.0.193-20260402143819-02b2a899 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1200 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:34:24] (03merge) 10dcaro: maintain-kubeusers: bump to 0.0.193-20260402143819-02b2a899 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1200 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:34:40] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11783777 (10Dzahn) Noticed the puppet change was not applied on the test instance. Found out the local puppetmaster (puppe... [17:35:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [17:38:56] FIRING: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:41:49] (03approved) 10dcaro: loki.py: handle query time range error in do_follow [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/17 (owner: 10raymond-ndibe) [17:44:20] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11783814 (10Dzahn) I tried to do the scap deploy: ` debug1: Server host key: ... Host key verification failed. 17:42:34 c... [17:45:15] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11783816 (10Dzahn) @brennen Could you do a scap deploy to the test instance? See above how I tried to Rollback but "Y" was... [17:49:48] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11783834 (10brennen) > @brennen Could you do a scap deploy to the test instance? See above how I tried to Rollback but "Y"... [17:50:10] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11783835 (10brennen) ` == DEFAULT == :* phabricator-bullseye.devtools.eqiad1.wikimedia.cloud 17:48:54 Running remote deploy... [17:55:56] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11783879 (10Dzahn) Thank you, @brennen I see "mail_default_address: phabricator-no-reply@wmcloud.org" in `/etc/phabricato... [18:28:19] 06cloud-services-team, 10Data-Services, 10Toolforge: fiwiki_p.imagelinks view is broken on Toolforge replica (ERROR 1356), while enwiki_p works - https://phabricator.wikimedia.org/T422206 (10Zache) 03NEW [18:29:52] 06cloud-services-team, 10Data-Services: fiwiki_p.imagelinks view is broken on Toolforge replica (ERROR 1356), while enwiki_p works - https://phabricator.wikimedia.org/T422206#11784021 (10JJMC89) This is due to {T419635}. [18:54:11] (03update) 10danyya: Draft: Optimisation and date stats [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/1 [18:55:00] (03update) 10danyya: Optimisation and date stats [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/1 [18:55:22] (03merge) 10danyya: Optimisation and date stats [toolforge-repos/humaniki] - 10https://gitlab.wikimedia.org/toolforge-repos/humaniki/-/merge_requests/1 [19:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 5 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:39:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [20:14:11] (03approved) 10aghirelli: Fix typo [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/13 (owner: 10apaskulin) [20:14:16] (03merge) 10aghirelli: Fix typo [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/13 (owner: 10apaskulin) [20:31:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [20:36:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [21:07:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [21:14:56] 10Tool-wikimonitor: 30-Minute SseEmitter Timeout with Event Resumption - https://phabricator.wikimedia.org/T422194#11784423 (10Praffq) a:03Praffq I'll like to try to solve it [21:37:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-8:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [21:41:24] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11784455 (10A_smart_kitten) Thank you for reviewing/merging (& fixing the puppetmaster issue) @dzahn, & thank you for doing... [22:08:35] 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11784516 (10dancy) >>! In T388022#11783814, @Dzahn wrote: > I tried to do the scap deploy: > > ` > debug1: Server host key... [22:11:41] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/23924337776 (https://github.com/cluebotng/component-configs/commits/266152f9f673810b0c9460b5828cb86e7aee31d9) [22:11:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [22:21:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [23:01:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [23:05:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [23:06:55] FIRING: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity [23:35:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [23:41:55] RESOLVED: ToolforgeKubernetesCapacity: Kubernetes cluster k8s.tools.eqiad1.wikimedia.cloud:6443 in risk of running out of memory - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesCapacity - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesCapacity