[05:49:23] How to disconnect unrecognizable objects [06:34:08] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/59 [06:34:16] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/129 [06:34:25] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/125 [06:34:29] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/60 [06:34:39] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/125 [06:34:39] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/129 [06:34:42] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/73 [06:34:45] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/60 [06:34:55] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/27 [06:35:14] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/73 [06:35:21] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/27 [06:35:31] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/130 [06:35:33] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/23 [06:35:46] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/130 [06:35:47] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/23 [06:35:54] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/119 [06:35:55] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/29 [06:36:01] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/29 [06:36:04] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/30 [06:36:10] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/120 [06:36:12] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/30 [06:36:17] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/119 [06:36:45] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/206 [06:36:50] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/120 [06:36:50] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/90 [06:36:50] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/207 [06:36:51] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/91 [06:37:07] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/25 [06:37:15] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/206 [06:37:16] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/33 [06:37:17] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/77 [06:37:21] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/936 [06:37:22] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/51 [06:37:22] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/207 [06:37:26] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/62 [06:37:29] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/90 [06:37:31] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/91 [06:37:35] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/139 [06:37:39] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/25 [06:37:43] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/78 [06:37:47] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/16 [06:37:51] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/77 [06:37:55] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/33 [06:37:59] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/73 [06:38:03] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/270 [06:38:08] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/936 [06:38:11] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/51 [06:38:15] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/62 [06:38:20] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/52 [06:38:24] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/11 [06:38:28] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/52 [06:38:32] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/139 [06:38:36] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/78 [06:38:40] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/16 [06:38:44] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/73 [06:38:48] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/270 [06:38:52] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/52 [06:38:56] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/11 [06:39:00] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/52 [08:08:39] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11135048 (10dcaro) [08:26:07] !log dcaro@acme toolsbeta START - Cookbook wmcs.toolforge.run_tests [08:26:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [08:26:15] !log dcaro@acme toolsbeta END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97) [08:26:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [08:26:51] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [08:28:50] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11135130 (10dcaro) [08:32:10] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.29.15 to 1.30.14 (T402377) [08:32:15] T402377: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377 [08:32:30] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.29.15 to 1.30.14 (T402377) [08:34:51] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11135173 (10dcaro) [08:35:56] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 (T402377) [08:38:18] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [08:38:20] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [08:39:09] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 (T402377) [08:39:13] T402377: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377 [08:40:36] !log filippo@cloudcumin1001 wlm-it-visual START - Cookbook wmcs.openstack.quota_increase (T403165) [08:40:39] T403165: Quota increase for Wiki Loves Monuments app - https://phabricator.wikimedia.org/T403165 [08:40:43] !log filippo@cloudcumin1001 wlm-it-visual END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T403165) [08:41:15] 10Cloud-VPS (Quota-requests), 10WLM-Italy (WLM-Italy-WebApp): Quota increase for Wiki Loves Monuments app - https://phabricator.wikimedia.org/T403165#11135212 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi {{done}} per https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Projects_lifecycle#Mod... [08:42:56] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 (T402377) [08:42:58] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 (T402377) [08:47:57] 10Cloud-VPS (Quota-requests), 10Content-Transform-Team (Work In Progress): Quote increase request for wikitextexp - https://phabricator.wikimedia.org/T403114#11135308 (10fnegri) +1 [08:51:40] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11135330 (10dcaro) The upgrade of the first control node failed when uncordoning: ` 2025-09-01 08:39:01,749 dcaro 750609 [INFO cookbooks.wmcs.toolforge.k8s.... [08:51:49] !log filippo@cloudcumin1001 wikitextexp START - Cookbook wmcs.openstack.quota_increase (T403114) [08:51:53] T403114: Quote increase request for wikitextexp - https://phabricator.wikimedia.org/T403114 [08:51:56] !log filippo@cloudcumin1001 wikitextexp END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T403114) [08:55:08] !log filippo@cloudcumin1001 wlm-it-visual START - Cookbook wmcs.openstack.quota_increase (T403165) [08:55:12] T403165: Quota increase for Wiki Loves Monuments app - https://phabricator.wikimedia.org/T403165 [08:55:15] !log filippo@cloudcumin1001 wlm-it-visual END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T403165) [08:55:26] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11135368 (10dcaro) The apiserver was unhealthy for a second there while starting on the new version: ` │ Name: kube-apiserver-toolsbeta-test-k8s... [08:55:42] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [08:55:48] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [08:56:01] !log filippo@cloudcumin1001 wikitextexp START - Cookbook wmcs.openstack.quota_increase (T403114) [08:56:08] !log filippo@cloudcumin1001 wikitextexp END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T403114) [08:56:23] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.29.15 to 1.30.14 (T402377) [08:56:27] T402377: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377 [08:56:42] 10Cloud-VPS (Quota-requests), 10Content-Transform-Team (Work In Progress): Quote increase request for wikitextexp - https://phabricator.wikimedia.org/T403114#11135384 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi {{done}} [09:03:43] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [09:05:25] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [09:06:17] dcaro@cloudcumin1001 upgrade (PID 753333) is awaiting input [09:06:48] 06cloud-services-team: PuppetFailure Puppet has failed on clouddumps1001:9100 - https://phabricator.wikimedia.org/T403083#11135435 (10fnegri) 05Open→03Resolved a:03fnegri [09:06:49] 06cloud-services-team: PuppetFailure - https://phabricator.wikimedia.org/T403087#11135437 (10fnegri) 05Open→03Resolved a:03fnegri [09:07:30] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.29.15 to 1.30.14 (T402377) [09:07:35] T402377: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377 [09:08:18] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.29.15 to 1.30.14 (T402377) [09:08:25] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11135442 (10dcaro) [09:13:25] FIRING: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: toolsbeta-test-k8s-control-12.toolsbeta.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [09:15:27] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.29.15 to 1.30.14 (T402377) [09:15:32] T402377: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377 [09:18:25] RESOLVED: ToolforgeKubernetesHAproxyServerDown: Toolforge HAproxy server down: toolsbeta-test-k8s-control-12.toolsbeta.eqiad1.wikimedia.cloud - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesHAproxyServerDown - https://grafana.wmcloud.org/d/toolforge-k8s-haproxy/toolforge-k8s-haproxy?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesHAproxyServerDown [09:20:38] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11135485 (10dcaro) [09:22:22] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Evaluate higher level signals for nova troubles rather than paging on nova-compute down - https://phabricator.wikimedia.org/T402778#11135491 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This is done [09:22:50] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [09:22:56] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [09:26:23] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11135505 (10dcaro) [09:28:15] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade tools to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402378#11135508 (10dcaro) [09:29:12] 06cloud-services-team, 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30) - https://phabricator.wikimedia.org/T362869#11135512 (10dcaro) [09:30:11] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.29.15 to 1.30.14 (T402377) [09:30:16] T402377: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377 [09:31:10] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.29.15 to 1.30.14 (T402377) [09:34:06] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.29.15 to 1.30.14 (T402377) [09:34:40] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11135545 (10dcaro) [09:35:04] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.29.15 to 1.30.14 (T402377) [09:37:57] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 (T402377) [09:38:02] T402377: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377 [09:38:13] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 (T402377) [09:38:39] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11135556 (10dcaro) [09:39:09] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [09:39:15] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [09:39:38] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 (T402377) [09:39:47] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11135559 (10dcaro) One of the workers failed to upgrade because puppet was already running: ` ----- OUTPUT of 'sudo -i run-puppet-agent' -----... [09:40:22] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 (T402377) [09:41:11] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.29.15 to 1.30.14 (T402377) [09:42:09] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.29.15 to 1.30.14 (T402377) [09:44:42] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.29.15 to 1.30.14 (T402377) [09:44:47] T402377: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377 [09:45:38] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [09:45:45] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.29.15 to 1.30.14 (T402377) [09:46:15] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.29.15 to 1.30.14 (T402377) [09:47:10] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.29.15 to 1.30.14 (T402377) [09:47:32] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 (T402377) [09:48:24] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11135585 (10dcaro) [09:48:32] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 (T402377) [09:50:36] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [09:55:29] (03PS1) 10David Caro: k8s.prepare_upgrade: remove not used src-version param [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183620 [10:05:54] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [10:06:00] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [10:06:11] (03PS1) 10David Caro: k8s.worker.upgrade: allow not passing src_version [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183623 [10:09:36] (03CR) 10CI reject: [V:04-1] k8s.worker.upgrade: allow not passing src_version [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183623 (owner: 10David Caro) [10:22:22] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [10:22:28] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [10:39:03] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [10:39:09] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [10:52:35] (03CR) 10Nikerabbit: WIP Merge lego footer messages into one (031 comment) [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1003574 (https://phabricator.wikimedia.org/T355011) (owner: 10Amire80) [10:55:16] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [10:55:22] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [11:11:34] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [11:11:40] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [11:28:59] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [11:29:06] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [11:34:36] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade [11:35:05] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) [11:40:17] (03PS2) 10David Caro: k8s.worker.upgrade: allow not passing src_version [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183623 [11:40:23] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 [11:40:23] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 [11:40:35] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 [11:40:36] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 [11:41:04] (03PS3) 10David Caro: k8s.worker.upgrade: allow not passing src_version [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183623 [11:41:10] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 [11:41:10] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 [11:45:37] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [11:45:43] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [11:46:55] 06cloud-services-team, 10Ceph: Investigate big spikes up in wmcs ceph dashboards - https://phabricator.wikimedia.org/T403390 (10fgiunchedi) 03NEW [11:48:33] 06cloud-services-team, 10Ceph: Investigate big spikes up in wmcs ceph dashboards - https://phabricator.wikimedia.org/T403390#11135996 (10fgiunchedi) [11:52:03] 06cloud-services-team, 10Ceph: Investigate big spikes up in wmcs ceph dashboards - https://phabricator.wikimedia.org/T403390#11136015 (10fgiunchedi) Something worth noting from https://docs.ceph.com/en/latest/mgr/prometheus/ is that `scrape_interval` is set to 15s, whereas in production the effective scrape in... [12:01:55] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [12:02:01] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [12:14:23] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 [12:14:24] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 [12:18:24] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [12:18:30] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [12:30:03] 06cloud-services-team, 10Ceph: Investigate big spikes up in wmcs ceph dashboards - https://phabricator.wikimedia.org/T403390#11136184 (10fgiunchedi) [12:35:46] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [12:35:52] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [12:40:53] (03PS1) 10David Caro: k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) [12:43:19] (03PS2) 10David Caro: k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) [12:45:10] (03PS3) 10David Caro: k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) [12:45:15] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers no stuck workers found [12:45:15] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) no stuck workers found [12:45:58] (03PS4) 10David Caro: k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) [12:46:01] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers no stuck workers found [12:46:01] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) no stuck workers found [12:46:15] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 [12:47:23] (03PS5) 10David Caro: k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) [12:47:25] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 [12:47:33] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 [12:48:33] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/13 [12:49:33] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 [12:49:34] (03PS6) 10David Caro: k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) [12:49:43] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 [12:49:49] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 [12:50:20] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1183665 (owner: 10L10n-bot) [12:52:11] (03PS7) 10David Caro: k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) [12:52:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [12:52:18] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [12:52:50] (03PS8) 10David Caro: k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) [12:53:45] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 [12:54:31] (03PS9) 10David Caro: k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) [12:54:32] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 [12:54:38] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 [12:55:29] (03PS10) 10David Caro: k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) [12:55:29] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 [12:55:34] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 [12:55:39] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 [12:55:54] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 [12:55:58] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 [12:56:04] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 [12:56:49] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 [12:56:53] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 [12:57:08] (03PS11) 10David Caro: k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) [12:57:09] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 [12:57:14] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 [12:57:31] (03merge) 10fnegri: Add pre-commit and ruff linter [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/6 [12:57:33] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 [12:57:34] (03update) 10fnegri: Remove build_deb script, use gitlab-ci-local [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/7 [12:57:40] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 [12:57:44] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 [12:57:50] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 [12:57:52] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 [12:58:03] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 [12:58:12] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 [12:58:44] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-ingress-9, toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 [12:58:48] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-ingress-9, toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s [12:58:48] -worker-13 [12:58:57] (03merge) 10fnegri: Remove build_deb script, use gitlab-ci-local [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/7 [12:59:02] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-ingress-9, toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11 [12:59:04] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-ingress-9, toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11 [13:00:28] 06cloud-services-team, 10Ceph: Investigate big spikes up in wmcs ceph dashboards - https://phabricator.wikimedia.org/T403390#11136328 (10dcaro) >>! In T403390#11136015, @fgiunchedi wrote: > Something worth noting from https://docs.ceph.com/en/latest/mgr/prometheus/ is that `scrape_interval` is set to 15s, wher... [13:00:47] (03CR) 10CI reject: [V:04-1] k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) (owner: 10David Caro) [13:01:15] 06cloud-services-team, 10Ceph: Investigate big spikes up in wmcs ceph dashboards - https://phabricator.wikimedia.org/T403390#11136329 (10fgiunchedi) re: case 2, mgr was restarted, apparently manually, and IIRC it was during the ceph outage ` Aug 28 15:50:20 cloudcephmon1004 ceph-mgr[318826]: 2025-08-28T15:50:... [13:02:02] 06cloud-services-team, 10Ceph: Investigate big spikes up in wmcs ceph dashboards - https://phabricator.wikimedia.org/T403390#11136330 (10dcaro) That's actually a relief :) [13:09:38] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [13:09:44] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [13:13:56] 06cloud-services-team, 10Cloud-VPS, 10Toolforge, 10Observability-Alerting, and 2 others: Move WMCS off of Icinga and introduce alertmanager - https://phabricator.wikimedia.org/T328502#11136366 (10fgiunchedi) [13:26:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [13:26:19] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [13:34:32] (03PS12) 10David Caro: k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) [13:34:32] (03PS1) 10David Caro: k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183682 (https://phabricator.wikimedia.org/T402377) [13:35:13] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 [13:35:14] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 [13:36:22] (03PS2) 10David Caro: k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183682 (https://phabricator.wikimedia.org/T402377) [13:36:41] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 [13:38:49] (03CR) 10CI reject: [V:04-1] k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) (owner: 10David Caro) [13:39:34] (03PS13) 10David Caro: k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) [13:39:41] dcaro@cloudcumin1001 upgrade_ingresses (PID 790121) is awaiting input [13:39:54] (03PS3) 10David Caro: k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183682 (https://phabricator.wikimedia.org/T402377) [13:40:34] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 [13:40:41] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 [13:41:00] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 [13:41:19] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 (T402377) [13:41:23] T402377: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377 [13:41:49] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 (T402377) [13:43:23] (03PS4) 10David Caro: k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183682 (https://phabricator.wikimedia.org/T402377) [13:43:33] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 (T402377) [13:43:44] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [13:43:50] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [13:45:04] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [13:45:10] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [13:47:54] !log dcaro@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=97) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 (T402377) [13:47:59] T402377: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377 [13:48:44] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 (T402377) [13:49:35] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 (T402377) [13:51:12] (03PS14) 10David Caro: k8s.worker.upgrade_workers: add a cookbook to upgrade a batch [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183657 (https://phabricator.wikimedia.org/T402377) [13:51:12] (03PS5) 10David Caro: k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183682 (https://phabricator.wikimedia.org/T402377) [13:51:51] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 (T402377) [13:52:00] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 (T402377) [13:54:06] 06cloud-services-team, 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11136532 (10dcaro) [14:01:26] (03PS6) 10David Caro: k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183682 (https://phabricator.wikimedia.org/T402377) [14:02:37] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [14:02:43] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [14:04:45] (03CR) 10CI reject: [V:04-1] k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183682 (https://phabricator.wikimedia.org/T402377) (owner: 10David Caro) [14:12:57] (03PS7) 10David Caro: k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183682 (https://phabricator.wikimedia.org/T402377) [14:12:57] (03PS1) 10David Caro: k8s.worker.upgrade_bastions: add cookbook to upgrade bastions [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183697 (https://phabricator.wikimedia.org/T402377) [14:16:17] (03PS2) 10David Caro: k8s.worker.upgrade_bastions: add cookbook to upgrade bastions [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183697 (https://phabricator.wikimedia.org/T402377) [14:16:19] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T402377) [14:16:23] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=99) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T402377) [14:16:25] T402377: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377 [14:16:35] (03CR) 10CI reject: [V:04-1] k8s.worker.upgrade_bastions: add cookbook to upgrade bastions [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183697 (https://phabricator.wikimedia.org/T402377) (owner: 10David Caro) [14:16:42] (03PS3) 10David Caro: k8s.worker.upgrade_bastions: add cookbook to upgrade bastions [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183697 (https://phabricator.wikimedia.org/T402377) [14:16:44] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T402377) [14:16:54] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=99) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T402377) [14:19:48] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [14:19:54] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [14:20:03] (03CR) 10CI reject: [V:04-1] k8s.worker.upgrade_bastions: add cookbook to upgrade bastions [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183697 (https://phabricator.wikimedia.org/T402377) (owner: 10David Caro) [14:21:39] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T402377) [14:21:43] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T402377) [14:21:43] T402377: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377 [14:22:38] (03PS4) 10David Caro: k8s.worker.upgrade_bastions: add cookbook to upgrade bastions [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183697 (https://phabricator.wikimedia.org/T402377) [14:22:42] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T402377) [14:22:49] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud (T402377) [14:23:51] 06cloud-services-team, 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11136626 (10dcaro) [14:24:30] 06cloud-services-team, 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11136627 (10dcaro) [14:24:52] 06cloud-services-team, 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11136628 (10dcaro) Used the cookbooks to upgrade too, will update the docs once they are reviewed [14:25:14] 06cloud-services-team, 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [k8s,infra] Upgrade toolsbeta to Uwubernetes 1.30 - https://phabricator.wikimedia.org/T402377#11136629 (10dcaro) 05In progress→03Resolved [14:25:48] (03CR) 10CI reject: [V:04-1] k8s.worker.upgrade_bastions: add cookbook to upgrade bastions [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183697 (https://phabricator.wikimedia.org/T402377) (owner: 10David Caro) [14:26:59] 06cloud-services-team, 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [k8s,infra] Upgrade Toolforge to Uwubernetes (1.30) - https://phabricator.wikimedia.org/T362869#11136632 (10dcaro) [14:29:26] (03CR) 10Filippo Giunchedi: "LGTM overall, see inline" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183682 (https://phabricator.wikimedia.org/T402377) (owner: 10David Caro) [14:29:35] (03PS5) 10David Caro: k8s.worker.upgrade_bastions: add cookbook to upgrade bastions [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183697 (https://phabricator.wikimedia.org/T402377) [14:31:27] 06cloud-services-team, 10Ceph: Investigate big spikes up in wmcs ceph dashboards - https://phabricator.wikimedia.org/T403390#11136652 (10fgiunchedi) p:05Triage→03Medium [14:31:49] 06cloud-services-team, 10Toolforge: [builds-api] return image digest - https://phabricator.wikimedia.org/T403322#11136653 (10fgiunchedi) p:05Triage→03Medium [14:31:57] (03PS6) 10David Caro: k8s.worker.upgrade_bastions: add cookbook to upgrade bastions [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183697 (https://phabricator.wikimedia.org/T402377) [14:32:03] 06cloud-services-team, 10Toolforge: [components-api] restart rather than delete/create continuous jobs - https://phabricator.wikimedia.org/T403321#11136654 (10fgiunchedi) p:05Triage→03Medium [14:32:19] 06cloud-services-team, 10Toolforge, 07Upstream: Build service tools built with --use-latest-versions are missing /etc/mime.types - https://phabricator.wikimedia.org/T403303#11136655 (10fgiunchedi) p:05Triage→03Medium [14:32:30] 06cloud-services-team, 10Toolforge: `webservice shell` loses output when not run with a pty - https://phabricator.wikimedia.org/T403295#11136656 (10fgiunchedi) p:05Triage→03Medium [14:32:41] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [components-api] Validate `SourceReference` components point to `SourceBuild` components - https://phabricator.wikimedia.org/T403287#11136660 (10fgiunchedi) p:05Triage→03Medium [14:32:53] 06cloud-services-team, 10Toolforge: `webservice shell` sometimes loses output - https://phabricator.wikimedia.org/T403286#11136661 (10fgiunchedi) p:05Triage→03Medium [14:36:09] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [14:36:15] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [14:36:18] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [components-api] reuse_from components are not explicitly re-created in jobs-api - https://phabricator.wikimedia.org/T403285#11136662 (10fgiunchedi) p:05Triage→03Medium [14:36:27] 06cloud-services-team, 10Toolforge: [components-api] Intermittent internal API failures / retry internal requests - https://phabricator.wikimedia.org/T403175#11136663 (10fgiunchedi) p:05Triage→03Medium [14:36:40] 06cloud-services-team, 10Toolforge: [bulids-api] Figure out how to handle better build retention - https://phabricator.wikimedia.org/T403172#11136664 (10fgiunchedi) p:05Triage→03Medium [14:37:44] 06cloud-services-team, 10Toolforge: [webservice] returning default 404 even though webservice is healthy - https://phabricator.wikimedia.org/T403168#11136665 (10fgiunchedi) p:05Triage→03Medium [14:38:46] 06cloud-services-team, 10Toolforge: [components-api] rebuilds un-changed images - https://phabricator.wikimedia.org/T403167#11136667 (10fgiunchedi) p:05Triage→03Medium [14:38:58] 06cloud-services-team, 10Cloud-VPS: Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#11136669 (10fgiunchedi) p:05Triage→03Medium [14:39:14] 06cloud-services-team, 10Cloud-VPS: Upgrade cloudcumin hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403153#11136671 (10fgiunchedi) p:05Triage→03Medium [14:40:01] 06cloud-services-team, 10Horizon, 07Upstream: Horizon: Selected server groups do not get cleared after deleting them - https://phabricator.wikimedia.org/T403026#11136672 (10fgiunchedi) p:05Triage→03Medium [14:40:07] 06cloud-services-team, 10Toolforge: [components-api] split source from config - https://phabricator.wikimedia.org/T402790#11136673 (10fgiunchedi) p:05Triage→03Medium [14:40:18] 06cloud-services-team, 10Toolforge: toolforge logs appears to suffer from intermittent latency - https://phabricator.wikimedia.org/T402736#11136674 (10fgiunchedi) p:05Triage→03Medium [14:40:29] 06cloud-services-team, 10Toolforge: `toolforge build start` returns success status on build failure - https://phabricator.wikimedia.org/T402648#11136675 (10fgiunchedi) p:05Triage→03Medium [14:40:42] 06cloud-services-team, 10Toolforge: [toolforge-deploy] detect deployed cli versions in lima-kilo - https://phabricator.wikimedia.org/T401649#11136676 (10fgiunchedi) p:05Triage→03Medium [14:53:52] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [14:53:58] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [14:56:59] (03CR) 10David Caro: k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses (032 comments) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183682 (https://phabricator.wikimedia.org/T402377) (owner: 10David Caro) [15:10:06] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [15:10:12] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:27:42] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [15:27:49] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:41:21] (03PS8) 10David Caro: k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183682 (https://phabricator.wikimedia.org/T402377) [15:41:21] (03CR) 10David Caro: k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183682 (https://phabricator.wikimedia.org/T402377) (owner: 10David Caro) [15:41:21] (03PS7) 10David Caro: k8s.worker.upgrade_bastions: add cookbook to upgrade bastions [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183697 (https://phabricator.wikimedia.org/T402377) [15:45:01] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [15:45:07] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:51:57] 10Tools: Qrank: Cloud object storage size reached - https://phabricator.wikimedia.org/T403409 (10dcaro) 03NEW [15:57:37] 06cloud-services-team, 10Cloud-VPS: Mwoffliner: cloud vps object storage size quota over 90% - https://phabricator.wikimedia.org/T403410 (10dcaro) 03NEW [15:58:16] 10VPS-Projects: Mwoffliner: cloud vps object storage size quota over 90% - https://phabricator.wikimedia.org/T403410#11136918 (10dcaro) [15:59:52] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/52 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:59:55] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/52 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:00:21] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/11 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:00:26] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/wmcs-k8s-metrics] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/wmcs-k8s-metrics/-/merge_requests/11 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:00:33] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/52 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:00:36] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/52 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:00:49] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/270 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:00:53] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/270 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:01:02] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/73 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:01:05] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/73 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:01:14] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/16 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:01:16] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/16 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:01:24] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [16:01:26] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/78 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:01:30] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/78 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:01:30] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [16:01:46] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/139 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:01:54] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: wmcs-k8s-metrics: bump to 0.0.22-20250901160038-e95a0711 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/937 [16:01:55] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/139 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:02:01] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/62 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:02:09] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/62 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:02:18] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/51 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:02:20] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/936 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:02:23] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/936 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:02:34] (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/77 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:03:12] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/125 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:03:18] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/125 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:03:21] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/129 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:03:25] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/129 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:04:09] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/27 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:04:13] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/27 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:04:18] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/23 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:04:27] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/23 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:04:29] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/119 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:04:42] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/119 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:04:44] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/29 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:04:45] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-builder: bump to 0.0.133-20250901160118-b35bfe0a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/938 [16:04:49] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-builder: bump to 0.0.133-20250901160118-b35bfe0a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/938 [16:05:02] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/29 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:05:09] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/206 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:05:10] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-harbor: bump to 0.0.59-20250901160048-fc2cd60d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/939 [16:05:10] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: calico: bump to 0.0.16-20250901160128-176866d0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/940 [16:05:12] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-harbor: bump to 0.0.59-20250901160048-fc2cd60d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/939 [16:05:13] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: calico: bump to 0.0.16-20250901160128-176866d0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/940 [16:05:15] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/206 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:05:21] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/91 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:05:24] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/25 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:05:29] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/91 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:05:32] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/25 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:05:33] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/33 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:05:43] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/33 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:06:01] (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/59 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:06:12] (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/59 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:06:22] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.76-20250901160147-447fa428 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/941 [16:06:23] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: api-gateway: bump to 0.0.76-20250901160147-447fa428 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/941 [16:07:53] 10VPS-Projects: Mwoffliner: cloud vps object storage size quota over 90% - https://phabricator.wikimedia.org/T403410#11136937 (10Benoit74) @Audiodude do you have any idea what this object storage is used for? Looks like DB backups from Oct 1 2024, but these are not supposed to be there and they never got updated... [16:08:24] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.154-20250901160354-3fed2259 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/942 [16:08:27] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.154-20250901160354-3fed2259 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/942 [16:08:39] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.65-20250901160549-72180a79 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/943 [16:08:42] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.65-20250901160549-72180a79 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/943 [16:09:42] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.409-20250901160553-6c62fc4d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/945 [16:09:45] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: registry-admission: bump to 0.0.64-20250901160442-a6e4617f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/944 [16:09:47] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.409-20250901160553-6c62fc4d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/945 [16:10:18] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-admission: bump to 0.0.32-20250901160601-7995b656 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/946 [16:11:19] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.70-20250901160605-cdddc2d4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/947 [16:11:28] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.63-20250901160502-2b19eee4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/948 [16:11:32] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.63-20250901160502-2b19eee4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/948 [16:11:41] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-api: bump to 0.0.73-20250901160226-77b2b741 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/949 [16:11:44] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-api: bump to 0.0.73-20250901160226-77b2b741 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/949 [16:13:10] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.197-20250901160312-09fbc959 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/950 [16:13:12] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.197-20250901160312-09fbc959 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/950 [16:18:57] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [16:19:01] 10VPS-Projects: Mwoffliner: cloud vps object storage size quota over 90% - https://phabricator.wikimedia.org/T403410#11136974 (10Audiodude) You're right, it's for database backups originating from Horizon. You're also right about the date it was enabled. Of course, since we have our own database backups, these a... [16:19:03] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [16:28:37] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#11137030 (10fgiunchedi) I gave this a stab, and got blocked on bigtop / hadoop dependency; specifically I'm not sure if/what bigtop + java combinations are supported [16:29:01] (03CR) 10Filippo Giunchedi: [C:03+1] k8s.worker.upgrade_ingresses: add cookbook to upgrade ingresses (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1183682 (https://phabricator.wikimedia.org/T402377) (owner: 10David Caro) [16:35:27] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [16:35:33] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [16:53:09] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [16:53:16] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [16:54:53] (03open) 10dcaro: gitlab: move to bookworm image [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/23 [16:55:38] (03update) 10dcaro: gitlab: move to bookworm image [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/23 [16:57:52] (03open) 10dcaro: gitlab: move to bookworm images [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/15 [16:59:23] (03open) 10dcaro: gitlab: move to bookworm images [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/8 [17:00:03] (03open) 10dcaro: gitlab: move to bookworm images [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/86 [17:00:27] (03update) 10dcaro: gitlab: move to bookworm image [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/23 [17:03:45] (03open) 10dcaro: gitlab: move to bookworm images [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/14 [17:04:09] (03approved) 10dcaro: gitlab: move to bookworm images [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/14 [17:04:12] (03merge) 10dcaro: gitlab: move to bookworm images [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/14 [17:05:38] (03open) 10dcaro: gitlab: move to bookworm images [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/7 [17:05:47] (03update) 10dcaro: gitlab: move to bookworm images [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/7 [17:06:28] (03approved) 10dcaro: gitlab: move to bookworm images [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/7 [17:06:30] (03merge) 10dcaro: gitlab: move to bookworm images [toolforge-repos/k8s-status] - 10https://gitlab.wikimedia.org/toolforge-repos/k8s-status/-/merge_requests/7 [17:07:46] (03open) 10dcaro: gitlab: move to bookworm images [toolforge-repos/toolstats] - 10https://gitlab.wikimedia.org/toolforge-repos/toolstats/-/merge_requests/4 [17:09:18] (03update) 10dcaro: gitlab: move to bookworm images [toolforge-repos/toolstats] - 10https://gitlab.wikimedia.org/toolforge-repos/toolstats/-/merge_requests/4 [17:09:31] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [17:09:37] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [17:10:48] (03open) 10dcaro: gitlab: move to bookworm images [toolforge-repos/toolviews] - 10https://gitlab.wikimedia.org/toolforge-repos/toolviews/-/merge_requests/12 [17:11:44] (03update) 10dcaro: gitlab: move to bookworm images [toolforge-repos/toolviews] - 10https://gitlab.wikimedia.org/toolforge-repos/toolviews/-/merge_requests/12 [17:14:36] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#11137151 (10fnegri) Also be mindful of {T391369} [17:14:53] 10cloud-services-team (FY2025/26-Q1), 10Cloud-VPS, 10Toolforge: If the inactive clouddumps host goes down, it causes a ripple effect on Cloud VPS and Toolforge - https://phabricator.wikimedia.org/T391369#11137155 (10fnegri) [17:14:57] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#11137154 (10fnegri) [17:16:23] (03approved) 10fnegri: gitlab: move to bookworm image [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/23 (owner: 10dcaro) [17:16:50] (03merge) 10dcaro: gitlab: move to bookworm image [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/23 [17:27:02] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [17:27:09] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [17:43:18] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [17:43:24] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [18:00:42] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [18:00:48] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [18:18:02] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [18:18:08] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [18:35:38] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [18:35:44] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [18:39:28] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/13 (owner: 10l10n-bot) [18:39:30] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/lexeme-forms] - 10https://gitlab.wikimedia.org/toolforge-repos/lexeme-forms/-/merge_requests/13 (owner: 10l10n-bot) [18:52:18] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [18:52:25] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [19:00:41] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Upgrade clouddumps hosts to bookworm/trixie - https://phabricator.wikimedia.org/T403154#11137332 (10MoritzMuehlenhoff) >>! In T403154#11137030, @fgiunchedi wrote: > I gave this a stab, and got blocked on bigtop / hadoop dependency; specifically I'm not... [19:10:02] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [19:10:08] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [19:11:18] (03open) 10raymond-ndibe: Draft: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [19:59:30] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [19:59:36] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [20:16:07] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [20:16:13] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [20:31:34] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [20:31:40] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [20:48:13] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [20:48:20] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [21:05:39] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [21:05:45] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [21:17:06] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [22:34:10] (03CR) 10Krinkle: [C:03+2] build: Commit missing .phan/config.php file (031 comment) [labs/tools/wikiinfo] - 10https://gerrit.wikimedia.org/r/1173454 (owner: 10Krinkle) [23:02:13] 10VPS-project-Codesearch: Codesearch down/unreachable (2025-09-02) - https://phabricator.wikimedia.org/T403434 (10SomeRandomDeveloper) 03NEW