[00:00:56] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:26:10] FIRING: MaxConntrack: Max conntrack at 82.12% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:31:10] RESOLVED: MaxConntrack: Max conntrack at 81.08% on cloudvirt1050:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [06:50:56] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:00:56] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:26:20] (03update) 10dcaro: db: add database check and boileplate [toolforge-repos/sample-complex-app-backend] (add_worker) - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/2 (https://phabricator.wikimedia.org/T370317) [08:30:37] (03update) 10dcaro: db: add database check and boileplate [toolforge-repos/sample-complex-app-backend] (add_worker) - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/2 (https://phabricator.wikimedia.org/T370317) [08:31:37] (03merge) 10dcaro: worker: add simple task and worker process [toolforge-repos/sample-complex-app-backend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/1 (https://phabricator.wikimedia.org/T370321) [08:31:38] (03update) 10dcaro: db: add database check and boileplate [toolforge-repos/sample-complex-app-backend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/2 (https://phabricator.wikimedia.org/T370317) [08:33:07] (03merge) 10dcaro: db: add database check and boileplate [toolforge-repos/sample-complex-app-backend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-backend/-/merge_requests/2 (https://phabricator.wikimedia.org/T370317) [08:34:50] (03update) 10dcaro: show task status [toolforge-repos/sample-complex-app-frontend] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/2 (https://phabricator.wikimedia.org/T370321) [08:37:04] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/29 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:37:09] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/29 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:37:39] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/187 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:37:43] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/187 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:38:01] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/28 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:38:09] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/61 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:38:12] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/61 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:38:20] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/88 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:38:23] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/88 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:38:28] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/67 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:38:32] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/67 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:38:57] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/60 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:39:04] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/87 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:39:29] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/60 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:39:38] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/87 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:39:42] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/28 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:39:45] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/28 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:40:32] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/60 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:40:39] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/87 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:43:32] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/7 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:43:38] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/7 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:46:07] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: components-api: bump to 0.0.15-20240902084355-d4743502 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/495 [08:48:02] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [08:48:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:48:44] !log dcaro@urcuchillay tools END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api [08:48:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:48:49] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [08:48:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [08:49:34] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/8 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:49:42] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/8 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:50:13] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/33 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:50:16] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/33 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:52:14] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: components-api: bump to 0.0.15-20240902084355-d4743502 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/495 [08:55:28] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [08:55:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:00:07] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [09:00:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:05:26] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/39 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:05:31] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/39 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:05:31] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/40 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:05:42] (03update) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/40 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:06:01] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [09:06:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:06:22] (03approved) 10dcaro: components-api: bump to 0.0.15-20240902084355-d4743502 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/495 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:06:25] (03merge) 10dcaro: components-api: bump to 0.0.15-20240902084355-d4743502 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/495 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:07:18] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/40 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:07:43] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.42-20240902090543-ea887a72 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/496 [09:07:46] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.42-20240902090543-ea887a72 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/496 [09:09:16] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.42-20240902090543-ea887a72 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/496 [09:09:44] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [09:09:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:14:28] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 14), 13Patch-For-Review: toolforge: prometheus server died - https://phabricator.wikimedia.org/T370143#10109595 (10fnegri) 05Open→03In progress [09:14:50] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [09:14:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:16:13] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [09:16:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:17:02] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/110 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:17:06] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/110 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:21:41] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [09:21:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:21:55] (03approved) 10dcaro: api-gateway: bump to 0.0.42-20240902090543-ea887a72 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/496 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:21:59] (03merge) 10dcaro: api-gateway: bump to 0.0.42-20240902090543-ea887a72 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/496 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:22:08] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/13 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:22:13] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/13 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:24:55] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-api: bump to 0.0.169-20240902091723-606731f0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/497 [09:25:47] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: registry-admission: bump to 0.0.49-20240902092221-86679dbf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/498 [09:27:58] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [09:28:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:33:45] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: create a cookbook automation to run tofu - https://phabricator.wikimedia.org/T370414#10109653 (10aborrero) 05In progress→03Resolved [09:34:20] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [09:34:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:39:29] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance cloudinfra-idp-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [09:43:17] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission [09:43:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:47:00] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643#10109681 (10dcaro) >>! In T348643#10104246, @wiki_willy wrote: > Hi @dcaro - just following up on this to see if you... [09:48:58] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission [09:49:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:50:53] (03approved) 10dcaro: registry-admission: bump to 0.0.49-20240902092221-86679dbf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/498 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:50:55] (03merge) 10dcaro: registry-admission: bump to 0.0.49-20240902092221-86679dbf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/498 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:51:09] (03update) 10dcaro: builds-api: bump to 0.0.169-20240902091723-606731f0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/497 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:51:16] !log dcaro@urcuchillay toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [09:51:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:57:23] !log dcaro@urcuchillay toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [09:57:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:58:44] !log dcaro@urcuchillay tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [09:58:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:05:18] !log dcaro@urcuchillay tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [10:05:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:08:36] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [10:08:48] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [10:09:01] (03approved) 10fnegri: local: remove requests/limits from k8s [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/494 (owner: 10dcaro) [10:15:14] (03PS1) 10Arturo Borrero Gonzalez: Revert "openstack.tofu: use run_script instead of reimplementing it" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1069987 [10:15:35] (03approved) 10fnegri: lima-vm: use 16 cpus and remove workers [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/186 (owner: 10dcaro) [10:15:42] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [10:15:53] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [10:18:49] (03PS2) 10Arturo Borrero Gonzalez: Revert "openstack.tofu: use run_script instead of reimplementing it" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1069987 [10:25:37] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [10:25:39] (03update) 10aborrero: tofu-infra: add projects [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 (https://phabricator.wikimedia.org/T371393) [10:26:07] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [10:48:08] (03CR) 10Arturo Borrero Gonzalez: "recheck" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1069987 (owner: 10Arturo Borrero Gonzalez) [10:54:02] (03merge) 10aborrero: tofu-infra: add projects [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 (https://phabricator.wikimedia.org/T371393) [10:54:09] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [10:54:41] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [10:55:28] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [10:56:42] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [10:57:01] 06cloud-services-team, 10Cloud-VPS, 07Epic: tofu-infra: investigate S3 spurious endpoint errors - https://phabricator.wikimedia.org/T370660#10109834 (10aborrero) had this again today. [11:07:16] (03PS1) 10Arturo Borrero Gonzalez: wmcs.vps.create_project: replace logic with message about deprecation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1069994 (https://phabricator.wikimedia.org/T371393) [11:26:05] (03PS3) 10Arturo Borrero Gonzalez: Revert "openstack.tofu: use run_script instead of reimplementing it" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1069987 [11:27:35] (03approved) 10dcaro: builds-api: bump to 0.0.169-20240902091723-606731f0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/497 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:27:38] (03merge) 10dcaro: builds-api: bump to 0.0.169-20240902091723-606731f0 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/497 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [11:28:59] (03CR) 10David Caro: [C:03+1] "LGTM, though we should remove the tech debt this re-introduces as soon as feasible" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1069987 (owner: 10Arturo Borrero Gonzalez) [11:30:04] (03approved) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/53 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [11:30:08] (03merge) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/53 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [11:30:08] (03update) 10dcaro: pre-commit: Autoupdate [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/53 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [11:36:58] !log dcaro@urcuchillay admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [11:36:58] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch [11:37:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:37:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:37:30] !log dcaro@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [11:37:49] !log dcaro@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [11:42:15] !log dcaro@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [11:42:40] !log dcaro@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [11:43:46] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.25.16 to 1.26.15 [11:45:09] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.25.16 to 1.26.15 [11:45:10] 06cloud-services-team, 10Toolforge (Toolforge iteration 14): [infra,k8s] Upgrade Tools to k8s version 1.26 - https://phabricator.wikimedia.org/T370249#10109970 (10Slst2020) 05Open→03In progress [11:46:48] (03open) 10dcaro: DONOTMERGE: silly test [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/29 [11:46:57] !log dcaro@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/29 [11:47:25] !log dcaro@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/29 [11:48:00] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.25.16 to 1.26.15 [11:50:38] (03CR) 10David Caro: [C:03+1] "I can't reproduce the issue though, can you open a task with a reproducer?" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1069987 (owner: 10Arturo Borrero Gonzalez) [11:59:57] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.25.16 to 1.26.15 [12:00:28] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.25.16 to 1.26.15 [12:11:16] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.25.16 to 1.26.15 [12:12:16] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.25.16 to 1.26.15 [12:12:40] (03CR) 10CI reject: [V:04-1] wmcs.vps.create_project: replace logic with message about deprecation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1069994 (https://phabricator.wikimedia.org/T371393) (owner: 10Arturo Borrero Gonzalez) [12:16:08] (03PS2) 10Arturo Borrero Gonzalez: wmcs.vps.create_project: replace logic with message about deprecation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1069994 (https://phabricator.wikimedia.org/T371393) [12:16:43] (03update) 10raymond-ndibe: [jobs-emailer] add topologySpreadConstraints to deployment [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/6 (https://phabricator.wikimedia.org/T358203) [12:17:07] (03update) 10raymond-ndibe: [builds-builder] add topologySpreadConstraints to deployment [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/58 (https://phabricator.wikimedia.org/T358203) [12:18:46] (03CR) 10FNegri: [C:03+1] wmcs.vps.create_project: replace logic with message about deprecation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1069994 (https://phabricator.wikimedia.org/T371393) (owner: 10Arturo Borrero Gonzalez) [12:21:24] (03CR) 10Raymond Ndibe: [C:03+1] "LGTM. merging next" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1067911 (owner: 10David Caro) [12:21:27] (03CR) 10Raymond Ndibe: [C:03+2] docs: add wmcs.yaml to the generated config with default values [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1067911 (owner: 10David Caro) [12:22:27] (03update) 10raymond-ndibe: [registry-admission] add topologySpreadConstraints to deployment [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/11 (https://phabricator.wikimedia.org/T358203) [12:22:36] (03approved) 10raymond-ndibe: [jobs-emailer] add topologySpreadConstraints to deployment [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/6 (https://phabricator.wikimedia.org/T358203) [12:22:41] (03merge) 10raymond-ndibe: [jobs-emailer] add topologySpreadConstraints to deployment [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/6 (https://phabricator.wikimedia.org/T358203) [12:22:48] (03approved) 10raymond-ndibe: [ingress-admission] add topologySpreadConstraints to deployment [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/8 (https://phabricator.wikimedia.org/T358203) [12:22:54] (03merge) 10raymond-ndibe: [ingress-admission] add topologySpreadConstraints to deployment [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/8 (https://phabricator.wikimedia.org/T358203) [12:23:03] (03update) 10raymond-ndibe: [builds-builder] add topologySpreadConstraints to deployment [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/58 (https://phabricator.wikimedia.org/T358203) [12:23:15] (03update) 10raymond-ndibe: [jobs-api] add topologySpreadConstraints to deployment [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/117 (https://phabricator.wikimedia.org/T358203) [12:23:39] (03approved) 10raymond-ndibe: [volume-admission] add topologySpreadConstraints to deployment [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/15 (https://phabricator.wikimedia.org/T358203) [12:23:46] (03merge) 10raymond-ndibe: [volume-admission] add topologySpreadConstraints to deployment [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/15 (https://phabricator.wikimedia.org/T358203) [12:23:50] (03approved) 10raymond-ndibe: [envvars-admission] add topologySpreadConstraints to deployment [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/10 (https://phabricator.wikimedia.org/T358203) [12:23:55] (03merge) 10raymond-ndibe: [envvars-admission] add topologySpreadConstraints to deployment [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/10 (https://phabricator.wikimedia.org/T358203) [12:24:04] (03update) 10raymond-ndibe: [api-gateway] add topologySpreadConstraints to deployment [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/35 (https://phabricator.wikimedia.org/T358203) [12:24:31] (03CR) 10Arturo Borrero Gonzalez: [C:03+2] Revert "openstack.tofu: use run_script instead of reimplementing it" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1069987 (owner: 10Arturo Borrero Gonzalez) [12:24:32] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.25.16 to 1.26.15 [12:24:47] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15 [12:24:51] (03PS1) 10David Caro: with_temporary_file: don't fail if the temp file does not exist [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070018 [12:25:14] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-emailer: bump to 0.0.43-20240902122251-1677ec5f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/499 (https://phabricator.wikimedia.org/T358203) [12:26:07] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15 [12:26:38] (03CR) 10Arturo Borrero Gonzalez: "recheck" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1069994 (https://phabricator.wikimedia.org/T371393) (owner: 10Arturo Borrero Gonzalez) [12:26:47] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: ingress-admission: bump to 0.0.49-20240902122319-b0b4ce16 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/500 (https://phabricator.wikimedia.org/T358203) [12:26:47] (03update) 10raymond-ndibe: [jobs-api] multi-replica support for continuous jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/115 (https://phabricator.wikimedia.org/T341066) [12:26:50] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: ingress-admission: bump to 0.0.49-20240902122319-b0b4ce16 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/500 (https://phabricator.wikimedia.org/T358203) [12:27:43] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: volume-admission: bump to 0.0.54-20240902122354-18500992 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/501 (https://phabricator.wikimedia.org/T358203) [12:27:58] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15 [12:28:33] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-admission: bump to 0.0.17-20240902122423-94921cad [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/502 (https://phabricator.wikimedia.org/T358203) [12:28:54] (03approved) 10raymond-ndibe: [registry-admission] add topologySpreadConstraints to deployment [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/11 (https://phabricator.wikimedia.org/T358203) [12:28:59] (03merge) 10raymond-ndibe: [registry-admission] add topologySpreadConstraints to deployment [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/11 (https://phabricator.wikimedia.org/T358203) [12:29:05] (03approved) 10raymond-ndibe: [builds-builder] add topologySpreadConstraints to deployment [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/58 (https://phabricator.wikimedia.org/T358203) [12:29:10] (03merge) 10raymond-ndibe: [builds-builder] add topologySpreadConstraints to deployment [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/58 (https://phabricator.wikimedia.org/T358203) [12:29:13] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15 [12:29:15] (03approved) 10raymond-ndibe: [jobs-api] add topologySpreadConstraints to deployment [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/117 (https://phabricator.wikimedia.org/T358203) [12:29:21] (03merge) 10raymond-ndibe: [jobs-api] add topologySpreadConstraints to deployment [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/117 (https://phabricator.wikimedia.org/T358203) [12:29:29] (03approved) 10raymond-ndibe: [api-gateway] add topologySpreadConstraints to deployment [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/35 (https://phabricator.wikimedia.org/T358203) [12:29:34] (03merge) 10raymond-ndibe: [api-gateway] add topologySpreadConstraints to deployment [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/35 (https://phabricator.wikimedia.org/T358203) [12:29:48] (03approved) 10raymond-ndibe: [envvars-api] add topologySpreadConstraints to deployment [repos/cloud/toolforge/envvars-api] (node-selector-to-test-topology-constraints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T358203) [12:30:34] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-builder: bump to 0.0.117-20240902122920-3b0529c7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/503 (https://phabricator.wikimedia.org/T358203) [12:30:42] (03Merged) 10jenkins-bot: docs: add wmcs.yaml to the generated config with default values [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1067911 (owner: 10David Caro) [12:30:51] (03Merged) 10jenkins-bot: Revert "openstack.tofu: use run_script instead of reimplementing it" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1069987 (owner: 10Arturo Borrero Gonzalez) [12:31:22] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15 [12:31:33] (03CR) 10CI reject: [V:04-1] wmcs.vps.create_project: replace logic with message about deprecation [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1069994 (https://phabricator.wikimedia.org/T371393) (owner: 10Arturo Borrero Gonzalez) [12:31:37] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.44-20240902122941-a7ca33e8 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/504 (https://phabricator.wikimedia.org/T358203) [12:31:37] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: registry-admission: bump to 0.0.50-20240902122909-507a2f0d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/505 (https://phabricator.wikimedia.org/T358203) [12:31:58] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.332-20240902122934-464b67dd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/506 (https://phabricator.wikimedia.org/T358203) [12:32:30] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15 [12:32:41] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15 [12:33:25] (03PS1) 10David Caro: Revert^2 "openstack.tofu: use run_script instead of reimplementing it" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070020 [12:33:55] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15 [12:34:06] (03PS2) 10David Caro: with_temporary_file: don't fail if the temp file does not exist [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070018 [12:34:18] (03PS3) 10David Caro: with_temporary_file: don't fail if the temp file does not exist [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070018 [12:34:20] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15 [12:35:34] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15 [12:40:28] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15 [12:40:38] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [12:40:43] !log aborrero@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [12:40:49] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [12:41:14] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [12:41:47] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15 [12:42:20] (03update) 10raymond-ndibe: [envvars-api] add topologySpreadConstraints to deployment [repos/cloud/toolforge/envvars-api] (node-selector-to-test-topology-constraints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T358203) [12:42:52] (03update) 10raymond-ndibe: [envvars-api] add topologySpreadConstraints to deployment [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T358203) [12:43:14] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15 [12:45:12] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15 [12:45:53] (03CR) 10Arturo Borrero Gonzalez: [C:04-1] "This needs further testing. We still need to prove this is better than the previous code." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070020 (owner: 10David Caro) [12:45:58] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15 [12:47:06] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15 [12:47:43] (03update) 10raymond-ndibe: [envvars-api] add topologySpreadConstraints to deployment [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T358203) [12:47:50] (03merge) 10raymond-ndibe: [envvars-api] add topologySpreadConstraints to deployment [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/46 (https://phabricator.wikimedia.org/T358203) [12:48:03] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [12:48:43] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [12:51:25] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-api: bump to 0.0.60-20240902124757-342c4931 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/507 (https://phabricator.wikimedia.org/T358203) [12:51:49] (03update) 10raymond-ndibe: [builds-api] add topologySpreadConstraints to deployment [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/82 (https://phabricator.wikimedia.org/T358203) [12:52:19] (03update) 10raymond-ndibe: [builds-api] add topologySpreadConstraints to deployment [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/82 (https://phabricator.wikimedia.org/T358203) [12:52:50] (03update) 10raymond-ndibe: [builds-api] add topologySpreadConstraints to deployment [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/82 (https://phabricator.wikimedia.org/T358203) [12:54:00] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 (T370249) [12:54:00] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15 [12:54:04] T370249: [infra,k8s] Upgrade Tools to k8s version 1.26 - https://phabricator.wikimedia.org/T370249 [12:55:13] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 (T370249) [12:55:14] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 (T370249) [12:55:18] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15 [12:56:15] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15 [12:56:26] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 (T370249) [12:56:27] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 (T370249) [12:57:13] (03merge) 10raymond-ndibe: [builds-api] add topologySpreadConstraints to deployment [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/82 (https://phabricator.wikimedia.org/T358203) [12:57:30] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15 [12:57:48] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 (T370249) [12:57:49] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 (T370249) [12:58:34] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15 [12:59:18] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 (T370249) [12:59:19] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 (T370249) [12:59:22] T370249: [infra,k8s] Upgrade Tools to k8s version 1.26 - https://phabricator.wikimedia.org/T370249 [12:59:47] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15 [13:00:07] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15 [13:00:32] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 (T370249) [13:00:33] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 (T370249) [13:01:21] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15 [13:01:40] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 (T370249) [13:01:41] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 (T370249) [13:01:50] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15 [13:02:56] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 (T370249) [13:02:57] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 (T370249) [13:03:02] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15 [13:04:05] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 (T370249) [13:04:06] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 (T370249) [13:04:48] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15 [13:05:17] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 (T370249) [13:05:18] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 (T370249) [13:05:19] T370249: [infra,k8s] Upgrade Tools to k8s version 1.26 - https://phabricator.wikimedia.org/T370249 [13:05:56] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15 [13:06:08] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15 [13:06:23] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 (T370249) [13:06:24] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 (T370249) [13:06:28] !log dcaro@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [13:07:06] !log dcaro@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [13:07:20] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15 [13:07:28] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15 [13:07:30] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 (T370249) [13:07:31] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 (T370249) [13:08:42] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15 [13:08:45] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 (T370249) [13:08:46] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 (T370249) [13:08:59] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15 [13:09:52] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 (T370249) [13:09:53] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 (T370249) [13:10:06] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15 [13:11:04] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 (T370249) [13:11:05] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 (T370249) [13:11:07] T370249: [infra,k8s] Upgrade Tools to k8s version 1.26 - https://phabricator.wikimedia.org/T370249 [13:11:08] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15 [13:12:12] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-api: bump to 0.0.170-20240902125720-e7b15322 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/508 (https://phabricator.wikimedia.org/T358203) [13:12:15] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 (T370249) [13:12:16] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 (T370249) [13:12:22] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15 [13:13:09] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15 [13:13:24] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 (T370249) [13:13:25] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 (T370249) [13:14:24] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15 [13:14:38] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 (T370249) [13:14:39] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 (T370249) [13:15:42] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15 [13:15:51] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 (T370249) [13:15:52] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 (T370249) [13:16:55] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15 [13:17:02] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 (T370249) [13:17:03] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 (T370249) [13:17:04] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 (T370249) [13:17:05] T370249: [infra,k8s] Upgrade Tools to k8s version 1.26 - https://phabricator.wikimedia.org/T370249 [13:17:47] (03PS2) 10David Caro: Revert^2 "openstack.tofu: use run_script instead of reimplementing it" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070020 [13:18:55] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 (T370249) [13:19:27] !log dcaro@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [13:19:51] !log dcaro@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [13:20:08] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 (T370249) [13:20:09] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 (T370249) [13:20:31] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15 [13:20:56] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:21:16] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 (T370249) [13:21:17] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 (T370249) [13:21:22] (03CR) 10CI reject: [V:04-1] Revert^2 "openstack.tofu: use run_script instead of reimplementing it" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070020 (owner: 10David Caro) [13:21:47] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15 [13:22:29] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 (T370249) [13:22:31] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 (T370249) [13:22:31] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15 [13:22:32] T370249: [infra,k8s] Upgrade Tools to k8s version 1.26 - https://phabricator.wikimedia.org/T370249 [13:23:03] (03PS3) 10David Caro: Revert^2 "openstack.tofu: use run_script instead of reimplementing it" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070020 [13:23:25] !log dcaro@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [13:23:43] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15 [13:23:46] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 (T370249) [13:23:47] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 (T370249) [13:23:54] !log dcaro@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [13:24:32] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15 [13:24:52] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 (T370249) [13:24:53] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 (T370249) [13:24:55] T370249: [infra,k8s] Upgrade Tools to k8s version 1.26 - https://phabricator.wikimedia.org/T370249 [13:25:46] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15 [13:25:59] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 (T370249) [13:26:00] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 (T370249) [13:26:42] (03CR) 10CI reject: [V:04-1] Revert^2 "openstack.tofu: use run_script instead of reimplementing it" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070020 (owner: 10David Caro) [13:27:09] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 (T370249) [13:27:10] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 (T370249) [13:27:26] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15 [13:27:31] !log dcaro@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [13:27:57] !log dcaro@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28 [13:28:17] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 (T370249) [13:28:18] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 (T370249) [13:28:34] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15 [13:29:00] (03PS4) 10David Caro: Revert^2 "openstack.tofu: use run_script instead of reimplementing it" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070020 [13:29:18] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15 [13:29:26] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 (T370249) [13:29:27] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 (T370249) [13:30:15] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 (T370249) [13:30:19] T370249: [infra,k8s] Upgrade Tools to k8s version 1.26 - https://phabricator.wikimedia.org/T370249 [13:30:29] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15 [13:30:55] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15 [13:30:56] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:32:02] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15 [13:32:08] (03CR) 10David Caro: [V:03+1] "Tested with plan (https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/28#note_103048), ping me whenever you hav" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070020 (owner: 10David Caro) [13:32:31] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15 [13:32:49] (03CR) 10CI reject: [V:04-1] Revert^2 "openstack.tofu: use run_script instead of reimplementing it" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070020 (owner: 10David Caro) [13:33:50] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15 [14:10:37] FIRING: [2x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [14:11:51] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Quarry: Support queries against Quarry's own database and ToolsDB - https://phabricator.wikimedia.org/T151158#10110532 (10fnegri) 05In progress→03Resolved I have updated all the documentation I could find, let me know if I missed anything: * https://meta.wi... [14:15:37] RESOLVED: [2x] HAProxyBackendUnavailable: HAProxy service keystone-admin-api_backend backend cloudcontrol1005.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [14:16:49] (03PS5) 10David Caro: Revert^2 "openstack.tofu: use run_script instead of reimplementing it" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1070020 [14:17:10] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 (T370249) [14:17:14] T370249: [infra,k8s] Upgrade Tools to k8s version 1.26 - https://phabricator.wikimedia.org/T370249 [14:17:55] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 (T370249) [14:19:06] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 (T370249) [14:20:10] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 (T370249) [14:24:53] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 (T370249) [14:24:57] T370249: [infra,k8s] Upgrade Tools to k8s version 1.26 - https://phabricator.wikimedia.org/T370249 [14:26:02] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 (T370249) [14:26:03] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 (T370249) [14:27:22] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 (T370249) [14:27:23] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 (T370249) [14:28:30] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 (T370249) [14:28:31] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 (T370249) [14:29:36] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 (T370249) [14:29:37] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 (T370249) [14:30:52] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 (T370249) [14:30:52] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 (T370249) [14:30:55] T370249: [infra,k8s] Upgrade Tools to k8s version 1.26 - https://phabricator.wikimedia.org/T370249 [14:31:57] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 (T370249) [14:50:56] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:05:56] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:58:03] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [16:54:12] (03approved) 10dcaro: [jobs-cli] remove unknown keys from dump [repos/cloud/toolforge/jobs-cli] (update_autocomplete_and_man_files) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/64 (https://phabricator.wikimedia.org/T341066) (owner: 10raymond-ndibe) [17:02:50] (03approved) 10dcaro: [jobs-cli] update autocomplete and man files [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/66 (owner: 10raymond-ndibe) [17:04:27] (03approved) 10dcaro: [jobs-api] multi-replica support for continuous jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/115 (https://phabricator.wikimedia.org/T341066) (owner: 10raymond-ndibe) [17:04:48] (03update) 10dcaro: [jobs-api] multi-replica support for continuous jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/115 (https://phabricator.wikimedia.org/T341066) (owner: 10raymond-ndibe) [17:11:48] Ok [17:14:34] (03update) 10dcaro: local: remove requests/limits from k8s [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/494 [17:15:00] (03merge) 10dcaro: local: remove requests/limits from k8s [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/494 [17:17:08] (03open) 10dcaro: registry-admission.tools: leave default limits [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/509 [17:17:22] (03approved) 10dcaro: registry-admission.tools: leave default limits [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/509 [17:17:26] (03merge) 10dcaro: registry-admission.tools: leave default limits [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/509 [17:17:39] (03update) 10dcaro: lima-vm: use 16 cpus and remove workers [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/186 [17:18:36] (03merge) 10dcaro: lima-vm: use 16 cpus and remove workers [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/186 [17:22:45] (03open) 10dcaro: k8s: upgrade to 1.27.16 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/188 [17:24:25] (03update) 10dcaro: k8s: upgrade to 1.27.16 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/188 (https://phabricator.wikimedia.org/T359641) [17:24:33] (03update) 10dcaro: k8s: upgrade to 1.27.16 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/188 (https://phabricator.wikimedia.org/T359641) [17:26:11] 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.27 - https://phabricator.wikimedia.org/T359641#10111076 (10dcaro) [18:50:56] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:00:56] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:44:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:49:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:58:33] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [21:20:56] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:30:56] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:10:19] (03update) 10raymond-ndibe: [jobs-api] refactor validate_kube_quant [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/118 (https://phabricator.wikimedia.org/T361120) [23:15:56] (03update) 10raymond-ndibe: [jobs-api] convert all quotas to appropriate units [repos/cloud/toolforge/jobs-api] (refactor_validate_kube_quant) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/119 (https://phabricator.wikimedia.org/T361120) [23:16:40] (03update) 10raymond-ndibe: [jobs-api] convert all quotas to appropriate units [repos/cloud/toolforge/jobs-api] (refactor_validate_kube_quant) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/119 (https://phabricator.wikimedia.org/T361120) [23:20:07] (03update) 10raymond-ndibe: [jobs-api] refactor validate_kube_quant [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/118 (https://phabricator.wikimedia.org/T361120) [23:20:24] (03update) 10raymond-ndibe: [jobs-api] refactor validate_kube_quant [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/118 (https://phabricator.wikimedia.org/T361120) [23:58:33] FIRING: PuppetFailure: Puppet has failed on cloudcontrol2006-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure