[00:00:31] (03update) 10chuckonwumelu: Draft: Start [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/1 [00:07:51] (03update) 10chuckonwumelu: Draft: Start [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/1 [00:18:05] (03update) 10chuckonwumelu: Draft: Start [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/1 [00:28:42] (03update) 10chuckonwumelu: Draft: Start [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/1 [00:42:11] (03update) 10chuckonwumelu: Draft: Start [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/1 [01:11:21] (03approved) 10raymond-ndibe: [jobs-api] fix cron parsing bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/155 (https://phabricator.wikimedia.org/T391786) [01:11:28] (03merge) 10raymond-ndibe: [jobs-api] fix cron parsing bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/155 (https://phabricator.wikimedia.org/T391786) [01:49:27] (03update) 10raymond-ndibe: [envvars-api] return custom message for invalid EnvvarName [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T360147) [01:49:57] (03update) 10raymond-ndibe: [envvars-api] return custom message for invalid EnvvarName [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T360147) [03:10:46] 06cloud-services-team, 10Toolforge: Delete unused toolforge project - https://phabricator.wikimedia.org/T391912 (10Huji) 03NEW [03:38:06] (03update) 10chuckonwumelu: Draft: Start [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/1 [03:42:07] (03update) 10chuckonwumelu: Draft: Start [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/1 [03:46:10] (03update) 10chuckonwumelu: Draft: Start [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/1 [03:48:21] (03update) 10chuckonwumelu: Start [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/1 [06:34:07] (03CR) 10Jelto: [V:03+2 C:03+2] gitlab: use a wmflib::expand_path compatible path for apus keys [labs/private] - 10https://gerrit.wikimedia.org/r/1136391 (https://phabricator.wikimedia.org/T378922) (owner: 10Jelto) [07:04:54] 06cloud-services-team, 10Cloud-VPS, 10Observability-Metrics, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q4): Remove librenms -> graphite integration, replace with gnmi - https://phabricator.wikimedia.org/T372457#10742363 (10fgiunchedi) [07:19:09] 06cloud-services-team, 10Toolforge: Delete unused toolforge project - https://phabricator.wikimedia.org/T391912#10742415 (10Bugreporter) See https://wikitech.wikimedia.org/wiki/Help:Toolforge/Tool_accounts#Delete_a_tool_account [07:46:16] 06cloud-services-team, 10Toolforge: Delete unused toolforge project - https://phabricator.wikimedia.org/T391912#10742479 (10taavi) 05Open→03Resolved a:03Huji Looks like you've already disabled it? [07:46:37] 10Tools: Delete unused toolforge project - https://phabricator.wikimedia.org/T391912#10742483 (10taavi) [09:40:51] (03merge) 10dcaro: namespace: add comment about the tenancy label usage [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/66 [10:53:54] 10Toolforge (Toolforge iteration 19): [gitlab-ci] create-toolforge-deploy-upgrade-branch job failing - https://phabricator.wikimedia.org/T391945 (10Raymond_Ndibe) 03NEW [10:55:45] (03update) 10raymond-ndibe: Draft: [envvars-api] return custom message for invalid EnvvarName [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/53 (https://phabricator.wikimedia.org/T360147) [11:19:39] (03open) 10group_203_bot_0baec7e1908816bc38475b0da887f76c: jobs-api: bump to 0.0.367-20250415011142-c4e5b2cb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/744 (https://phabricator.wikimedia.org/T391786) [11:20:30] 10Toolforge (Toolforge iteration 19): [gitlab-ci] create-toolforge-deploy-upgrade-branch job failing - https://phabricator.wikimedia.org/T391945#10743061 (10Raymond_Ndibe) 05Open→03Resolved a:03Raymond_Ndibe [11:22:01] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [11:26:41] (03update) 10raymond-ndibe: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118) [11:27:21] (03update) 10raymond-ndibe: [jobs-api] use pydantic for all models [repos/cloud/toolforge/jobs-api] (move_most_custom_validations_out_of_api_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T389118) [11:33:04] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [11:34:44] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [11:39:54] (03update) 10raymond-ndibe: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] (use_pydantic_for_core_job_model) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) [11:40:19] (03update) 10raymond-ndibe: [jobs-api] use pydantic for all models [repos/cloud/toolforge/jobs-api] (move_most_custom_validations_out_of_api_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T389118) [11:51:38] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [11:54:56] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: openstack: improve networktests for newer network setup - https://phabricator.wikimedia.org/T391325#10743137 (10aborrero) 05In progress→03Resolved a:03aborrero completed. We have now 86 tests covering a variety of uses cases, with i... [11:59:57] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.367-20250415011142-c4e5b2cb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/744 (https://phabricator.wikimedia.org/T391786) (owner: 10ghost) [11:59:59] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.367-20250415011142-c4e5b2cb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/744 (https://phabricator.wikimedia.org/T391786) (owner: 10ghost) [12:00:03] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.367-20250415011142-c4e5b2cb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/744 (https://phabricator.wikimedia.org/T391786) (owner: 10ghost) [12:00:48] 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [jobs-api] can't list jobs for certain old jobs - https://phabricator.wikimedia.org/T391786#10743177 (10Raymond_Ndibe) 05In progress→03Resolved [12:02:42] 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [jobs-api] can't list jobs for certain old jobs - https://phabricator.wikimedia.org/T391786#10743186 (10Raymond_Ndibe) 05Resolved→03In progress [12:03:23] (03update) 10fnegri: Import some db-related DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/180 (https://phabricator.wikimedia.org/T374953) [12:19:49] (03update) 10fnegri: Import some db-related DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/180 (https://phabricator.wikimedia.org/T374953) [12:27:42] (03update) 10fnegri: Import some db-related DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/180 (https://phabricator.wikimedia.org/T374953) [12:28:14] (03approved) 10aborrero: Import some db-related DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/180 (https://phabricator.wikimedia.org/T374953) (owner: 10fnegri) [12:29:18] (03merge) 10fnegri: Import some db-related DNS records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/180 (https://phabricator.wikimedia.org/T374953) [12:30:49] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:31:53] !log fnegri@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [12:33:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:33:54] (03open) 10fnegri: Remove import blocks [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/194 (https://phabricator.wikimedia.org/T374953) [12:35:07] (03approved) 10aborrero: Remove import blocks [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/194 (https://phabricator.wikimedia.org/T374953) (owner: 10fnegri) [12:35:32] (03merge) 10fnegri: Remove import blocks [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/194 (https://phabricator.wikimedia.org/T374953) [12:37:54] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:38:23] !log fnegri@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:38:40] (03open) 10fnegri: cloudinfra/dns: remove legacy gw records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/195 (https://phabricator.wikimedia.org/T374953) [12:38:59] (03open) 10raymond-ndibe: [jobs-api] fix cron parsing bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/156 (https://phabricator.wikimedia.org/T391786) [12:43:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:47:11] (03update) 10fnegri: cloudinfra/dns: remove legacy gw records [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/195 (https://phabricator.wikimedia.org/T374953) [12:47:39] (03update) 10fnegri: cloudinfra/dns: remove legacy gw record [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/195 (https://phabricator.wikimedia.org/T374953) [12:48:52] (03approved) 10aborrero: cloudinfra/dns: remove legacy gw record [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/195 (https://phabricator.wikimedia.org/T374953) (owner: 10fnegri) [12:49:09] (03merge) 10fnegri: cloudinfra/dns: remove legacy gw record [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/195 (https://phabricator.wikimedia.org/T374953) [12:49:21] !log fnegri@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [12:49:59] !log fnegri@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [12:52:41] (03approved) 10raymond-ndibe: [jobs-api] fix cron parsing bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/156 (https://phabricator.wikimedia.org/T391786) [12:52:45] (03merge) 10raymond-ndibe: [jobs-api] fix cron parsing bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/156 (https://phabricator.wikimedia.org/T391786) [12:52:46] (03update) 10raymond-ndibe: [jobs-api] fix cron parsing bug [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/156 (https://phabricator.wikimedia.org/T391786) [12:55:06] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.368-20250415125256-59bf37cb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/745 (https://phabricator.wikimedia.org/T391786) [12:55:10] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.368-20250415125256-59bf37cb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/745 (https://phabricator.wikimedia.org/T391786) [12:55:52] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:59:00] (03open) 10aborrero: eqiad1: add new VXLAN and IPv6 network settings [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/196 [12:59:18] 06cloud-services-team, 10Toolforge: [jobs-api] Indicate when a job is too big to be scheduled - https://phabricator.wikimedia.org/T383515#10743551 (10Raymond_Ndibe) a:03Raymond_Ndibe [12:59:51] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [jobs-api] Indicate when a job is too big to be scheduled - https://phabricator.wikimedia.org/T383515#10743554 (10Raymond_Ndibe) [13:03:18] (03open) 10aborrero: eqiad1: refresh FQDN for neutron virtual router [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/197 (https://phabricator.wikimedia.org/T380174) [13:04:39] (03update) 10aborrero: eqiad1: refresh FQDN for neutron virtual router [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/197 (https://phabricator.wikimedia.org/T380174) [13:04:43] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [jobs-api,jobs-cli] Replace already completed one-off jobs when starting a new one - https://phabricator.wikimedia.org/T352989#10743566 (10Raymond_Ndibe) a:03Raymond_Ndibe [13:05:56] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [builds-cli,builds-api] `build quota` fails if tool has no builds - https://phabricator.wikimedia.org/T353701#10743572 (10Raymond_Ndibe) a:03Raymond_Ndibe [13:06:00] 10Tools: Delete unused toolforge project - https://phabricator.wikimedia.org/T391912#10743574 (10Huji) The part I didn't know and Bugreporter educated me on was that deletion happens automatically after 40 days from archiving. So yes, all set here. [13:06:12] (03update) 10aborrero: eqiad1: refresh FQDN for neutron virtual router [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/197 (https://phabricator.wikimedia.org/T380174) [13:07:20] (03update) 10aborrero: eqiad1: refresh FQDN for neutron virtual router [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/197 (https://phabricator.wikimedia.org/T380174) [13:07:52] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:09:31] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:22:40] 10Toolforge (Toolforge iteration 19): [envvars-api] bug in envvars-api EnvvarName validation Regext - https://phabricator.wikimedia.org/T391966 (10Raymond_Ndibe) 03NEW [13:23:01] 10Toolforge (Toolforge iteration 19): [envvars-api] bug in envvars-api EnvvarName validation Regext - https://phabricator.wikimedia.org/T391966#10743626 (10Raymond_Ndibe) [13:23:12] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:25:08] 10Toolforge (Toolforge iteration 19): [envvars-api] bug in envvars-api EnvvarName validation Regext - https://phabricator.wikimedia.org/T391966#10743640 (10Raymond_Ndibe) [13:28:13] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.368-20250415125256-59bf37cb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/745 (https://phabricator.wikimedia.org/T391786) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:28:16] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.368-20250415125256-59bf37cb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/745 (https://phabricator.wikimedia.org/T391786) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:28:22] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.368-20250415125256-59bf37cb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/745 (https://phabricator.wikimedia.org/T391786) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:30:45] 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [jobs-api] can't list jobs for certain old jobs - https://phabricator.wikimedia.org/T391786#10743670 (10Raymond_Ndibe) 05In progress→03Resolved [13:35:17] (03open) 10dcaro: Refresh vcr tests and fix recording script [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/67 [13:54:37] (03update) 10dcaro: Refresh vcr tests and fix recording script [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/67 [14:23:13] 06cloud-services-team, 10Cloud-VPS: Options/thoughts for faster VM provisioning - https://phabricator.wikimedia.org/T390822#10743936 (10Andrew) @fgiunchedi I'm going to close this unless you have more ideas [14:24:57] (03update) 10aborrero: eqiad1: refresh FQDN for neutron virtual router [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/197 (https://phabricator.wikimedia.org/T380174) [14:33:15] (03update) 10aborrero: eqiad1: refresh FQDN for neutron virtual router [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/197 (https://phabricator.wikimedia.org/T380174) [14:33:42] 06cloud-services-team, 10Cloud-VPS: Options/thoughts for faster VM provisioning - https://phabricator.wikimedia.org/T390822#10743991 (10fgiunchedi) >>! In T390822#10743936, @Andrew wrote: > @fgiunchedi I'm going to close this unless you have more ideas I'd like to run tests with the metadata you suggested and... [14:48:26] (03update) 10chuckonwumelu: Start [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/1 [15:13:12] (03merge) 10aborrero: eqiad1: refresh FQDN for neutron virtual router [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/197 (https://phabricator.wikimedia.org/T380174) [15:13:19] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:13:43] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [15:15:58] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:16:33] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [15:16:43] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:17:08] (03update) 10chuckonwumelu: Start [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/1 [15:17:11] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [15:17:35] (03update) 10aborrero: eqiad1: add new VXLAN and IPv6 network settings [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/196 [15:27:45] 06cloud-services-team, 10Cloud-VPS, 07Documentation: Consolidate and deduplicate docs about generating SSH keys - https://phabricator.wikimedia.org/T391989 (10TBurmeister) 03NEW [15:29:37] (03update) 10aborrero: eqiad1: add new VXLAN and IPv6 network settings [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/196 [15:31:10] 06cloud-services-team, 10Cloud-VPS, 07Documentation: Consolidate and deduplicate docs about generating SSH keys - https://phabricator.wikimedia.org/T391989#10744289 (10TBurmeister) [15:32:35] (03update) 10aborrero: eqiad1: add new VXLAN and IPv6 network settings [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/196 [15:39:15] 06cloud-services-team, 10Toolforge: [infra,k8s] Move to kubernetes VAPs and drop kyverno - https://phabricator.wikimedia.org/T364293#10744336 (10fnegri) [15:43:08] (03PS2) 10Volans: wmcs.common: update wrap_with_sudo_icinga [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1132676 [15:43:35] (03CR) 10Volans: "I'll leave it up to WMCS to decide how/when to merge this and update spicerack on the cloudcumin hosts." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1132676 (owner: 10Volans) [15:51:02] 10cloud-services-team (FY2024/2025-Q3-Q4), 06DC-Ops, 10ops-eqiad: Temperature Inlet Temp issue on clouddumps1001:9290 - https://phabricator.wikimedia.org/T383723#10744402 (10Jclark-ctr) @fnegri i had looked at this briefly kinda looks like might be a bad intake sensor and might not be over heating comparin... [16:03:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:28:11] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:52:00] (03update) 10raymond-ndibe: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118) [16:54:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudgw1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [17:02:20] (03update) 10raymond-ndibe: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118) [17:04:48] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on cloudgw1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [17:08:55] (03update) 10raymond-ndibe: [jobs-api] use pydantic for all models [repos/cloud/toolforge/jobs-api] (move_most_custom_validations_out_of_api_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T389118) [17:33:10] (03update) 10raymond-ndibe: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] (use_pydantic_for_core_job_model) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) [17:45:27] (03CR) 10David Caro: [C:03+1] "Thanks! If I find some time tomorrow I'll give it a test" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1132676 (owner: 10Volans) [17:45:51] (03CR) 10David Caro: [V:03+1 C:03+2] vps.create_project: update the project file created [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1136402 (https://phabricator.wikimedia.org/T375283) (owner: 10David Caro) [17:49:22] (03Merged) 10jenkins-bot: vps.create_project: update the project file created [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1136402 (https://phabricator.wikimedia.org/T375283) (owner: 10David Caro) [17:56:34] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of pawsdev project - https://phabricator.wikimedia.org/T392004 (10Andrew) 03NEW [17:59:24] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of pawsdev project - https://phabricator.wikimedia.org/T392004#10744888 (10dcaro) +1 [18:06:04] (03PS6) 10Arendpieter: Switch username validation to Bitu API [labs/striker] - 10https://gerrit.wikimedia.org/r/1134724 (https://phabricator.wikimedia.org/T364605) [18:43:07] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of pawsdev project - https://phabricator.wikimedia.org/T392004#10745069 (10Andrew) 05Open→03Stalled Hold off on this for a bit, it might still be better to do this in codfw1dev [18:47:27] (03open) 10raymond-ndibe: [builds-api] create harbor project before getting quota [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/128 (https://phabricator.wikimedia.org/T353701) [18:47:39] (03update) 10raymond-ndibe: [builds-api] create harbor project before getting quota [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/128 (https://phabricator.wikimedia.org/T353701) [18:48:49] (03update) 10raymond-ndibe: [builds-api] create harbor project before getting quota [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/128 (https://phabricator.wikimedia.org/T353701) [18:52:28] 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [builds-cli,builds-api] `build quota` fails if tool has no builds - https://phabricator.wikimedia.org/T353701#10745107 (10Raymond_Ndibe) 05Stalled→03In progress [18:52:31] 10Toolforge (Toolforge iteration 19): [envvars-api] bug in envvars-api EnvvarName validation Regex - https://phabricator.wikimedia.org/T391966#10745108 (10Raymond_Ndibe) [18:58:30] (03open) 10raymond-ndibe: [envvars-api] fix envvars-api EnvvarName regex bug [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/54 (https://phabricator.wikimedia.org/T391966) [19:03:51] (03update) 10raymond-ndibe: [envvars-api] fix envvars-api EnvvarName regex bug [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/54 (https://phabricator.wikimedia.org/T391966) [20:06:36] (03open) 10raymond-ndibe: [jobs-api] delete completed one-off jobs when getting jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/157 (https://phabricator.wikimedia.org/T352989) [20:24:48] 06cloud-services-team, 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [jobs-api,jobs-cli] Replace already completed one-off jobs when starting a new one - https://phabricator.wikimedia.org/T352989#10745574 (10Raymond_Ndibe) 05Open→03In progress [20:28:35] 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [envvars-api] bug in envvars-api EnvvarName validation Regex - https://phabricator.wikimedia.org/T391966#10745584 (10Raymond_Ndibe) 05Open→03In progress [20:55:31] 06cloud-services-team, 10Cloud-VPS (Project-requests): Request creation of pawsdev project - https://phabricator.wikimedia.org/T392004#10745656 (10Raymond_Ndibe) >>! In T392004#10745069, @Andrew wrote: > Hold off on this for a bit, it might still be better to do this in codfw1dev ok @Andrew [21:36:39] 06cloud-services-team, 10Cloud-VPS: openstack magnum (or heat) resource leak - https://phabricator.wikimedia.org/T392031 (10Andrew) 03NEW [21:37:19] 06cloud-services-team, 10Cloud-VPS: openstack magnum (or heat) resource leak - https://phabricator.wikimedia.org/T392031#10745769 (10Andrew) {F59249487} [22:27:19] FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [22:42:19] RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [23:09:31] 06cloud-services-team, 10Cloud-VPS, 07Documentation: Consolidate and deduplicate docs about generating SSH keys - https://phabricator.wikimedia.org/T391989#10745934 (10bd808) > Targeted at Cloud VPS users, but that isn't clear from the title / page location From my POV all pages in Wikitech's Help namespace... [23:16:51] 06cloud-services-team, 10Cloud-VPS, 07Documentation: Consolidate and deduplicate docs about generating SSH keys - https://phabricator.wikimedia.org/T391989#10745945 (10bd808) > https://wikitech.wikimedia.org/wiki/Generate_an_SSH_Key Apparently Sarah R made that one, possibly because she didn't notice https:/... [23:29:19] 10VPS-project-Wikistats: Add nupwiki to wikistats - https://phabricator.wikimedia.org/T391856#10745961 (10Dzahn) waiting on T390384