[01:21:28] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:32:41] (03PS2) 10Reedy: README.md: Markup [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1080101 [01:33:42] (03CR) 10Reedy: [C:03+2] README.md: Markup [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1080101 (owner: 10Reedy) [01:34:29] (03Merged) 10jenkins-bot: README.md: Markup [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1080101 (owner: 10Reedy) [01:43:16] (03PS2) 10Reedy: frontend/composer.json: Add missing ext- [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1080102 [02:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:24:08] (03update) 10raymond-ndibe: [lima-kilo] refactor the project to suit a multi VM configuration [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [04:31:14] (03update) 10raymond-ndibe: [lima-kilo] refactor the project to suit a multi VM configuration [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/198 [04:41:08] (03update) 10raymond-ndibe: [lima-kilo] cache container images [repos/cloud/toolforge/lima-kilo] (refactor_in_preparation_for_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/196 [04:45:29] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] (add_cache_registry) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [04:53:17] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] (add_cache_registry) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [04:53:41] (03update) 10raymond-ndibe: [lima-kilo] configure high-availability [repos/cloud/toolforge/lima-kilo] (add_cache_registry) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/189 (https://phabricator.wikimedia.org/T374585) [04:54:24] (03update) 10raymond-ndibe: [lima-kilo] cache container images [repos/cloud/toolforge/lima-kilo] (refactor_in_preparation_for_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/196 [04:54:41] (03update) 10raymond-ndibe: [lima-kilo] cache container images [repos/cloud/toolforge/lima-kilo] (refactor_in_preparation_for_cache) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/196 [04:55:49] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [04:59:27] (03update) 10raymond-ndibe: [lima-kilo] test k8s 1.28 upgrade [repos/cloud/toolforge/lima-kilo] (configure_high_availability) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/193 (https://phabricator.wikimedia.org/T362867) [05:21:28] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:44:59] (03approved) 10raymond-ndibe: ansible.toolforge-deploy: allow passing any kind of ref [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/197 (owner: 10dcaro) [06:00:38] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1080003 (owner: 10L10n-bot) [06:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:06:47] (03open) 10sstefanova: openapi: generate operationIds from route names [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/27 (https://phabricator.wikimedia.org/T356261) [07:33:17] 10Toolforge (Toolforge iteration 16): [toolforge] simplify calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377#10227647 (10dcaro) [07:34:04] 10Toolforge (Toolforge iteration 16), 07Upstream: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#10227649 (10dcaro) [07:34:12] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 05Goal: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#10227655 (10dcaro) [07:35:22] 10Toolforge (Toolforge iteration 16), 07Upstream: [builds-builder,jobs-api,upstream] Calling nontrivial Procfile commands with arguments results in confusing error (“no such file or directory”) - https://phabricator.wikimedia.org/T356016#10227651 (10dcaro) [07:35:35] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [jobs-cli,jobs-api] quota shows different units for limit and usage - https://phabricator.wikimedia.org/T361120#10227645 (10dcaro) [07:35:46] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review, 07Upstream: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417#10227661 (10dcaro) [07:35:48] 10Toolforge (Toolforge iteration 16), 07Documentation: [harbor,docs] Improve Harbor quota handling and docs - https://phabricator.wikimedia.org/T351092#10227653 (10dcaro) [07:37:05] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: Toolforge: Replace all bastion with grid-less bookworm based bastion hosts - https://phabricator.wikimedia.org/T314665#10227657 (10dcaro) [07:37:34] 10Toolforge (Toolforge iteration 16): [builds-cli,builds-api] `build quota` fails if tool has no builds - https://phabricator.wikimedia.org/T353701#10227663 (10dcaro) [07:37:35] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): toolforge: Refresh certs that are not controlled by kubeadm (mid 2024 edition) - https://phabricator.wikimedia.org/T309782#10227659 (10dcaro) [07:38:38] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 07Epic: [components-api] First iteration of the component API - https://phabricator.wikimedia.org/T362051#10227674 (10dcaro) [07:38:59] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 07Epic: [Hypotesis] 6.3.5 Develop the sustainability score - https://phabricator.wikimedia.org/T376896#10227670 (10dcaro) [07:39:01] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 07Epic: [Hypothesis] WE6.3.4 If we enable the automatic deployment of a minimal tool, we will be able to evaluate the end to end flow and set the groundwork for adding support f... - https://phabricator.wikimedia.org/T375199#10227672 [07:39:23] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [jobs-api] Split the API, business, and k8s models - https://phabricator.wikimedia.org/T359808#10227680 (10dcaro) [07:39:52] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [components-api] Develop the webhook mechanism to trigger a deployment - https://phabricator.wikimedia.org/T362066#10227668 (10dcaro) [07:40:18] (03merge) 10dcaro: api_client: when loading cert data from kubeconfig try base64 too [repos/cloud/toolforge/toolforge-weld] (add_default_namespace) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/63 [07:40:21] (03update) 10dcaro: kubernetes_config: use the default namespace if non in kubeconfig [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/62 [07:40:30] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#10227682 (10dcaro) [07:40:43] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): [infra,k8s,monitoring] Add an alert to warn when the prometheus k8s cert is about to expire - https://phabricator.wikimedia.org/T366579#10227686 (10dcaro) [07:41:45] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16): Intermittent redis connection timeouts in Toolforge - https://phabricator.wikimedia.org/T318479#10227684 (10dcaro) [07:41:57] (03merge) 10dcaro: kubernetes_config: use the default namespace if non in kubeconfig [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/62 [07:42:05] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.28 - https://phabricator.wikimedia.org/T362867#10227692 (10dcaro) [07:42:09] 10Toolforge (Toolforge iteration 16): [maintain-dbusers] When it stops working (ex. nfs got stuck), it still replies as ok to prometheus - https://phabricator.wikimedia.org/T375224#10227693 (10dcaro) [07:42:13] 10Toolforge (Toolforge iteration 16): lima-kilo installation giving inconsistent result. Sometimes it works, sometimes it doesn't - https://phabricator.wikimedia.org/T375163#10227694 (10dcaro) [07:42:17] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [components-api] Get a skeleton of API webservice and implement `/tool//deploy` with build-only features - https://phabricator.wikimedia.org/T362069#10227676 (10dcaro) [07:42:22] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [lima-kilo] allow for the creation of a multi-node high availability cluster - https://phabricator.wikimedia.org/T374585#10227695 (10dcaro) [07:42:22] 10Toolforge (Toolforge iteration 16): [jobs-api] prepend date and pod name to filelog lines - https://phabricator.wikimedia.org/T372025#10227696 (10dcaro) [07:42:24] 10Toolforge (Toolforge iteration 16): [k8s,infra,cookbook] change the hiera under the -k8s-control prefix whet adding/removing an etcd node - https://phabricator.wikimedia.org/T371370#10227697 (10dcaro) [07:42:26] 10Toolforge (Toolforge iteration 16): Support HTTP health checks in jobs framework - https://phabricator.wikimedia.org/T362621#10227698 (10dcaro) [07:42:31] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): [api-gateway] add alert for uptime - https://phabricator.wikimedia.org/T348633#10227700 (10dcaro) [07:42:34] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#10227699 (10dcaro) [07:42:38] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS (Debian Buster Deprecation), 10Toolforge (Toolforge iteration 16), 07Epic, 05Goal: Toolforge: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T311897#10227701 (10dcaro) [07:42:54] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [jobs-api] Save business models in a DB - https://phabricator.wikimedia.org/T359650#10227678 (10dcaro) [07:44:02] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: restarting a continuous jobs causes for some seconds two jobs are running side by side - https://phabricator.wikimedia.org/T375366#10227688 (10dcaro) [07:57:00] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10227770 (10hashar) > fatal:... [08:10:05] 06cloud-services-team, 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 10Release-Engineering-Team (Seen): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10227787 (10hashar) I went t... [08:34:55] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 10Temporary accounts (Blockers to minor pilot wiki deployment), 10Trust and Safety Product Sprint (Sprint Cello (Oct 7 - 18)): Implement Global Contributions a... - https://phabricator.wikimedia.org/T376612#10227848 [09:11:39] (03CR) 10Osvaldorino: "recheck" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1080101 (owner: 10Reedy) [09:11:41] (03CR) 10Osvaldorino: "recheck" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1080101 (owner: 10Reedy) [09:11:41] (03CR) 10Osvaldorino: "recheck" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1080101 (owner: 10Reedy) [09:13:19] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 10Temporary accounts (Blockers to minor pilot wiki deployment), 10Trust and Safety Product Sprint (Sprint Cello (Oct 7 - 18)): Implement Global Contributions a... - https://phabricator.wikimedia.org/T376612#10227975 [09:13:37] 10Tool-Global-user-contributions, 10Special:GlobalContributions, 06Stewards-and-global-tools, 10Temporary accounts (Blockers to minor pilot wiki deployment), 10Trust and Safety Product Sprint (Sprint Cello (Oct 7 - 18)): Implement Global Contributions a... - https://phabricator.wikimedia.org/T376612#10227976 [09:21:28] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:37:32] (03open) 10dcaro: allow backend ca [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/44 (https://phabricator.wikimedia.org/T362066) [09:37:54] (03update) 10dcaro: allow backend ca [repos/cloud/toolforge/api-gateway] (add_superuser_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/44 (https://phabricator.wikimedia.org/T362066) [09:56:33] (03update) 10dcaro: Add superuser support [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/43 (https://phabricator.wikimedia.org/T362066) [09:56:45] (03update) 10dcaro: allow backend ca [repos/cloud/toolforge/api-gateway] (add_superuser_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/44 (https://phabricator.wikimedia.org/T362066) [09:57:41] (03update) 10dcaro: allow backend ca [repos/cloud/toolforge/api-gateway] (add_superuser_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/44 (https://phabricator.wikimedia.org/T362066) [10:00:23] (03update) 10dcaro: Add superuser support [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/43 (https://phabricator.wikimedia.org/T362066) [10:00:28] (03update) 10dcaro: allow backend ca [repos/cloud/toolforge/api-gateway] (add_superuser_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/44 (https://phabricator.wikimedia.org/T362066) [10:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [10:09:49] (03update) 10dcaro: Draft: Add the creation of a single continuous job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/23 (https://phabricator.wikimedia.org/T362066) [10:11:51] (03update) 10dcaro: Draft: Add the creation of a single continuous job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/23 (https://phabricator.wikimedia.org/T362066) [10:12:26] (03update) 10dcaro: Add superuser support [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/43 (https://phabricator.wikimedia.org/T362066) [10:13:50] (03update) 10dcaro: allow backend ca [repos/cloud/toolforge/api-gateway] (add_superuser_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/44 (https://phabricator.wikimedia.org/T362066) [10:15:17] (03update) 10dcaro: Draft: Add the creation of a single continuous job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/23 (https://phabricator.wikimedia.org/T362066) [10:17:50] (03merge) 10aborrero: codfw1dev: default secgroup: refresh IPv6 settings [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/96 (https://phabricator.wikimedia.org/T374714) [10:17:59] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [10:18:32] 10Tool-itwiki: Understand why BotCancellazioni is stuck (after every forced wiki read-only mode) apparently for cURL reasons - https://phabricator.wikimedia.org/T375937#10228377 (10valerio.bozzolan) [10:20:03] 10Tool-itwiki: Understand why BotCancellazioni is stuck (after every forced wiki read-only mode) apparently for cURL reasons - https://phabricator.wikimedia.org/T375937#10228386 (10valerio.bozzolan) [10:22:01] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [10:22:07] (03open) 10aborrero: modules/routers: set enable_snat parameter [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/97 [10:26:15] (03update) 10aborrero: modules/routers: set enable_snat parameter [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/97 [10:27:09] (03update) 10aborrero: modules/routers: set enable_snat parameter [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/97 [10:27:46] (03update) 10dcaro: Draft: Add the creation of a single continuous job [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/23 (https://phabricator.wikimedia.org/T362066) [10:28:52] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [10:29:43] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [10:31:06] (03open) 10aborrero: codfw1dev: default secgroup: fix ICMP in IPv6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/98 (https://phabricator.wikimedia.org/T374714) [10:32:32] (03merge) 10aborrero: codfw1dev: default secgroup: fix ICMP in IPv6 [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/98 (https://phabricator.wikimedia.org/T374714) [10:32:42] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [10:33:02] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [10:33:10] (03update) 10aborrero: modules/routers: set enable_snat parameter [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/97 [10:34:38] (03update) 10aborrero: modules/routers: set enable_snat parameter [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/97 [10:36:06] (03merge) 10aborrero: modules/routers: set enable_snat parameter [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/97 [10:36:11] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [10:36:40] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [11:22:06] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops, 06SRE: openstack: initial IPv6 support in neutron - https://phabricator.wikimedia.org/T375847#10228569 (10aborrero) 05In progress→03Resolved I think we can consider this to be completed. We may reopen if required. [11:41:27] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [11:41:49] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [12:12:25] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: clarify IPv6 firewalling - https://phabricator.wikimedia.org/T374714#10228685 (10aborrero) problem detected: `remote_group_name` parameter in tofu-infra security group rules is not getting resolved. [12:50:53] 10Tool-nlwikibots: e85bot/tvpmelder reports to the nominator in some cases - https://phabricator.wikimedia.org/T85102#10228809 (10FrankGeerlings) 05Open→03Resolved a:03FrankGeerlings tvpmelder was superseded by tbpmelder a few years ago, and its behaviour is currently such that whoever started the arti... [12:51:48] 06cloud-services-team, 10Toolforge: Deleting an envvar breaks ReplicaSet driven automatic restarts of a Pod (CreateContainerConfigError) - https://phabricator.wikimedia.org/T365048#10228813 (10dcaro) >> (btw. note that directly deleting the pod is not a supported flow of doing that, you should always toolforge... [13:00:03] 10Tool-nlwikibots: tbpmelder doesn't always inform the article's first author - https://phabricator.wikimedia.org/T377209 (10FrankGeerlings) 03NEW [13:21:28] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [14:38:25] 10Tool-itwiki: Understand why BotCancellazioni is stuck (after every forced wiki read-only mode) apparently for cURL reasons - https://phabricator.wikimedia.org/T375937#10229476 (10valerio.bozzolan) Can reproduce \o/ {F57616944} [14:45:57] 10Tool-itwiki: Understand why BotCancellazioni is stuck (after every forced wiki read-only mode) apparently for cURL reasons - https://phabricator.wikimedia.org/T375937#10229585 (10valerio.bozzolan) Trying with ` diff --git a/include/network/HTTPRequest.php b/include/network/HTTPRequest.php index 5e93643..45f55... [15:31:09] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [15:31:09] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch [15:31:42] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [15:31:43] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch [15:32:05] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [15:32:06] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch [15:32:44] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [15:32:44] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch [15:35:30] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [15:35:30] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch [15:35:46] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [15:35:46] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch [15:38:04] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [15:38:05] !log aborrero@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for main branch [15:41:45] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [15:42:12] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [15:48:45] !log aborrero@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [15:50:22] !log aborrero@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [16:01:53] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [16:08:09] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [16:13:09] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for JJMC89 - https://phabricator.wikimedia.org/T375041#10230081 (10bd808) 05Open→03Resolved a:03bd808 >>! In T374993#10222631, @KFrancis wrote: > Hi all, I am confirming @JJMC89 has an NDA on file. Thanks! I have added @JJMC89 to th... [16:13:11] 06Toolforge-standards-committee: Facilitate Volunteer NDA application process for 2024 Toolforge standards committee appointees - https://phabricator.wikimedia.org/T374993#10230086 (10bd808) [16:14:09] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [16:16:11] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway [16:16:42] 10Tool-itwiki: Understand why BotCancellazioni is stuck (after every forced wiki read-only mode) apparently for cURL reasons - https://phabricator.wikimedia.org/T375937#10230110 (10valerio.bozzolan) This change is in production right now https://gitpull.it/R21:a2f0fad757c30a9841ee7ba1adff40bcaea614cf [16:21:14] 10Tool-itwiki: Understand why BotCancellazioni is stuck (after every forced wiki read-only mode) apparently for cURL reasons - https://phabricator.wikimedia.org/T375937#10230121 (10valerio.bozzolan) 05Open→03Resolved I sincerely believe that the root problem is now resolved. Let's re-open if I'm wrong. T... [16:35:37] 10Tool-itwiki: Understand why BotCancellazioni is stuck (after every forced wiki read-only mode) apparently for cURL reasons - https://phabricator.wikimedia.org/T375937#10230205 (10valerio.bozzolan) Ah, to apply the update, I've just followed the documentation here: https://wikitech.wikimedia.org/wiki/Tool:... [16:38:38] 10wikitech.wikimedia.org: Re-enable account creation on Wikitech - https://phabricator.wikimedia.org/T377074#10230240 (10bd808) [16:38:46] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10230239 (10bd808) [16:38:54] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for TheProtonade - https://phabricator.wikimedia.org/T375007#10230241 (10KFrancis) thanks, @theprotonade. I checked my inbox, but it doesn't seem to be there. Would you please send it again and include your user name? [16:41:25] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for Antonin Delpeuch (Pintoch) - https://phabricator.wikimedia.org/T374995#10230261 (10Pintoch) I've signed the NDA a few days ago, let me know if more is needed [16:43:16] 06Toolforge-standards-committee: Facilitate Volunteer NDA application process for 2024 Toolforge standards committee appointees - https://phabricator.wikimedia.org/T374993#10230274 (10KFrancis) -Confirming NDA's are complete for @Pintoch and @SD0001. Thanks! [16:50:28] 10wikitech.wikimedia.org: Re-enable account creation on Wikitech - https://phabricator.wikimedia.org/T377074#10230305 (10GTrang) [17:03:21] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Cloud-VPS, 10Ceph, 06DC-Ops, and 2 others: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643#10230424 (10BTullis) [17:08:06] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [17:10:41] 10Tool-refill: Update reFill toolforge metadata links - https://phabricator.wikimedia.org/T377248 (10Daask) 03NEW [17:11:14] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for TheProtonade - https://phabricator.wikimedia.org/T375007#10230452 (10KFrancis) Please disregard my last message. The NDA is complete. Thanks! [17:14:08] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [17:14:25] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [17:20:25] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [17:21:28] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:27:34] (03PS6) 10David Caro: toolforge.component.deploy: handle non-mr package deploys [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1078676 [17:31:36] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: handle non-mr package deploys [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1078676 (owner: 10David Caro) [17:32:57] (03PS7) 10David Caro: toolforge.component.deploy: handle non-mr package deploys [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1078676 [17:40:11] (03PS1) 10Scott French: hiddenparma: add fake keyholder secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1080333 (https://phabricator.wikimedia.org/T371782) [17:40:13] (03CR) 10David Caro: [C:03+2] toolforge.component.deploy: handle non-mr package deploys [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1078676 (owner: 10David Caro) [17:41:19] (03CR) 10David Caro: toolforge.component.deploy: handle non-mr package deploys (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1078676 (owner: 10David Caro) [17:44:09] (03CR) 10Scott French: [V:03+2 C:03+2] hiddenparma: add fake keyholder secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1080333 (https://phabricator.wikimedia.org/T371782) (owner: 10Scott French) [17:47:19] 10VPS-project-Phabricator: Confirm account on phabricator.wmcloud.org - https://phabricator.wikimedia.org/T377236#10230605 (10Pppery) [17:47:23] 10VPS-project-Phabricator: Confirm account on phabricator.wmcloud.org - https://phabricator.wikimedia.org/T377236#10230606 (10Pppery) [17:48:52] 10Tool-ldap, 10Phabricator: https://ldap.toolforge.org/ integration assumes that `cn` and `uid` are equivalent - https://phabricator.wikimedia.org/T376769#10230611 (10Legoktm) >>! In T376769#10223261, @matmarex wrote: > Hmm, they’re not even 404s, but 500s for me. Both the space and the ‘ń’ seem to cause that.... [17:55:19] 10VPS-project-Phabricator, 06collaboration-services: Confirm account on phabricator.wmcloud.org - https://phabricator.wikimedia.org/T377236#10230645 (10Dzahn) [17:57:31] 10VPS-project-Phabricator, 06collaboration-services: Confirm account on phabricator.wmcloud.org - https://phabricator.wikimedia.org/T377236#10230655 (10Dzahn) So, this is also to other people in my team. I did not have an admin account for the web UI, but this is possible via shell commands. Generally the sh... [17:57:32] 10VPS-project-Phabricator, 06collaboration-services: Confirm account on phabricator.wmcloud.org - https://phabricator.wikimedia.org/T377236#10230647 (10Dzahn) The original tags were correct. collaboration services does control services in the devtools project. [17:57:54] 10VPS-project-Phabricator, 06collaboration-services: Confirm account on phabricator.wmcloud.org - https://phabricator.wikimedia.org/T377236#10230657 (10Dzahn) 05Open→03In progress [18:05:15] 10Tool-nfp: tool-nfp should not manually concatenate SQL - https://phabricator.wikimedia.org/T336978#10230678 (10Legoktm) I believe I filed this ticket because certain developers were strongly pushing the use of query builders in MediaWiki, so it seemed like a good general pattern for tools too. Maybe the tool a... [18:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [18:37:23] 06cloud-services-team, 10Toolforge: "unsuccessful cred copy" with toolforge build - https://phabricator.wikimedia.org/T377255 (10Don-vip) 03NEW [18:40:17] 06cloud-services-team, 10Toolforge: Timeouts when pulling artifacts from github with toolforge build - https://phabricator.wikimedia.org/T377255#10230827 (10dcaro) [18:40:19] 06cloud-services-team, 10Toolforge: "can't connect to remote host" with toolforge build - https://phabricator.wikimedia.org/T377255#10230829 (10Don-vip) [18:40:58] 06cloud-services-team, 10Toolforge: "can't connect to remote host" with toolforge build - https://phabricator.wikimedia.org/T377255#10230835 (10Don-vip) Yes sorry I just seen it. FYI I just relaunched the job and it worked :) [18:41:15] 06cloud-services-team, 10Toolforge: Timeouts when pulling artifacts from github with toolforge build - https://phabricator.wikimedia.org/T377255#10230849 (10Don-vip) [18:43:05] 06cloud-services-team, 10Toolforge: Timeouts when pulling artifacts from github with toolforge build - https://phabricator.wikimedia.org/T377255#10230825 (10dcaro) The cred copy is a warning (annoying, but not so easy to get rid of), the actual error is buried a bit there: ` [step-inject-buildpacks] 2024-10-15... [18:56:59] 06cloud-services-team, 10wikitech.wikimedia.org, 06Trust-and-Safety: Account recovery help needed for Developer account jgiannelos - https://phabricator.wikimedia.org/T377117#10230876 (10bd808) As of 2024-10-01 #wikitech.wikimedia.org no longer configures, manages, or enforces MediaWiki's TOTP 2FA related to... [19:00:04] 06cloud-services-team, 10wikitech.wikimedia.org, 06Trust-and-Safety: Account recovery help needed for Developer account jgiannelos - https://phabricator.wikimedia.org/T377117#10230879 (10bd808) @Jgiannelos your report mentions Yubikey 2FA which I think is related to https://wikitech.wikimedia.org/wiki/Yubike... [19:51:00] 10wikitech.wikimedia.org, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Wikitech showing Wikipedia CentralNotice banners - https://phabricator.wikimedia.org/T377030#10231105 (10AKanji-WMF) p:05Triage→03High [20:28:05] 10Tool-ldap, 10Phabricator: https://ldap.toolforge.org/ integration assumes that `cn` and `uid` are equivalent - https://phabricator.wikimedia.org/T376769#10231278 (10bd808) >>! In T376769#10230611, @Legoktm wrote: > I'll spend a bit of time figuring out actual LDAP query escaping instead of using a too-restri... [20:30:46] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for TheProtonade - https://phabricator.wikimedia.org/T375007#10231283 (10bd808) 05Open→03Resolved a:03bd808 >>! In T375007#10230452, @KFrancis wrote: > Please disregard my last message. The NDA is complete. Thanks! @theprotonade ha... [20:31:07] 06Toolforge-standards-committee: Facilitate Volunteer NDA application process for 2024 Toolforge standards committee appointees - https://phabricator.wikimedia.org/T374993#10231304 (10bd808) [20:33:40] 06Toolforge-standards-committee: Facilitate Volunteer NDA application process for 2024 Toolforge standards committee appointees - https://phabricator.wikimedia.org/T374993#10231327 (10bd808) [20:34:32] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for Antonin Delpeuch (Pintoch) - https://phabricator.wikimedia.org/T374995#10231319 (10bd808) 05Open→03Resolved a:03bd808 >>! In T374993#10230274, @KFrancis wrote: > -Confirming NDA's are complete for @Pintoch and @SD0001. Thanks! @... [20:35:05] 06Toolforge-standards-committee: Facilitate Volunteer NDA application process for 2024 Toolforge standards committee appointees - https://phabricator.wikimedia.org/T374993#10231345 (10bd808) [20:35:41] 06Toolforge-standards-committee, 06WMF-NDA-Requests: Volunteer NDA for SD0001 - https://phabricator.wikimedia.org/T374998#10231339 (10bd808) 05Open→03Resolved a:03bd808 >>! In T374993#10230274, @KFrancis wrote: > -Confirming NDA's are complete for @Pintoch and @SD0001. Thanks! @SD0001 is now a memb... [20:40:24] 06Toolforge-standards-committee: Facilitate Volunteer NDA application process for 2024 Toolforge standards committee appointees - https://phabricator.wikimedia.org/T374993#10231355 (10bd808) 05In progress→03Resolved All 8 approved nominee have completed the NDA signing process and are now members of the... [20:48:34] 06Toolforge-standards-committee, 07User-notice: Refresh membership of Toolforge standards committee - https://phabricator.wikimedia.org/T370474#10231394 (10bd808) [20:49:19] 06Toolforge-standards-committee, 07User-notice: Refresh membership of Toolforge standards committee - https://phabricator.wikimedia.org/T370474#10231412 (10bd808) 05Stalled→03In progress [21:21:28] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:25:51] 10VPS-project-Phabricator, 06collaboration-services: Confirm account on phabricator.wmcloud.org - https://phabricator.wikimedia.org/T377236#10231600 (10TheresNoTime) Still can't log in: ` Check Your Email You must verify your email address to log in. You should have a new email message with verification inst... [21:36:14] 10VPS-project-Phabricator, 06collaboration-services: Confirm account on phabricator.wmcloud.org - https://phabricator.wikimedia.org/T377236#10231627 (10Dzahn) Ah! So this isn't approval, it's ` sudo ./auth verify ` Also `./user empower --user ` is to make someone admin. But even as ad... [21:39:14] 10VPS-project-Phabricator, 06collaboration-services: Confirm account on phabricator.wmcloud.org - https://phabricator.wikimedia.org/T377236#10231634 (10Dzahn) 05In progress→03Resolved a:03Dzahn [21:41:04] 10VPS-project-Phabricator, 06collaboration-services: Confirm account on phabricator.wmcloud.org - https://phabricator.wikimedia.org/T377236#10231641 (10Dzahn) P.S. I also made theresnotime an admin on the test instance to have badge-related permissions and for general testing. [22:07:19] FIRING: PuppetZeroResources: Puppet has failed generate resources on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [22:47:34] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate mwv-builder-03.mediawiki-vagrant.eqiad.wmflabs is about to expire in 9d 23h 58m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire