[00:35:53] (03open) 10andrew: Update app credentials in eqiad1 [repos/cloud/cloud-vps/tf-infra-test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tf-infra-test/-/merge_requests/4 [00:35:57] (03merge) 10andrew: Update app credentials in eqiad1 [repos/cloud/cloud-vps/tf-infra-test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tf-infra-test/-/merge_requests/4 [01:03:55] (03open) 10andrew: Update app credentials in eqiad1, again [repos/cloud/cloud-vps/tf-infra-test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tf-infra-test/-/merge_requests/5 [01:04:29] (03merge) 10andrew: Update app credentials in eqiad1, again [repos/cloud/cloud-vps/tf-infra-test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tf-infra-test/-/merge_requests/5 [01:43:36] (03update) 10raymond-ndibe: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] (use_pydantic_for_core_job_model) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) [01:52:18] (03update) 10raymond-ndibe: [jobs-api] refactor quota models [repos/cloud/toolforge/jobs-api] (use_pydantic_for_core_job_model) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/164 (https://phabricator.wikimedia.org/T389118) [04:05:18] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install cloudcontrol2010-dev - https://phabricator.wikimedia.org/T393102#10814692 (10Jhancock.wm) a:03Jhancock.wm [05:58:08] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1144540 (owner: 10L10n-bot) [06:51:03] 10Tool-campwiz-nxt, 06translatewiki.net, 10LPL Essential (LPL Essential 2025 Apr-Jun: CX), 07Unplanned-Sprint-Work: Add CampWiz NXT to translatewiki.net - https://phabricator.wikimedia.org/T393850#10814872 (10Nikerabbit) The format is supported, but we would prefer if you took language names from a library... [07:00:35] 06cloud-services-team, 10Toolforge: [components-api] add order to the components deployment - https://phabricator.wikimedia.org/T362075#10814877 (10dcaro) [07:14:03] 06cloud-services-team, 10Cloud-VPS, 10Beta-Cluster-Infrastructure: Route deployment-prep Prometheus alerts to the betacluster-alerts@lists.wikimedia.org mailing list - https://phabricator.wikimedia.org/T393975#10814921 (10taavi) a:03taavi >>! In T393975#10814317, @bd808 wrote: > I think what is needed here... [07:28:35] (03approved) 10taavi: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) (owner: 10aborrero) [07:34:00] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 [07:37:40] !log taavi@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-36 [07:37:54] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 [07:39:04] 10Tools, 07Privacy: jitrixis Tool Loads External Resource - https://phabricator.wikimedia.org/T334317#10814995 (10Aklapper) 05Open→03Invalid @komla: No reply; boldly closing. [07:43:12] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 [07:44:18] (03PS2) 10Majavah: inventory: Use VXLAN/IPv6-dualstack as internal network [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1142547 [07:51:15] (03PS1) 10Majavah: toolforge: k8s: Wait for host to come back up after hard reboot [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1145088 [07:54:41] (03CR) 10CI reject: [V:04-1] toolforge: k8s: Wait for host to come back up after hard reboot [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1145088 (owner: 10Majavah) [07:55:18] (03PS2) 10Majavah: toolforge: k8s: Wait for host to come back up after hard reboot [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1145088 [08:06:04] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-36 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:14:53] (03CR) 10Arturo Borrero Gonzalez: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1142547 (owner: 10Majavah) [08:17:09] (03CR) 10Majavah: [C:03+2] inventory: Use VXLAN/IPv6-dualstack as internal network [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1142547 (owner: 10Majavah) [08:21:20] (03Merged) 10jenkins-bot: inventory: Use VXLAN/IPv6-dualstack as internal network [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1142547 (owner: 10Majavah) [08:27:51] 10Tool-campwiz-nxt, 06translatewiki.net, 10LPL Essential (LPL Essential 2025 Apr-Jun: CX), 07Unplanned-Sprint-Work: Add CampWiz NXT to translatewiki.net - https://phabricator.wikimedia.org/T393850#10815186 (10Nokib_Sarkar) @Nikerabbit If you are saying that I should get the language codes from CLDR (like `... [08:46:15] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) [08:46:41] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) [08:49:14] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) [08:49:58] (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) [08:51:40] (03merge) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686) [08:52:43] (03update) 10aborrero: gitlab-ci: replace local logic with included one [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/23 (https://phabricator.wikimedia.org/T393686) [08:54:03] (03merge) 10aborrero: gitlab-ci: replace local logic with included one [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/23 (https://phabricator.wikimedia.org/T393686) [08:55:48] (03update) 10aborrero: gitlab-ci: replace local logic with included one [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/25 (https://phabricator.wikimedia.org/T393686) [08:57:02] (03merge) 10aborrero: gitlab-ci: replace local logic with included one [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/25 (https://phabricator.wikimedia.org/T393686) [09:10:15] (03open) 10aborrero: tofu-provisioning: report opentofu plan changes as job warning [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/57 (https://phabricator.wikimedia.org/T393686) [09:11:52] (03open) 10aborrero: Do not merge: commot to test pipeline changes [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/24 [09:12:00] (03update) 10aborrero: Do not merge: commot to test pipeline changes [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/24 [09:12:52] (03update) 10aborrero: Do not merge: commot to test pipeline changes [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/24 [09:15:18] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Create views for new wiki nupwiki - https://phabricator.wikimedia.org/T390714#10815403 (10fnegri) 05Resolved→03In progress @Pppery I was convinced I did run the DNS script a few days before adding the databases, but I must have miss... [09:18:44] (03update) 10aborrero: tofu-provisioning: report opentofu plan changes as job warning [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/57 (https://phabricator.wikimedia.org/T393686) [09:20:15] (03update) 10aborrero: Do not merge: commit to test pipeline changes [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/24 [09:22:07] (03update) 10aborrero: tofu-provisioning: report opentofu plan changes as job warning [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/57 (https://phabricator.wikimedia.org/T393686) [09:26:07] (03CR) 10Majavah: [C:03+2] Support specifying a protocol [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1143541 (https://phabricator.wikimedia.org/T274386) (owner: 10Majavah) [09:26:45] (03Merged) 10jenkins-bot: Support specifying a protocol [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1143541 (https://phabricator.wikimedia.org/T274386) (owner: 10Majavah) [09:28:24] (03open) 10arthurtaylor: Resolve lastSuccessfulBuild id to cache jobs by job_id [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/8 (https://phabricator.wikimedia.org/T392892) [09:33:54] (03open) 10aborrero: do not merge: fake commit to test pipeline changes [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/25 [09:37:19] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Create views for new wiki nupwiki - https://phabricator.wikimedia.org/T390714#10815445 (10fnegri) 05In progress→03Resolved Done, the DNS records are there now. ` 2025-05-13T09:18:02Z __main__ INFO : Syncing CNAME records... [09:37:35] (03approved) 10taavi: tofu-provisioning: report opentofu plan changes as job warning [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/57 (https://phabricator.wikimedia.org/T393686) (owner: 10aborrero) [09:37:46] (03merge) 10aborrero: tofu-provisioning: report opentofu plan changes as job warning [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/57 (https://phabricator.wikimedia.org/T393686) [09:38:03] (03close) 10aborrero: do not merge: fake commit to test pipeline changes [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/25 [09:38:42] (03close) 10aborrero: Do not merge: commit to test pipeline changes [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/24 [09:47:14] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [09:48:01] (03open) 10aborrero: tofu-provisioning: fix cached tofu plan [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/58 (https://phabricator.wikimedia.org/T393686) [09:49:31] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [09:51:19] (03merge) 10aborrero: tofu-provisioning: fix cached tofu plan [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/58 (https://phabricator.wikimedia.org/T393686) [09:52:23] 06cloud-services-team, 10Horizon: Allow encrypting connection from the shared Cloud VPS web proxy to the backend - https://phabricator.wikimedia.org/T274386#10815482 (10taavi) 05Open→03Resolved Horizon change deployed, and documented at https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reac... [10:02:02] (03open) 10aborrero: tofu-provisioning: don't cache opentofu plan across jobs [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/59 (https://phabricator.wikimedia.org/T393686) [10:02:12] (03merge) 10aborrero: tofu-provisioning: don't cache opentofu plan across jobs [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/59 (https://phabricator.wikimedia.org/T393686) [10:19:00] 06cloud-services-team, 10Cloud-VPS, 10Data-Engineering (Q4 2025 April 1st - June 30th), 07IPv6, 13Patch-For-Review: Add new WMCS IP ranges to analytics - https://phabricator.wikimedia.org/T392468#10815574 (10JAllemandou) a:03JAllemandou [10:20:46] 06cloud-services-team, 10Cloud-VPS, 10Data-Engineering (Q4 2025 April 1st - June 30th), 07IPv6, 13Patch-For-Review: Add new WMCS IP ranges to analytics - https://phabricator.wikimedia.org/T392468#10815577 (10JAllemandou) [10:29:17] (03update) 10arthurtaylor: Resolve lastSuccessfulBuild id to cache jobs by job_id [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/8 (https://phabricator.wikimedia.org/T392892) [10:32:58] (03update) 10arthurtaylor: Resolve lastSuccessfulBuild id to cache jobs by job_id [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/8 (https://phabricator.wikimedia.org/T392892) [10:47:56] (03update) 10arthurtaylor: Resolve lastSuccessfulBuild id to cache jobs by job_id [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/8 (https://phabricator.wikimedia.org/T392892) [10:48:38] (03open) 10aborrero: tofu-provisioning: drop providers.tf file logic [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/60 (https://phabricator.wikimedia.org/T393686) [10:49:34] (03open) 10aborrero: tofu-provisioning: drop providers.tf special logic [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/26 [10:50:33] (03update) 10aborrero: tofu-provisioning: drop providers.tf special logic [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/26 [10:52:53] (03PS1) 10Majavah: build: Update Poetry dependencies [labs/striker] - 10https://gerrit.wikimedia.org/r/1145139 [10:52:54] (03PS1) 10Majavah: dev(docker): Upgrade Keystone container [labs/striker] - 10https://gerrit.wikimedia.org/r/1145140 [10:52:54] (03PS1) 10Majavah: dev(docker): Add missing OpenLDAP dependency for Keystone [labs/striker] - 10https://gerrit.wikimedia.org/r/1145141 [10:53:20] (03approved) 10taavi: tofu-provisioning: drop providers.tf file logic [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/60 (https://phabricator.wikimedia.org/T393686) (owner: 10aborrero) [10:53:41] (03approved) 10taavi: tofu-provisioning: drop providers.tf special logic [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/26 (owner: 10aborrero) [10:54:18] (03open) 10aborrero: tofu-provisioning: drop providers.tf special logic [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/26 [10:54:29] (03merge) 10aborrero: tofu-provisioning: drop providers.tf file logic [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/60 (https://phabricator.wikimedia.org/T393686) [10:54:52] (03update) 10aborrero: tofu-provisioning: drop providers.tf special logic [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/26 [10:55:40] (03CR) 10CI reject: [V:04-1] dev(docker): Add missing OpenLDAP dependency for Keystone [labs/striker] - 10https://gerrit.wikimedia.org/r/1145141 (owner: 10Majavah) [10:55:44] (03CR) 10CI reject: [V:04-1] dev(docker): Upgrade Keystone container [labs/striker] - 10https://gerrit.wikimedia.org/r/1145140 (owner: 10Majavah) [10:55:58] (03update) 10aborrero: tofu-provisioning: drop providers.tf special logic [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/26 [10:56:15] (03update) 10aborrero: tofu-provisioning: drop providers.tf special logic [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/26 [10:56:21] (03merge) 10aborrero: tofu-provisioning: drop providers.tf special logic [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/26 [10:57:27] (03approved) 10lucaswerkmeister-wmde: Resolve lastSuccessfulBuild id to cache jobs by job_id [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/8 (https://phabricator.wikimedia.org/T392892) (owner: 10arthurtaylor) [10:57:54] (03approved) 10taavi: tofu-provisioning: drop providers.tf special logic [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/26 (owner: 10aborrero) [10:58:01] (03merge) 10aborrero: tofu-provisioning: drop providers.tf special logic [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/26 [11:01:02] (03open) 10aborrero: tools: relocate dns module call [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/27 [11:01:48] (03merge) 10aborrero: tools: relocate dns module call [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/27 [11:02:49] (03merge) 10arthurtaylor: Resolve lastSuccessfulBuild id to cache jobs by job_id [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/8 (https://phabricator.wikimedia.org/T392892) [11:03:04] (03PS2) 10Majavah: dev(docker): Upgrade Keystone container [labs/striker] - 10https://gerrit.wikimedia.org/r/1145140 [11:03:04] (03PS2) 10Majavah: dev(docker): Add missing OpenLDAP dependency for Keystone [labs/striker] - 10https://gerrit.wikimedia.org/r/1145141 [11:03:04] (03PS4) 10Majavah: Refresh user group membership after membership is granted [labs/striker] - 10https://gerrit.wikimedia.org/r/1009232 (https://phabricator.wikimedia.org/T144943) [11:03:04] (03PS1) 10Majavah: register: Fix undefined key access [labs/striker] - 10https://gerrit.wikimedia.org/r/1145143 [11:04:37] (03CR) 10Majavah: [C:03+2] build: Update Poetry dependencies [labs/striker] - 10https://gerrit.wikimedia.org/r/1145139 (owner: 10Majavah) [11:05:12] 10Striker, 13Patch-For-Review: Groups and tools only refreshed at login - https://phabricator.wikimedia.org/T144943#10815767 (10taavi) a:03taavi [11:05:32] 10Striker: Dev environment Keystone crashes - https://phabricator.wikimedia.org/T359972#10815768 (10taavi) a:03taavi The Keystone update done in https://gerrit.wikimedia.org/r/c/labs/striker/+/1145140 fixes the issue at least for me. [11:06:05] (03Merged) 10jenkins-bot: build: Update Poetry dependencies [labs/striker] - 10https://gerrit.wikimedia.org/r/1145139 (owner: 10Majavah) [11:08:20] (03update) 10arthurtaylor: Resolve lastSuccessfulBuild id to cache jobs by job_id [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/8 (https://phabricator.wikimedia.org/T392892) [11:11:51] (03update) 10aborrero: tofu-provisioning: drop imports [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/23 [11:31:26] (03open) 10lucaswerkmeister-wmde: Three minor fixups [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/9 [11:36:57] (03update) 10lucaswerkmeister-wmde: Three minor fixups [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/9 [11:41:41] (03close) 10aborrero: Draft: gitlab-ci: use msmtp to send emails [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/22 [11:42:54] (03approved) 10arthurtaylor: Three minor fixups [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/9 (owner: 10lucaswerkmeister-wmde) [11:44:51] (03open) 10aborrero: tofu-provisioning: introduce gitlab CI/CD based workflow [repos/cloud/metricsinfra/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/metricsinfra/tofu-provisioning/-/merge_requests/2 [11:45:25] (03update) 10aborrero: tofu-provisioning: introduce gitlab CI/CD based workflow [repos/cloud/metricsinfra/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/metricsinfra/tofu-provisioning/-/merge_requests/2 [11:58:28] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate Puppet CA: toolsbeta-puppetmaster-04.toolsbeta.eqiad.wmflabs is about to expire in 27d 23h 58m 43s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [12:03:27] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: tofu-provisioning: factorize gitlab pipeline logic - https://phabricator.wikimedia.org/T393686#10815956 (10aborrero) 05In progress→03Resolved a:03aborrero This is mostly completed. There are 2 repos using this setup: * https://gitlab.wikimedia... [12:12:53] 10Tool-campwiz-nxt, 06translatewiki.net, 10LPL Essential (LPL Essential 2025 Apr-Jun: CX), 07Unplanned-Sprint-Work: Add CampWiz NXT to translatewiki.net - https://phabricator.wikimedia.org/T393850#10815992 (10Nikerabbit) [12:14:24] (03PS3) 10Majavah: dev(docker): Upgrade Keystone container [labs/striker] - 10https://gerrit.wikimedia.org/r/1145140 (https://phabricator.wikimedia.org/T359972) [12:14:25] (03PS3) 10Majavah: dev(docker): Add missing OpenLDAP dependency for Keystone [labs/striker] - 10https://gerrit.wikimedia.org/r/1145141 [12:14:25] (03PS2) 10Majavah: register: Fix undefined key access [labs/striker] - 10https://gerrit.wikimedia.org/r/1145143 [12:14:25] (03PS5) 10Majavah: Refresh user group membership after membership is granted [labs/striker] - 10https://gerrit.wikimedia.org/r/1009232 (https://phabricator.wikimedia.org/T144943) [12:21:27] (03CR) 10Majavah: [C:03+2] dev(docker): Upgrade Keystone container [labs/striker] - 10https://gerrit.wikimedia.org/r/1145140 (https://phabricator.wikimedia.org/T359972) (owner: 10Majavah) [12:21:34] (03CR) 10Majavah: [C:03+2] dev(docker): Add missing OpenLDAP dependency for Keystone [labs/striker] - 10https://gerrit.wikimedia.org/r/1145141 (owner: 10Majavah) [12:22:54] (03Merged) 10jenkins-bot: dev(docker): Upgrade Keystone container [labs/striker] - 10https://gerrit.wikimedia.org/r/1145140 (https://phabricator.wikimedia.org/T359972) (owner: 10Majavah) [12:24:27] (03Merged) 10jenkins-bot: dev(docker): Add missing OpenLDAP dependency for Keystone [labs/striker] - 10https://gerrit.wikimedia.org/r/1145141 (owner: 10Majavah) [12:27:39] 10Striker, 13Patch-For-Review: Dev environment Keystone crashes - https://phabricator.wikimedia.org/T359972#10816034 (10taavi) 05Open→03Resolved [12:30:16] 06cloud-services-team, 10Toolforge: Recover missing .kube/config for Toolforge tool `inat2wiki` - https://phabricator.wikimedia.org/T394023 (10TiagoLubiana) 03NEW [12:31:33] 06cloud-services-team, 10Toolforge: Recover missing .kube/config for Toolforge tool `inat2wiki` - https://phabricator.wikimedia.org/T394023#10816090 (10TiagoLubiana) [12:32:22] 06cloud-services-team, 10Toolforge: Recover missing .kube/config for Toolforge tool `inat2wiki` - https://phabricator.wikimedia.org/T394023#10816096 (10taavi) 05Open→03Resolved a:03taavi `lang=shell-session taavi@tools-bastion-12:~ $ kubectl sudo delete cm -n tool-inat2wiki maintain-kubeusers-inat2wi... [12:32:36] (03CR) 10Majavah: [C:03+2] register: Fix undefined key access [labs/striker] - 10https://gerrit.wikimedia.org/r/1145143 (owner: 10Majavah) [12:34:06] (03Merged) 10jenkins-bot: register: Fix undefined key access [labs/striker] - 10https://gerrit.wikimedia.org/r/1145143 (owner: 10Majavah) [12:34:44] (03CR) 10Majavah: [C:03+2] "Fixed with Ib7fc2feb293a901d019528b14bcb823eff91ffe7." [labs/striker] - 10https://gerrit.wikimedia.org/r/1009232 (https://phabricator.wikimedia.org/T144943) (owner: 10Majavah) [12:36:15] (03Merged) 10jenkins-bot: Refresh user group membership after membership is granted [labs/striker] - 10https://gerrit.wikimedia.org/r/1009232 (https://phabricator.wikimedia.org/T144943) (owner: 10Majavah) [12:37:21] (03PS5) 10Majavah: Drop bootstrap-theme.min.css [labs/striker] - 10https://gerrit.wikimedia.org/r/1108154 [12:37:21] (03PS5) 10Majavah: Convert SSH key view to Codex [labs/striker] - 10https://gerrit.wikimedia.org/r/1108155 (https://phabricator.wikimedia.org/T380114) [12:40:41] (03CR) 10Majavah: [C:03+2] Drop bootstrap-theme.min.css [labs/striker] - 10https://gerrit.wikimedia.org/r/1108154 (owner: 10Majavah) [12:40:52] 10Tools: Start a rewrite the inat2wiki tool - https://phabricator.wikimedia.org/T392822#10816135 (10TiagoLubiana) 05Open→03Resolved [12:41:03] (03merge) 10lucaswerkmeister-wmde: Three minor fixups [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/9 [12:41:28] (03CR) 10Majavah: [C:03+2] Convert SSH key view to Codex [labs/striker] - 10https://gerrit.wikimedia.org/r/1108155 (https://phabricator.wikimedia.org/T380114) (owner: 10Majavah) [12:42:08] (03Merged) 10jenkins-bot: Drop bootstrap-theme.min.css [labs/striker] - 10https://gerrit.wikimedia.org/r/1108154 (owner: 10Majavah) [12:42:09] 06cloud-services-team, 10Toolforge: Recover missing .kube/config for Toolforge tool `inat2wiki` - https://phabricator.wikimedia.org/T394023#10816137 (10TiagoLubiana) Thank you, taavi !! [12:44:31] (03Merged) 10jenkins-bot: Convert SSH key view to Codex [labs/striker] - 10https://gerrit.wikimedia.org/r/1108155 (https://phabricator.wikimedia.org/T380114) (owner: 10Majavah) [12:46:23] (03PS4) 10Majavah: Upgrade to Django 3.2 LTS [labs/striker] - 10https://gerrit.wikimedia.org/r/1009594 (https://phabricator.wikimedia.org/T359217) [12:48:03] (03CR) 10CI reject: [V:04-1] Upgrade to Django 3.2 LTS [labs/striker] - 10https://gerrit.wikimedia.org/r/1009594 (https://phabricator.wikimedia.org/T359217) (owner: 10Majavah) [12:52:25] (03update) 10aborrero: kubecerts: stop renewing certs on a random basis [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/69 (https://phabricator.wikimedia.org/T365681) [13:16:44] (03approved) 10fnegri: kubecerts: stop renewing certs on a random basis [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/69 (https://phabricator.wikimedia.org/T365681) (owner: 10aborrero) [13:29:30] 10Cloud Services Proposals, 10Striker: [DRAFT] Decision request - Tool account management and Striker - https://phabricator.wikimedia.org/T394035 (10taavi) 03NEW [13:48:45] 10Cloud Services Proposals, 10Striker: [DRAFT] Decision request - Tool account management and Striker - https://phabricator.wikimedia.org/T394035#10816438 (10taavi) [13:50:27] 10Cloud Services Proposals, 10Striker: [DRAFT] Decision request - Tool account management and Striker - https://phabricator.wikimedia.org/T394035#10816446 (10taavi) [13:56:07] 10Cloud Services Proposals, 10Striker: [DRAFT] Decision request - Tool account management and Striker - https://phabricator.wikimedia.org/T394035#10816464 (10taavi) [13:58:22] 06cloud-services-team, 10Toolforge: [Session] Introducing and exploring Toolforge UI with prospective users - https://phabricator.wikimedia.org/T383149#10816482 (10Sarai-WMF) [14:04:44] (03PS5) 10Majavah: Upgrade to Django 3.2 LTS [labs/striker] - 10https://gerrit.wikimedia.org/r/1009594 (https://phabricator.wikimedia.org/T359217) [14:23:19] (03PS1) 10Majavah: Replace ugettext_lazy with gettext_lazy [labs/striker] - 10https://gerrit.wikimedia.org/r/1145211 [14:34:48] (03merge) 10aborrero: kubecerts: stop renewing certs on a random basis [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/69 (https://phabricator.wikimedia.org/T365681) [14:37:10] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-kubeusers: bump to 0.0.173-20250513143459-bcd0852b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/779 (https://phabricator.wikimedia.org/T365681) [14:57:49] 10Cloud Services Proposals, 10Striker: [DRAFT] Decision request - Tool account management and Striker - https://phabricator.wikimedia.org/T394035#10816904 (10aborrero) I can definitely see the benefit of the HTTP API, way easier to use than the LDAP interface. But also, I'm trying to think, how is exposing th... [15:05:39] (03update) 10ttaylor: Draft: Refactored project structure to add Python API to relay events [toolforge-repos/listen-to-wiki-changes] - 10https://gitlab.wikimedia.org/toolforge-repos/listen-to-wiki-changes/-/merge_requests/1 [15:19:10] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [15:21:51] (03open) 10bd808: wikimedia-releng: Autovoice wmcs-alerts [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/25 (https://phabricator.wikimedia.org/T393975) [15:26:34] (03merge) 10bd808: wikimedia-releng: Autovoice wmcs-alerts [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/25 (https://phabricator.wikimedia.org/T393975) [15:30:55] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [15:31:54] 10Cloud Services Proposals, 10Striker: [DRAFT] Decision request - Tool account management and Striker - https://phabricator.wikimedia.org/T394035#10817106 (10taavi) >>! In T394035#10816904, @aborrero wrote: > I can definitely see the benefit of the HTTP API, way easier to use than the LDAP interface. > > But... [15:33:35] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers [15:33:59] 10Cloud Services Proposals, 10Striker: [DRAFT] Decision request - Tool account management and Striker - https://phabricator.wikimedia.org/T394035#10817110 (10fnegri) Thanks @taavi for starting this discussion and describing all the options! I think two valuable goals are: * Uncoupling UI and backend logic * B... [15:37:45] 06cloud-services-team, 10Cloud-VPS, 10Beta-Cluster-Infrastructure: Route deployment-prep Prometheus alerts to the betacluster-alerts@lists.wikimedia.org mailing list - https://phabricator.wikimedia.org/T393975#10817123 (10bd808) >>! In T393975#10814921, @taavi wrote: > Please enable autovoice on the `wmcs-al... [15:39:09] 06cloud-services-team, 10Cloud-VPS, 10Beta-Cluster-Infrastructure: Route deployment-prep Prometheus alerts to the betacluster-alerts@lists.wikimedia.org mailing list - https://phabricator.wikimedia.org/T393975#10817127 (10taavi) 05Open→03Resolved [15:43:14] (03update) 10ttaylor: Draft: Refactored project structure to add Python API to relay events [toolforge-repos/listen-to-wiki-changes] - 10https://gitlab.wikimedia.org/toolforge-repos/listen-to-wiki-changes/-/merge_requests/1 [15:46:41] 06cloud-services-team, 10Cloud-VPS: OpenTofu vs. radosgw in codfw1dev - https://phabricator.wikimedia.org/T394061 (10Andrew) 03NEW [15:47:24] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers [15:49:00] (03merge) 10aborrero: maintain-kubeusers: bump to 0.0.173-20250513143459-bcd0852b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/779 (https://phabricator.wikimedia.org/T365681) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:52:06] 10Cloud Services Proposals, 10Striker: [DRAFT] Decision request - Tool account management and Striker - https://phabricator.wikimedia.org/T394035#10817188 (10taavi) >>! In T394035#10817110, @fnegri wrote: > In option Purple, would it be possible to deploy the new "LDAP API" service to wikiprod hardware Yes. >... [15:52:27] 10Cloud Services Proposals, 10Striker: Decision request - Tool account management and Striker - https://phabricator.wikimedia.org/T394035#10817190 (10taavi) [16:35:41] (03PS1) 10Andrew Bogott: Dummy passwords for upcoming octavia install [labs/private] - 10https://gerrit.wikimedia.org/r/1145262 (https://phabricator.wikimedia.org/T393783) [16:37:03] (03PS2) 10Andrew Bogott: Dummy passwords for upcoming octavia install [labs/private] - 10https://gerrit.wikimedia.org/r/1145262 (https://phabricator.wikimedia.org/T393783) [16:52:02] 10Cloud Services Proposals, 10Striker: Decision request - Tool account management and Striker - https://phabricator.wikimedia.org/T394035#10817585 (10fnegri) > it's better to make a Toolforge-specific service than to make something that's in theory generic but in practice only used by us Agreed! But we should... [16:59:15] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge (Toolforge iteration 19): [toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2025-05-09 - https://phabricator.wikimedia.org/T393766#10817608 (10fnegri) The replica is still processing the same transaction. Judging from the output of `SHOW ENGINE INNODB STATUS... [17:20:39] (03update) 10ttaylor: Draft: Refactored project structure to add Python API to relay events [toolforge-repos/listen-to-wiki-changes] - 10https://gitlab.wikimedia.org/toolforge-repos/listen-to-wiki-changes/-/merge_requests/1 [17:21:17] (03update) 10ttaylor: Draft: Refactored project structure to add Python API to relay events [toolforge-repos/listen-to-wiki-changes] - 10https://gitlab.wikimedia.org/toolforge-repos/listen-to-wiki-changes/-/merge_requests/1 [17:25:28] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-bastion-13 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [17:58:15] 10Cloud Services Proposals, 10Striker: Decision request - Tool account management and Striker - https://phabricator.wikimedia.org/T394035#10817989 (10dcaro) I was holding back on deciding this until it was blocking some work, I guess that it has started blocking :) Some random comments. > Agreed! But we shou... [19:21:07] (03CR) 10Andrew Bogott: [V:03+2 C:03+2] Dummy passwords for upcoming octavia install [labs/private] - 10https://gerrit.wikimedia.org/r/1145262 (https://phabricator.wikimedia.org/T393783) (owner: 10Andrew Bogott) [19:30:36] 10Striker: Move `make watch` vite builder functionality into always-on container - https://phabricator.wikimedia.org/T394092 (10bd808) 03NEW [19:47:34] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install cloudvirt10[68-76] - https://phabricator.wikimedia.org/T382492#10818294 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudvirt1076.eqiad.wmnet with OS bookworm [20:06:11] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install cloudvirt10[68-76] - https://phabricator.wikimedia.org/T382492#10818324 (10Jhancock.wm) [20:20:00] 06cloud-services-team, 10Cloud-VPS: Understand Octavia network needs - https://phabricator.wikimedia.org/T394099 (10Andrew) 03NEW [20:21:00] 06cloud-services-team, 10Cloud-VPS: Understand Octavia network needs - https://phabricator.wikimedia.org/T394099#10818393 (10Andrew) @aborrero can you read the above (and probably the associated docs) and confirm that this can be done securely with our existing network model? Bonus points if you want to write... [20:21:04] 06cloud-services-team, 10Cloud-VPS: Understand Octavia network needs - https://phabricator.wikimedia.org/T394099#10818394 (10Andrew) a:05Andrew→03aborrero [20:24:37] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install cloudvirt10[68-76] - https://phabricator.wikimedia.org/T382492#10818403 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudvirt1076.eqiad.wmnet with OS bookworm complet... [20:56:56] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install cloudvirt10[68-76] - https://phabricator.wikimedia.org/T382492#10818504 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudvirt1075.eqiad.wmnet with OS bookworm [20:57:27] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install cloudvirt10[68-76] - https://phabricator.wikimedia.org/T382492#10818517 (10Jhancock.wm) [21:04:41] (03update) 10raymond-ndibe: [toolforge-deploy] run specific tests on deploy [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/755 (https://phabricator.wikimedia.org/T381011) [21:11:19] (03update) 10raymond-ndibe: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) (owner: 10dcaro) [21:15:35] (03Abandoned) 10BryanDavis: dev: Allow insecure apt actions in Keystone and MediaWiki Docker [labs/striker] - 10https://gerrit.wikimedia.org/r/1077474 (owner: 10BryanDavis) [21:45:24] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install cloudvirt10[68-76] - https://phabricator.wikimedia.org/T382492#10818632 (10VRiley-WMF) [21:45:41] (03CR) 10BryanDavis: [V:03+1 C:03+1] "I ran through creating an account, applying for Toolforge membership, approving a membership request, creating a tool, adding a co-maintai" [labs/striker] - 10https://gerrit.wikimedia.org/r/1009594 (https://phabricator.wikimedia.org/T359217) (owner: 10Majavah) [21:45:54] (03CR) 10BryanDavis: [V:03+1 C:03+1] Replace ugettext_lazy with gettext_lazy [labs/striker] - 10https://gerrit.wikimedia.org/r/1145211 (owner: 10Majavah) [22:20:28] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-bastion-13 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [22:27:33] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install cloudvirt10[68-76] - https://phabricator.wikimedia.org/T382492#10818731 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudvirt1075.eqiad.wmnet with OS bookworm complet...