[04:26:28] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [04:26:39] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [05:38:27] 10Quarry: Quarry down - web service unreachable - https://phabricator.wikimedia.org/T395201#10898038 (10Liz) Okay, this glitch ot fixed late last night but now it's down again! What is the problem here? [05:44:51] (03CR) 10Abijeet Patro: "recheck" [labs/tools/intuition-web] - 10https://gerrit.wikimedia.org/r/1154813 (owner: 10L10n-bot) [06:02:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [06:53:43] (03CR) 10Abijeet Patro: [C:03+1] Localisation updates from https://translatewiki.net. [labs/tools/intuition-web] - 10https://gerrit.wikimedia.org/r/1154813 (owner: 10L10n-bot) [07:11:18] 10Quarry: quarry is leaking tmp files - https://phabricator.wikimedia.org/T395237#10898144 (10dcaro) Saw another instance of this today, it was paired with redis being down (shown as 'completed', so the pod did not restart by itself). [07:13:26] 10Quarry: Quarry down - web service unreachable - https://phabricator.wikimedia.org/T395201#10898154 (10dcaro) >>! In T395201#10898038, @Liz wrote: > Okay, this glitch got fixed late last night but now it's down again! What is the problem here? > > Edit: @taavi, can you look at this? Just manually restarted r... [07:15:44] 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [jobs-api] expose health_check.type deprecation metrics - https://phabricator.wikimedia.org/T396236#10898165 (10dcaro) p:05Triage→03High [07:15:52] 10Toolforge (Toolforge iteration 20): [jobs-api] bug in runtime diff_with_running_job function - https://phabricator.wikimedia.org/T394734#10898166 (10dcaro) p:05Triage→03Medium [07:17:25] (03update) 10dcaro: [deploy] support health-checks and port [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/75 (https://phabricator.wikimedia.org/T362072) (owner: 10raymond-ndibe) [07:21:20] 10Quarry: Quarry down - web service unreachable - https://phabricator.wikimedia.org/T395201#10898170 (10Liz) Okay, it's working again. Let's hope this doesn't happen again tomorrow. Thanks! [07:23:16] 10Quarry: Quarry down - web service unreachable - https://phabricator.wikimedia.org/T395201#10898172 (10taavi) 05Open→03Resolved [07:25:24] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [07:26:58] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:45:23] (03update) 10dcaro: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) (owner: 10raymond-ndibe) [07:45:42] (03merge) 10taavi: Don’t show tool name in element [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/13 (owner: 10lucaswerkmeister) [07:57:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:02:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:07:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:12:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [08:14:28] dependabot[bot] opened https://github.com/toolforge/paws/pull/487 [08:50:13] (03update) 10raymond-ndibe: [components.deploy_task] fix do_run image_name bug [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/81 (https://phabricator.wikimedia.org/T395533) [08:51:33] (03update) 10raymond-ndibe: [components.deployment.create] add force-build and force-run option [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/33 (https://phabricator.wikimedia.org/T389044) [08:52:48] 06cloud-services-team, 10Cloud-VPS: Support IPv6 for Cloud VPS DNS services - https://phabricator.wikimedia.org/T396448 (10taavi) 03NEW p:05Triage→03Medium [08:52:49] (03update) 10raymond-ndibe: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] (refactor_components_smoke_test) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) [08:52:57] 06cloud-services-team, 10Cloud-VPS: Support IPv6 for Cloud VPS DNS services - https://phabricator.wikimedia.org/T396448#10898498 (10taavi) [08:52:59] 06cloud-services-team, 10Cloud-VPS, 07IPv6: IPv6 for cloud-realm services - https://phabricator.wikimedia.org/T379282#10898499 (10taavi) [08:53:12] 06cloud-services-team, 10Cloud-VPS, 07IPv6: Support IPv6 for Cloud VPS DNS services - https://phabricator.wikimedia.org/T396448#10898503 (10taavi) [08:56:01] 06cloud-services-team, 10Cloud-VPS, 07IPv6: Support IPv6 for OpenStack API endpoints - https://phabricator.wikimedia.org/T396450 (10taavi) 03NEW p:05Triage→03Medium [08:56:13] 06cloud-services-team, 10Cloud-VPS, 07IPv6: Support IPv6 for OpenStack API endpoints - https://phabricator.wikimedia.org/T396450#10898586 (10taavi) [08:56:14] 06cloud-services-team, 10Cloud-VPS, 07IPv6: IPv6 for cloud-realm services - https://phabricator.wikimedia.org/T379282#10898587 (10taavi) [08:58:06] 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 07IPv6: Support IPv6 for Wiki Replicas cloudlb endpoints - https://phabricator.wikimedia.org/T396451 (10taavi) 03NEW p:05Triage→03Low [08:58:15] 06cloud-services-team, 10Cloud-VPS, 07IPv6: IPv6 for cloud-realm services - https://phabricator.wikimedia.org/T379282#10898600 (10taavi) [08:58:15] 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 07IPv6: Support IPv6 for Wiki Replicas cloudlb endpoints - https://phabricator.wikimedia.org/T396451#10898601 (10taavi) [08:58:58] 06cloud-services-team, 10Cloud-VPS, 07IPv6: IPv6 for cloud-realm services - https://phabricator.wikimedia.org/T379282#10898603 (10taavi) 05Open→03Resolved Closing. The infrastructure for these is in place and I've filed separate parent tasks for rolling out v6 support for individual services. [09:03:36] (03update) 10dcaro: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) (owner: 10raymond-ndibe) [09:05:40] 06cloud-services-team, 10Cloud-VPS, 07IPv6: Support IPv6 for OpenStack API endpoints - https://phabricator.wikimedia.org/T396450#10898633 (10taavi) a:03taavi [09:21:28] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 2 deleted instances on project-proxy-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [09:40:19] 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [components-api] Add alerts and runbooks for basic service health - https://phabricator.wikimedia.org/T394275#10898820 (10Raymond_Ndibe) https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview?orgId=1&from=now-6h&to=now&timezone=utc&var-clust... [09:44:52] 06cloud-services-team, 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [functional-tests,deploy,cookbook] Run only selected tests when deploying a component - https://phabricator.wikimedia.org/T381011#10898827 (10Raymond_Ndibe) 05In progress→03Resolved [09:51:09] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10898866 (10fnegri) [09:51:40] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10898868 (10fnegri) clouddb1019 upgraded and repooled. [10:02:46] (03update) 10dcaro: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) (owner: 10raymond-ndibe) [10:06:28] (03update) 10dcaro: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) (owner: 10raymond-ndibe) [10:06:46] (03update) 10dcaro: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) (owner: 10raymond-ndibe) [10:08:41] FIRING: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:09:17] (03update) 10dcaro: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) (owner: 10raymond-ndibe) [10:11:01] (03update) 10dcaro: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) (owner: 10raymond-ndibe) [10:27:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [10:32:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [10:46:24] 06cloud-services-team, 10Cloud-VPS, 07IPv6: IPv6 support in cloud-private - https://phabricator.wikimedia.org/T379283#10899070 (10taavi) 05Open→03Resolved a:03taavi [11:33:55] (03update) 10dcaro: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) (owner: 10raymond-ndibe) [11:36:30] (03approved) 10dcaro: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) (owner: 10raymond-ndibe) [11:36:45] (03merge) 10dcaro: [components-smoke-test] use sourcebuild in tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/798 (https://phabricator.wikimedia.org/T389044 https://phabricator.wikimedia.org/T395533) (owner: 10raymond-ndibe) [11:36:47] (03update) 10dcaro: [components-smoke-test] add components-api conditional build and run tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/797 (https://phabricator.wikimedia.org/T389044) (owner: 10raymond-ndibe) [11:37:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1041 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [11:38:56] FIRING: SystemdUnitDown: The service unit labs-ip-alias-dump.service is in failed status on host cloudservices1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudservices1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [11:53:56] RESOLVED: SystemdUnitDown: The service unit labs-ip-alias-dump.service is in failed status on host cloudservices1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudservices1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [11:57:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1041 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [12:02:54] (03update) 10dcaro: [components.deploy_task] fix do_run image_name bug [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/81 (https://phabricator.wikimedia.org/T395533) (owner: 10raymond-ndibe) [12:07:28] (03update) 10fnegri: components-api: bump to 0.0.112-20250609142128-f12ef86d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/805 (https://phabricator.wikimedia.org/T394273 https://phabricator.wikimedia.org/T394277) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:10:13] !log fnegri@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api (T394277) [12:10:18] T394277: [components-cli,components-api] Add a warning message saying it's 'beta' - https://phabricator.wikimedia.org/T394277 [12:11:06] (03update) 10dcaro: [components.deploy_task] fix do_run image_name bug [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/81 (https://phabricator.wikimedia.org/T395533) (owner: 10raymond-ndibe) [12:11:06] (03approved) 10dcaro: [components.deploy_task] fix do_run image_name bug [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/81 (https://phabricator.wikimedia.org/T395533) (owner: 10raymond-ndibe) [12:11:22] (03merge) 10dcaro: [components.deploy_task] fix do_run image_name bug [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/81 (https://phabricator.wikimedia.org/T395533) (owner: 10raymond-ndibe) [12:11:30] (03update) 10dcaro: [deploy] skip build if refs are same [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/77 (https://phabricator.wikimedia.org/T389044) (owner: 10raymond-ndibe) [12:13:48] 06cloud-services-team, 10Cloud-VPS, 07IPv6: Support IPv6 for OpenStack API endpoints - https://phabricator.wikimedia.org/T396450#10899333 (10taavi) 05Open→03Resolved [12:13:58] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.113-20250610121140-bcc18922 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/805 (https://phabricator.wikimedia.org/T394273 https://phabricator.wikimedia.org/T394277 https://phabricator.wikimedia.org/T395533) [12:14:03] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.113-20250610121140-bcc18922 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/805 (https://phabricator.wikimedia.org/T394273 https://phabricator.wikimedia.org/T394277 https://phabricator.wikimedia.org/T395533) [12:17:39] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: Enable IPv6 for the Cloud VPS web proxy - https://phabricator.wikimedia.org/T379175#10899348 (10taavi) Closing, as T379283 meant I could fix the NFS edge cases and the remaining one (UTRS) is tracked elsewhere. [12:19:32] 06cloud-services-team, 10Cloud-VPS, 10DNS: Move some of wikimediacloud.org 185.15.56.0/23 to Netbox - https://phabricator.wikimedia.org/T268621#10899370 (10taavi) [12:21:42] !log fnegri@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api (T394277) [12:21:47] T394277: [components-cli,components-api] Add a warning message saying it's 'beta' - https://phabricator.wikimedia.org/T394277 [12:29:14] (03PS1) 10David Caro: toolforge.component.deploy: run only component tests by default [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1155205 (https://phabricator.wikimedia.org/T389044) [12:29:22] 06cloud-services-team, 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [functional-tests,deploy,cookbook] Run only selected tests when deploying a component - https://phabricator.wikimedia.org/T381011#10899390 (10dcaro) 05Resolved→03In progress [12:29:33] 06cloud-services-team, 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [functional-tests,deploy,cookbook] Run only selected tests when deploying a component - https://phabricator.wikimedia.org/T381011#10899393 (10dcaro) a:05Raymond_Ndibe→03dcaro [12:32:43] (03CR) 10CI reject: [V:04-1] toolforge.component.deploy: run only component tests by default [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1155205 (https://phabricator.wikimedia.org/T389044) (owner: 10David Caro) [12:43:59] 06cloud-services-team, 10Cloud-VPS, 07IPv6: IPv6 for cloud-realm services - https://phabricator.wikimedia.org/T379282#10899451 (10taavi) [12:44:04] 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 07IPv6, 13Patch-For-Review: Support IPv6 for Wiki Replicas cloudlb endpoints - https://phabricator.wikimedia.org/T396451#10899452 (10taavi) a:03taavi [12:56:41] (03PS2) 10David Caro: toolforge.component.deploy: run only component tests by default [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1155205 (https://phabricator.wikimedia.org/T389044) [13:10:26] 06cloud-services-team, 10Cloud-VPS, 07IPv6: Support IPv6 for Cloud VPS DNS services - https://phabricator.wikimedia.org/T396448#10899560 (10taavi) a:03taavi [13:11:30] (03update) 10dcaro: [deploy] skip build if refs are same [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/77 (https://phabricator.wikimedia.org/T389044) (owner: 10raymond-ndibe) [13:27:05] (03PS1) 10Kamila Součková: Add fake hcaptcha proxy secrets. [labs/private] - 10https://gerrit.wikimedia.org/r/1155221 (https://phabricator.wikimedia.org/T381265) [13:28:36] !log dcaro@acme toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [13:28:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [13:32:00] !log dcaro@acme toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [13:32:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [13:33:43] (03update) 10dcaro: [deploy] skip build if refs are same [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/77 (https://phabricator.wikimedia.org/T389044) (owner: 10raymond-ndibe) [13:36:36] (03PS3) 10David Caro: toolforge.component.deploy: run only component tests by default [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1155205 (https://phabricator.wikimedia.org/T389044) [13:36:52] !log dcaro@acme toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [13:36:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [13:38:28] !log dcaro@acme toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [13:38:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [13:45:34] (03CR) 10David Caro: "Tested deploying in toolsbeta, see the output here:" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1155205 (https://phabricator.wikimedia.org/T389044) (owner: 10David Caro) [13:46:22] (03approved) 10dcaro: components-api: bump to 0.0.113-20250610121140-bcc18922 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/805 (https://phabricator.wikimedia.org/T394273 https://phabricator.wikimedia.org/T394277 https://phabricator.wikimedia.org/T395533) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:46:29] (03merge) 10dcaro: components-api: bump to 0.0.113-20250610121140-bcc18922 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/805 (https://phabricator.wikimedia.org/T394273 https://phabricator.wikimedia.org/T394277 https://phabricator.wikimedia.org/T395533) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:47:01] 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [components-api] "No such image" error when running with prebuilt image - https://phabricator.wikimedia.org/T395533#10899766 (10dcaro) 05In progress→03Resolved [13:48:06] 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [components-api] add tool config version check - https://phabricator.wikimedia.org/T394273#10899777 (10dcaro) 05In progress→03Resolved [13:48:21] (03update) 10dcaro: [components.tool_router] add example config endpoint [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/82 (https://phabricator.wikimedia.org/T394753) (owner: 10raymond-ndibe) [13:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 4 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:15:28] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [14:23:11] (03CR) 10FNegri: [C:03+1] toolforge.component.deploy: run only component tests by default (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1155205 (https://phabricator.wikimedia.org/T389044) (owner: 10David Caro) [14:25:28] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [14:54:52] !log dcaro@acme toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [14:54:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:56:39] !log dcaro@acme toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [14:56:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:57:29] !log dcaro@acme tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [14:57:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:59:42] !log dcaro@acme tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [14:59:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:04:19] !log dcaro@acme toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [15:04:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:10:15] !log dcaro@acme toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [15:10:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:15:16] !log dcaro@acme tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission [15:15:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:18:21] (03update) 10dcaro: jobs-api: bump to 0.0.380-20250602174717-b1b0f757 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/808 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:21:12] !log dcaro@acme tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission [15:21:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:22:11] !log dcaro@acme toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:22:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:24:49] (03PS4) 10David Caro: toolforge.component.deploy: run only component tests by default [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1155205 (https://phabricator.wikimedia.org/T389044) [15:24:49] (03CR) 10David Caro: toolforge.component.deploy: run only component tests by default (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1155205 (https://phabricator.wikimedia.org/T389044) (owner: 10David Caro) [15:25:44] (03PS5) 10David Caro: toolforge.component.deploy: run only component tests by default [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1155205 (https://phabricator.wikimedia.org/T389044) [15:29:15] !log dcaro@acme toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:29:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:33:02] !log dcaro@acme tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:33:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:34:13] (03open) 10andrew: Manage default security group in Octavia [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/245 (https://phabricator.wikimedia.org/T393782) [15:34:16] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/245 [15:34:16] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/245 [15:39:19] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10900430 (10Alien333) Note: it looks like the new MariaDB version errors out on some selects involving assigm... [15:39:57] (03open) 10andrew: Replace cloudcontrol2004-dev with cloudcontrol2010-dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/246 [15:40:07] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/246 [15:40:07] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/246 [15:43:13] (03CR) 10David Caro: "Just a couple questions/nit, +1 if the question makes no sense :)" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1154802 (https://phabricator.wikimedia.org/T386480) (owner: 10Majavah) [15:43:16] (03PS1) 10Andrew Bogott: Replace use of cloudcontrol2004-dev with cloudcontrol2010-dev [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1155268 (https://phabricator.wikimedia.org/T396064) [15:45:08] !log dcaro@acme tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:45:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:48:07] (03approved) 10dcaro: registry-admission: local: Exempt local-path-storage [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/795 (owner: 10taavi) [15:50:40] (03CR) 10David Caro: [C:03+2] "Rephrased 😊, example:" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1155205 (https://phabricator.wikimedia.org/T389044) (owner: 10David Caro) [15:50:57] (03approved) 10dcaro: jobs-api: bump to 0.0.380-20250602174717-b1b0f757 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/808 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:51:01] (03merge) 10dcaro: jobs-api: bump to 0.0.380-20250602174717-b1b0f757 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/808 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:51:55] (03CR) 10David Caro: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1155268 (https://phabricator.wikimedia.org/T396064) (owner: 10Andrew Bogott) [15:52:39] (03CR) 10Andrew Bogott: [C:03+2] Replace use of cloudcontrol2004-dev with cloudcontrol2010-dev [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1155268 (https://phabricator.wikimedia.org/T396064) (owner: 10Andrew Bogott) [15:54:23] (03Merged) 10jenkins-bot: toolforge.component.deploy: run only component tests by default [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1155205 (https://phabricator.wikimedia.org/T389044) (owner: 10David Caro) [15:56:53] (03Merged) 10jenkins-bot: Replace use of cloudcontrol2004-dev with cloudcontrol2010-dev [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1155268 (https://phabricator.wikimedia.org/T396064) (owner: 10Andrew Bogott) [15:57:06] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/246 [15:57:06] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/246 [15:57:27] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.59-20250602174644-958a182c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/809 [15:58:43] (03open) 10chuckonwumelu: [toolforge-weld] Format the warning message for the beta [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/79 (https://phabricator.wikimedia.org/T394277) [15:58:45] !log dcaro@acme toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [15:58:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:59:09] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/246 [15:59:28] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/246 [16:00:14] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [16:00:15] (03merge) 10andrew: Replace cloudcontrol2004-dev with cloudcontrol2010-dev [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/246 [16:01:00] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [16:01:17] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/245 [16:01:35] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/245 [16:01:48] !log dcaro@acme toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer [16:01:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:01:53] !log dcaro@acme toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [16:01:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:02:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/245 [16:02:21] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/245 [16:03:03] (03update) 10andrew: Manage default security group in Octavia [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/245 (https://phabricator.wikimedia.org/T393782) [16:03:31] (03merge) 10andrew: Manage default security group in Octavia [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/245 (https://phabricator.wikimedia.org/T393782) [16:03:32] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [16:03:35] 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 07IPv6: Support IPv6 for Wiki Replicas cloudlb endpoints - https://phabricator.wikimedia.org/T396451#10900549 (10taavi) 05Open→03Resolved [16:04:19] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [16:04:40] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [16:05:15] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [16:05:37] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [16:06:12] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan+apply for main branch [16:06:51] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [16:07:29] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [16:12:35] !log dcaro@acme toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [16:12:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:21:12] !log dcaro@acme tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [16:21:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:26:13] !log dcaro@acme tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer [16:26:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:28:00] !log dcaro@acme tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [16:28:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:41:57] !log dcaro@acme tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [16:42:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:44:08] (03approved) 10dcaro: jobs-emailer: bump to 0.0.59-20250602174644-958a182c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/809 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:44:10] (03update) 10dcaro: jobs-emailer: bump to 0.0.59-20250602174644-958a182c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/809 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:44:16] (03merge) 10dcaro: jobs-emailer: bump to 0.0.59-20250602174644-958a182c [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/809 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:53:29] 10Toolforge (Toolforge iteration 20): [maintain-harbor,ci] the version number does not get bumped on every release - https://phabricator.wikimedia.org/T396504 (10dcaro) 03NEW [16:53:30] !log dcaro@acme toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [16:53:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:53:53] !log dcaro@acme toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor [16:53:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:54:16] !log dcaro@acme toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [16:54:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:57:12] !log dcaro@acme toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [16:57:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:59:34] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [17:00:27] !log dcaro@acme tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [17:00:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:03:46] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [17:04:15] !log dcaro@acme tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [17:04:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:05:41] 06cloud-services-team, 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [functional-tests,deploy,cookbook] Run only selected tests when deploying a component - https://phabricator.wikimedia.org/T381011#10900945 (10dcaro) 05In progress→03Resolved [17:33:46] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [17:36:28] 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [components-cli,components-api, toolforge_weld] Add a warning message saying it's 'beta' - https://phabricator.wikimedia.org/T394277#10901111 (10Chuckonwumelu) [17:37:22] 10Toolforge (Toolforge iteration 20), 13Patch-For-Review: [components-cli,components-api, toolforge-weld] Add a warning message saying it's 'beta' - https://phabricator.wikimedia.org/T394277#10901113 (10Chuckonwumelu) [17:42:11] (03approved) 10andrew: [toolforge-weld] Format the warning message for the beta [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/79 (https://phabricator.wikimedia.org/T394277) (owner: 10chuckonwumelu) [17:51:01] (03merge) 10chuckonwumelu: [toolforge-weld] Format the warning message for the beta [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/79 (https://phabricator.wikimedia.org/T394277) [18:14:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [18:49:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [18:49:33] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [18:59:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [18:59:48] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [19:54:33] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-58 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [20:40:03] (03PS2) 10Kamila Součková: Add fake hcaptcha proxy secrets. [labs/private] - 10https://gerrit.wikimedia.org/r/1155221 (https://phabricator.wikimedia.org/T381265) [22:58:36] 10Cloud-VPS (Project-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade): Request creation of zuul-runners VPS project - https://phabricator.wikimedia.org/T396540 (10bd808) 03NEW [22:58:43] 10Cloud-VPS (Project-requests), 10Continuous-Integration-Infrastructure (Zuul upgrade): Request creation of zuul-runners VPS project - https://phabricator.wikimedia.org/T396540#10902235 (10bd808) [23:41:36] (03approved) 10bd808: Use toolforge envvars if available [toolforge-repos/python-toolforge] - 10https://gitlab.wikimedia.org/toolforge-repos/python-toolforge/-/merge_requests/27 (https://phabricator.wikimedia.org/T339940) (owner: 10lucaswerkmeister) [23:42:29] (03merge) 10bd808: Use toolforge envvars if available [toolforge-repos/python-toolforge] - 10https://gitlab.wikimedia.org/toolforge-repos/python-toolforge/-/merge_requests/27 (https://phabricator.wikimedia.org/T339940) (owner: 10lucaswerkmeister) [23:48:55] (03open) 10lucaswerkmeister: Release 6.1.0 [toolforge-repos/python-toolforge] - 10https://gitlab.wikimedia.org/toolforge-repos/python-toolforge/-/merge_requests/28 (https://phabricator.wikimedia.org/T333727 https://phabricator.wikimedia.org/T333728 https://phabricator.wikimedia.org/T339940 https://phabricator.wikimedia.org/T396115)