[00:15:28] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/34 [00:15:32] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/34 [00:15:50] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/28 [00:15:54] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/28 [00:16:43] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [00:16:48] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [00:17:33] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/19 [00:17:36] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/19 [02:23:04] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Create Cloud VPS VM Migration Guide - https://phabricator.wikimedia.org/T429635#12063609 (10komla) 05In progress→03Resolved [02:24:08] 10Cloud-VPS (Debian Bullseye Deprecation), 06tools-platform-team: Create Cloud VPS VM Migration Guide - https://phabricator.wikimedia.org/T429635#12063610 (10komla) [[ https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_instance_operating_system_migration | Published ]] guide. [05:20:58] (03update) 10countcount: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/19 (owner: 10renovatebot) [05:24:18] (03update) 10countcount: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/19 (owner: 10renovatebot) [05:26:42] (03merge) 10countcount: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/19 (owner: 10renovatebot) [05:50:32] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:54:43] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [05:55:32] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:59:43] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [06:00:32] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [06:33:15] 10Cloud-Services, 06collaboration-services, 10Continuous-Integration-Infrastructure (Zuul upgrade): Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479 (10hashar) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https:/... [06:34:45] 06cloud-services-team, 10Data-Services, 06collaboration-services, 10Continuous-Integration-Infrastructure (Zuul upgrade): Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12063721 (10hashar) [06:57:32] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:02:32] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:14:32] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:19:32] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:38:24] (03CR) 10Filippo Giunchedi: [C:03+1] wmcs.nfs.migrate_service: allow overriding the dest port name [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1305914 (owner: 10Andrew Bogott) [07:42:15] 06cloud-services-team, 10Data-Services, 06collaboration-services, 10Continuous-Integration-Infrastructure (Zuul upgrade): Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12063909 (10hashar) From zuul1002.eqiad.wmnet I can reproduce using curl. I have tried two app... [07:44:32] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:44:46] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06tools-infrastructure-team: Put cloudvirt10[77-80] in service - https://phabricator.wikimedia.org/T429563#12063918 (10fgiunchedi) [07:46:37] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12063929 (10Marostegui) >>! In T409557#12054195, @Marostegui wrote: > After pooling clouddb1026 with the same weight as clouddb1013, there was n... [07:49:32] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [08:16:17] FIRING: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [08:16:38] 06cloud-services-team, 10Data-Services, 06collaboration-services, 10Continuous-Integration-Infrastructure (Zuul upgrade): Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12064055 (10hashar) If I try https://openstack.eqiad1.wikimediacloud.org:25000/ from contint10... [08:19:28] 10Quarry, 06tools-platform-team: [quarry] flaky during the weekend - https://phabricator.wikimedia.org/T430486 (10dcaro) 03NEW [08:19:38] 10Toolforge, 06tools-platform-team: [infra] ToolforgeWebHighErrorRate should not page if a single tool is down - https://phabricator.wikimedia.org/T429738#12064077 (10dcaro) [08:19:48] 10Toolforge, 06tools-platform-team: [infra,o11y] ToolforgeWebHighErrorRate should not page if a single tool is down - https://phabricator.wikimedia.org/T429738#12064082 (10dcaro) [08:20:50] 06cloud-services-team, 10Data-Services, 06collaboration-services, 10Continuous-Integration-Infrastructure (Zuul upgrade): Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12064092 (10hashar) Using webproxy (rather than url-downloader), the tunneling is rejected: `... [08:20:57] 06cloud-services-team, 10Data-Services, 06collaboration-services, 10Continuous-Integration-Infrastructure (Zuul upgrade): Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12064096 (10hashar) [08:29:50] 06cloud-services-team, 10Data-Services, 06collaboration-services, 10Continuous-Integration-Infrastructure (Zuul upgrade): Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12064131 (10hashar) [08:33:16] 10Quarry, 06tools-platform-team: [quarry] flaky during the weekend - https://phabricator.wikimedia.org/T430486#12064157 (10dcaro) I don't see any logs newer than Jun-26th on the web pods, and nothing on the worker pods either. [08:43:37] 06cloud-services-team, 10Data-Services, 06collaboration-services, 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12064239 (10hashar) a:03hashar [08:46:17] RESOLVED: PrometheusRestarted: Prometheus instance tools-prometheus-9:9902 restarted - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPrometheusRestarted [08:54:34] 10Quarry, 06tools-platform-team: [quarry] flaky during the weekend - https://phabricator.wikimedia.org/T430486#12064330 (10dcaro) Added some graps to the quarry dashboard https://grafana-rw.wmcloud.org/d/85040a6f-6ede-4970-aaa4-53d4becb7f12/quarry-overview?orgId=1&from=now-12h&to=now&timezone=utc, indeed the u... [08:54:56] FIRING: [6x] SystemdUnitDown: The service unit libvirtd-admin.socket is in failed status on host cloudvirt1080. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1080 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:59:56] RESOLVED: [6x] SystemdUnitDown: The service unit libvirtd-admin.socket is in failed status on host cloudvirt1080. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudvirt1080 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [09:09:30] (03open) 10dcaro: runtime: split delete_job/delete_jobs per-job type [repos/cloud/toolforge/jobs-api] (split_create_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/331 [09:09:47] (03update) 10dcaro: runtime: split delete_job/delete_jobs per-job type [repos/cloud/toolforge/jobs-api] (split_create_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/331 [09:09:49] (03update) 10dcaro: runtime: split delete_job/delete_jobs per-job type [repos/cloud/toolforge/jobs-api] (split_create_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/331 [09:10:40] (03update) 10dcaro: runtime: split delete_job/delete_jobs per-job type [repos/cloud/toolforge/jobs-api] (split_create_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/331 [09:10:50] (03update) 10dcaro: runtime: split delete_job/delete_jobs per-job type [repos/cloud/toolforge/jobs-api] (split_create_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/331 [09:38:24] 06cloud-services-team, 10Data-Services, 10API-Portal, 06DBA, and 4 others: Delete apiportalwiki from wikireplicas - https://phabricator.wikimedia.org/T430102#12064448 (10Marostegui) 05In progress→03Resolved a:03Marostegui I've dropped the database in production with `if exists` on the master so... [09:39:16] 06cloud-services-team, 10Toolforge, 06DBA: Trove DB offline - https://phabricator.wikimedia.org/T430503 (10magnusmanske) 03NEW [09:40:03] 06cloud-services-team, 10Toolforge: Trove DB offline - https://phabricator.wikimedia.org/T430503#12064479 (10Marostegui) #dba doesn't maintain this part of the infra. [10:08:23] (03open) 10fnegri: Add new alert ToolsDBHistoryLengthGrowing [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/64 (https://phabricator.wikimedia.org/T428139) [10:09:01] (03update) 10fnegri: Add new alert ToolsDBHistoryLengthGrowing [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/64 (https://phabricator.wikimedia.org/T428139) [10:09:14] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12064651 (10fnegri) It did stabilize a little: {F91068433} In https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_... [10:15:48] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install cloudcephosd105[3456] - https://phabricator.wikimedia.org/T419892#12064678 (10fgiunchedi) Would we have capacity (power, space) to move two hosts to their final allocation in E4/F4 ? [10:19:07] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10netops: bird bfd session with 172.20.1.1 down - Bad packet from 172.20.1.1 - unknown session id - https://phabricator.wikimedia.org/T427202#12064697 (10cmooney) 05Open→03Resolved a:03cmooney Gonna close this one for now, if it be... [10:21:06] (03update) 10fnegri: install-binary-from-url - add checksums for dest [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/263 (https://phabricator.wikimedia.org/T402684) (owner: 10damian) [10:21:10] (03update) 10fnegri: install-binary-from-url - add checksums for dest [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/263 (https://phabricator.wikimedia.org/T402684) (owner: 10damian) [10:21:11] (03approved) 10fnegri: install-binary-from-url - add checksums for dest [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/263 (https://phabricator.wikimedia.org/T402684) (owner: 10damian) [10:22:14] (03update) 10fnegri: global: use buildservice as image type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/324 (owner: 10dcaro) [10:22:29] (03update) 10fnegri: global: use buildservice as image type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/324 (owner: 10dcaro) [10:24:45] (03approved) 10dcaro: install-binary-from-url - add checksums for dest [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/263 (https://phabricator.wikimedia.org/T402684) (owner: 10damian) [10:24:52] (03merge) 10dcaro: install-binary-from-url - add checksums for dest [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/263 (https://phabricator.wikimedia.org/T402684) (owner: 10damian) [10:25:14] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [lima-kilo] Only download artefacts if target binary checksum does not match - https://phabricator.wikimedia.org/T402684#12064747 (10dcaro) 05Open→03Resolved [10:25:29] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [lima-kilo] Improve convergence - https://phabricator.wikimedia.org/T402672#12064752 (10dcaro) 05Open→03Resolved [10:29:02] (03update) 10fnegri: ansible: do harbor before kind [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/327 (owner: 10dcaro) [10:30:01] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [lima-kilo] kind cluster fails to bootstrap - https://phabricator.wikimedia.org/T429783#12064777 (10fnegri) a:03dcaro [10:30:27] (03update) 10dcaro: runtime: split delete_job/delete_jobs per-job type [repos/cloud/toolforge/jobs-api] (split_create_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/331 [10:31:32] (03approved) 10fnegri: ansible: do harbor before kind [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/327 (owner: 10dcaro) [10:32:34] (03update) 10fnegri: ansible: do harbor before kind [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/327 (owner: 10dcaro) [10:33:27] (03update) 10dcaro: global: use buildservice as image type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/324 [10:33:55] (03update) 10dcaro: global: use buildservice as image type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/324 [10:36:05] (03approved) 10dcaro: global: use buildservice as image type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/324 [10:36:15] (03merge) 10dcaro: global: use buildservice as image type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/324 [10:36:26] (03update) 10fnegri: blubber: don't chmod -R 777 as it's quite slow [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/163 (owner: 10dcaro) [10:36:31] (03update) 10fnegri: blubber: don't chmod -R 777 as it's quite slow [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/163 (owner: 10dcaro) [10:36:32] (03approved) 10fnegri: blubber: don't chmod -R 777 as it's quite slow [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/163 (owner: 10dcaro) [10:39:40] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.516-20260629103634-67f5e76e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1307 [10:39:46] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.516-20260629103634-67f5e76e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1307 [10:40:43] (03update) 10dcaro: ansible: do harbor before kind [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/327 [10:41:51] (03merge) 10dcaro: ansible: do harbor before kind [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/327 [10:43:34] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [lima-kilo] kind cluster fails to bootstrap - https://phabricator.wikimedia.org/T429783#12064841 (10dcaro) 05Open→03Resolved p:05Triage→03Medium [10:44:08] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [lima-kilo] toolforge_run_ansible does not detect ARM arch - https://phabricator.wikimedia.org/T429795#12064848 (10dcaro) [10:48:49] (03update) 10fnegri: cleanup by image name [repos/cloud/toolforge/builds-api] (dont_chmod_on_build) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 (https://phabricator.wikimedia.org/T402568) (owner: 10dcaro) [10:48:59] (03update) 10fnegri: cleanup by image name [repos/cloud/toolforge/builds-api] (dont_chmod_on_build) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 (owner: 10dcaro) [10:49:17] (03update) 10fnegri: cleanup by image name [repos/cloud/toolforge/builds-api] (dont_chmod_on_build) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 (https://phabricator.wikimedia.org/T402568) (owner: 10dcaro) [10:49:47] (03update) 10fnegri: cleanup by image name [repos/cloud/toolforge/builds-api] (dont_chmod_on_build) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 (https://phabricator.wikimedia.org/T402568) (owner: 10dcaro) [10:50:25] (03update) 10fnegri: cleanup by image name [repos/cloud/toolforge/builds-api] (dont_chmod_on_build) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 (owner: 10dcaro) [10:50:36] 10Toolforge, 06tools-platform-team: [lima-kilo] ansible deprecation errors - https://phabricator.wikimedia.org/T429343#12064882 (10dcaro) Related maybe T413782 [10:50:47] 06cloud-services-team, 10Toolforge, 06tools-platform-team: lima-kilo: INJECT_FACTS_AS_VARS default to `True` is deprecated - https://phabricator.wikimedia.org/T413782#12064884 (10dcaro) [10:51:03] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [lima-kilo] toolforge_deploy_mr.py tdoes not support merge requests from forks properly - https://phabricator.wikimedia.org/T374045#12064885 (10dcaro) [10:51:13] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [lima-kilo] toolforge_deploy does not support merge requests from forks properly - https://phabricator.wikimedia.org/T374045#12064890 (10dcaro) [10:54:53] (03update) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [10:56:38] (03update) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [10:58:07] (03update) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [10:59:09] (03update) 10dcaro: Move "no builds" message to stderr in order to avoid violating --json flag [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/139 (owner: 10buxtang) [10:59:37] (03approved) 10fnegri: jobs-api: bump to 0.0.516-20260629103634-67f5e76e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1307 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [11:02:44] (03update) 10fnegri: core: properly handle launcher when comparing storage/runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/323 (owner: 10dcaro) [11:02:47] (03approved) 10fnegri: core: properly handle launcher when comparing storage/runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/323 (owner: 10dcaro) [11:02:49] (03update) 10fnegri: core: properly handle launcher when comparing storage/runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/323 (owner: 10dcaro) [12:08:11] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1306233 (owner: 10L10n-bot) [12:42:38] (03open) 10dcaro: builds-api: the cli return the no builds message on stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1308 [12:42:56] (03update) 10dcaro: Move "no builds" message to stderr in order to avoid violating --json flag [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/139 (owner: 10buxtang) [12:43:07] (03update) 10dcaro: builds-api: the cli return the no builds message on stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1308 [12:44:09] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [12:44:50] (03update) 10dcaro: core: properly handle launcher when comparing storage/runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/323 [12:45:57] (03update) 10dcaro: builds-api: the cli return the no builds message on stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1308 [12:48:12] (03update) 10dcaro: functional.builds-api: the cli return the no builds message on stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1308 [12:52:25] (03update) 10dcaro: core: properly handle launcher when comparing storage/runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/323 [12:56:13] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [12:57:14] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:06:13] (03update) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [13:08:12] (03update) 10fnegri: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 (owner: 10dcaro) [13:09:29] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [lima-kilo] toolforge_deploy does not support merge requests from forks properly - https://phabricator.wikimedia.org/T374045#12065367 (10dcaro) Note, now for clis at least it works as expected (that includes tools-webservice, now named webservice-cli). [13:10:28] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:11:13] (03approved) 10dcaro: jobs-api: bump to 0.0.516-20260629103634-67f5e76e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1307 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:11:21] (03merge) 10dcaro: jobs-api: bump to 0.0.516-20260629103634-67f5e76e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1307 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:11:38] (03approved) 10dcaro: core: properly handle launcher when comparing storage/runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/323 [13:11:49] (03merge) 10dcaro: core: properly handle launcher when comparing storage/runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/323 [13:13:44] (03approved) 10dcaro: Add new alert ToolsDBHistoryLengthGrowing [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/64 (https://phabricator.wikimedia.org/T428139) (owner: 10fnegri) [13:13:58] (03update) 10fnegri: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 (owner: 10dcaro) [13:14:27] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [13:14:27] 06cloud-services-team, 06tools-infrastructure-team: Provision cloudvirt with firewall - https://phabricator.wikimedia.org/T430310#12065407 (10fgiunchedi) [13:15:39] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.517-20260629131206-af3c60b5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1309 (https://phabricator.wikimedia.org/T429231) [13:15:47] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.517-20260629131206-af3c60b5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1309 (https://phabricator.wikimedia.org/T429231) [13:16:15] (03approved) 10fnegri: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 (owner: 10dcaro) [13:16:34] (03update) 10fnegri: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 (owner: 10dcaro) [13:16:52] (03update) 10fnegri: Add new alert ToolsDBHistoryLengthGrowing [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/64 (https://phabricator.wikimedia.org/T428139) [13:16:57] (03merge) 10fnegri: Add new alert ToolsDBHistoryLengthGrowing [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/64 (https://phabricator.wikimedia.org/T428139) [13:17:34] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [13:17:49] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:21:33] (03update) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [13:24:37] (03merge) 10dcaro: blubber: don't chmod -R 777 as it's quite slow [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/163 [13:24:41] (03update) 10dcaro: cleanup by image name [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/165 [13:25:58] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12065474 (10fnegri) 05In progress→03Resolved The new alert is live: https://prometheus.svc.toolforge.org/tools/alerts?sear... [13:29:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:30:01] 06tools-platform-team: [jobs-api] when restarting a job that does not exist on runtime, you get an exception 404 - https://phabricator.wikimedia.org/T430524 (10dcaro) 03NEW [13:30:10] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:33:42] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.218-20260629132448-fecc8677 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1310 [13:33:43] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.218-20260629132448-fecc8677 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1310 [13:37:27] 10Data-Services, 06tools-platform-team, 06Data-Persistence: CPU spikes in clouddb1014@s2 - https://phabricator.wikimedia.org/T429892#12065585 (10fnegri) 05Open→03Resolved No more spikes in the rest of the week: {F91089625} Resolving as a one-off, will reopen if it happens again. [13:40:52] 06cloud-services-team, 10Data-Services, 06Data-Persistence, 10Observability-Metrics: [wikireplicas] Publish sys.user_summary to Prometheus - https://phabricator.wikimedia.org/T429993#12065625 (10fnegri) p:05Triage→03Low @jcrespo thanks, I agree with your plan. As I wrote in {T429892}, I don't plan to w... [13:43:10] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:44:44] (03approved) 10fnegri: deploy_task: wait for empty slot when starting build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/173 (owner: 10dcaro) [13:44:44] (03update) 10fnegri: deploy_task: wait for empty slot when starting build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/173 (owner: 10dcaro) [13:49:10] (03approved) 10dcaro: jobs-api: bump to 0.0.517-20260629131206-af3c60b5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1309 (https://phabricator.wikimedia.org/T429231) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:49:15] (03merge) 10dcaro: jobs-api: bump to 0.0.517-20260629131206-af3c60b5 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1309 (https://phabricator.wikimedia.org/T429231) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:49:19] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [13:52:41] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [13:56:58] 10VPS-project-devtools, 10VPS-project-Phabricator, 06collaboration-services, 10Phabricator, and 2 others: devtools: Create new Phab/Phorge test instance on: Debian Trixie, Debian Bookworm - https://phabricator.wikimedia.org/T424055#12065795 (10Arnoldokoth) a:03Arnoldokoth [13:58:03] (03update) 10fnegri: conftest: Added an environment guard for DEBUG [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/317 (owner: 10dcaro) [13:58:08] (03update) 10fnegri: conftest: Added an environment guard for DEBUG [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/317 (owner: 10dcaro) [13:58:09] (03approved) 10fnegri: conftest: Added an environment guard for DEBUG [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/317 (owner: 10dcaro) [13:59:50] (03update) 10fnegri: functional.builds-api: the cli return the no builds message on stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1308 (owner: 10dcaro) [13:59:53] (03approved) 10fnegri: functional.builds-api: the cli return the no builds message on stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1308 (owner: 10dcaro) [14:00:03] (03update) 10fnegri: functional.builds-api: the cli return the no builds message on stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1308 (owner: 10dcaro) [14:04:05] 06cloud-services-team, 06tools-infrastructure-team: Patch demo resource check - https://phabricator.wikimedia.org/T429990#12065882 (10Andrew) a:03Andrew [14:04:08] (03update) 10dcaro: conftest: Added an environment guard for DEBUG [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/317 [14:04:44] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [14:05:14] (03update) 10dcaro: builds-api: bump to 0.0.218-20260629132448-fecc8677 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1310 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:06:37] 06tools-platform-team: [jobs-api] when restarting or deleting a job that does not exist on runtime, you get an exception 404 - https://phabricator.wikimedia.org/T430524#12065900 (10dcaro) [14:07:36] (03merge) 10dcaro: conftest: Added an environment guard for DEBUG [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/317 [14:07:50] (03update) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [14:08:51] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [14:10:47] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.518-20260629140749-2a4fe09f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1311 [14:10:49] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.518-20260629140749-2a4fe09f [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1311 [14:11:26] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [14:11:32] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 87.43%) at host paws-nfs-2 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [14:11:33] (03update) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [14:12:13] 06cloud-services-team, 10Toolforge: Unhandled ChunkedEncodingError crash from `toolforge jobs logs -f` - https://phabricator.wikimedia.org/T417225#12065949 (10aputhin) @DamianZaremba for what you are trying to do for longer than 10 mins, what would be a good timeout that doesn't disrupt it? We could look int... [14:12:51] 10Toolforge, 06tools-platform-team: Unhandled ChunkedEncodingError crash from `toolforge jobs logs -f` - https://phabricator.wikimedia.org/T417225#12065952 (10aputhin) [14:16:45] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [14:21:42] 10Toolforge, 06tools-platform-team: [logs-api] failing to return logs for job - https://phabricator.wikimedia.org/T429265#12066027 (10aputhin) p:05Low→03Medium We'll check your hypothesis and try to dig into further. Thanks for the additional logs. [14:24:36] (03approved) 10dcaro: builds-api: bump to 0.0.218-20260629132448-fecc8677 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1310 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:24:40] (03merge) 10dcaro: builds-api: bump to 0.0.218-20260629132448-fecc8677 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1310 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:25:08] (03update) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [14:32:57] 10Toolforge, 06tools-platform-team: Unhandled ChunkedEncodingError crash from `toolforge jobs logs -f` - https://phabricator.wikimedia.org/T417225#12066137 (10DamianZaremba) >>! In T417225#12065949, @aputhin wrote: > @DamianZaremba for what you are trying to do for longer than 10 mins, what would be a good tim... [14:40:06] 06cloud-services-team, 10Striker, 06tools-platform-team: [striker] Unable to create phabricator project for tool odivoir – HTTP 500 Internal Server Error on toolsadmin - https://phabricator.wikimedia.org/T429857#12066169 (10Aklapper) This would welcome some feedback from the WMCS team. (Not sure if this is r... [14:44:50] (03update) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [14:46:11] (03update) 10dcaro: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 [14:46:47] (03update) 10dcaro: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 [14:47:41] (03update) 10dcaro: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] (ignore_runtime_jobs_not_in_storage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 [14:48:14] (03update) 10dcaro: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] (ignore_runtime_jobs_not_in_storage) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 [14:56:54] 06tools-platform-team: [jobs-api] flaky unit test test_get_one_off_job_status - https://phabricator.wikimedia.org/T430550 (10dcaro) 03NEW [14:57:19] (03approved) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [14:57:27] (03merge) 10dcaro: core: don't use runtime jobs as source of truth [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/314 [14:57:35] (03update) 10dcaro: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 [15:00:19] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.519-20260629145747-3dd26ec1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1311 (https://phabricator.wikimedia.org/T359650) [15:00:28] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.519-20260629145747-3dd26ec1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1311 (https://phabricator.wikimedia.org/T359650) [15:00:41] (03update) 10dcaro: functional.builds-api: the cli return the no builds message on stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1308 [15:00:44] (03approved) 10dcaro: functional.builds-api: the cli return the no builds message on stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1308 [15:01:27] (03merge) 10dcaro: functional.builds-api: the cli return the no builds message on stderr [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1308 [15:02:47] (03merge) 10dcaro: Move "no builds" message to stderr in order to avoid violating --json flag [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/139 (owner: 10buxtang) [15:03:56] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:13:43] (03update) 10dcaro: fix: handle expected 404 in `deployment list` instead of erroring [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/83 (owner: 10buxtang) [15:16:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:17:18] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [15:18:48] 06cloud-services-team, 10Cloud-VPS: Trove DB offline - https://phabricator.wikimedia.org/T430503#12066461 (10JJMC89) [15:19:06] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 05MW-1.47-notes (1.47.0-wmf.7; 2026-06-16), 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)): Support requestBody.description in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427355#12066468 (10HCoplin-WMF) R... [15:19:09] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 05MW-1.47-notes (1.47.0-wmf.7; 2026-06-16), 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)): Support response body examples in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427361#12066466 (10HCoplin-WMF) Re... [15:19:10] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 05MW-1.47-notes (1.47.0-wmf.7; 2026-06-16), 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)): Support root-level externalDocs in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427356#12066469 (10HCoplin-WMF) R... [15:19:23] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)), 07OKR-Work: Improve linting - enum descriptions - https://phabricator.wikimedia.org/T424210#12066475 (10HCoplin-WMF) Resolving tasks as part of closing the sprint. [15:19:31] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)), 07OKR-Work: Improve linting - detect examples in nested schema.properties - https://phabricator.wikimedia.org/T424002#12066483 (10HCoplin-WMF) Resolving tasks as part of closing the sprint. [15:19:44] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)), 07OKR-Work: Fix schema property example validation - https://phabricator.wikimedia.org/T426836#12066487 (10HCoplin-WMF) 05Declined→03Resolved Resolving tasks as part of closing the sprint. [15:26:42] (03open) 10dcaro: d/changelog: bump to 0.0.26 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/140 (https://phabricator.wikimedia.org/T380127 https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T422018 https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T424368) [15:31:15] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [15:40:18] (03approved) 10dcaro: d/changelog: bump to 0.0.26 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/140 (https://phabricator.wikimedia.org/T380127 https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T422018 https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T424368) [15:40:35] (03approved) 10dcaro: jobs-api: bump to 0.0.519-20260629145747-3dd26ec1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1311 (https://phabricator.wikimedia.org/T359650) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:40:39] (03update) 10dcaro: jobs-api: bump to 0.0.519-20260629145747-3dd26ec1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1311 (https://phabricator.wikimedia.org/T359650) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:41:03] (03merge) 10dcaro: jobs-api: bump to 0.0.519-20260629145747-3dd26ec1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1311 (https://phabricator.wikimedia.org/T359650) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [15:41:30] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-api] Create storage layer, and save business models in persistent storage - https://phabricator.wikimedia.org/T359650#12066723 (10dcaro) Runtime jobs are not longer considered a source of truth, what's left is the cleanup of the s... [15:41:35] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-cli [15:43:26] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli [15:44:01] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-cli [15:46:13] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli [15:47:57] (03merge) 10dcaro: d/changelog: bump to 0.0.26 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/140 (https://phabricator.wikimedia.org/T380127 https://phabricator.wikimedia.org/T408783 https://phabricator.wikimedia.org/T422018 https://phabricator.wikimedia.org/T422184 https://phabricator.wikimedia.org/T424368) [15:55:01] 10Toolforge, 06tools-platform-team: [jobs-cli] emits a warning to re-create valid jobs - https://phabricator.wikimedia.org/T429231#12066810 (10dcaro) 05In progress→03Resolved [15:59:33] (03open) 10dcaro: general: remove enable_storage config [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/332 [16:00:52] (03update) 10dcaro: general: remove enable_storage config [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/332 [16:22:43] (03update) 10dcaro: runtime, core: only fetch oneoffs from runtime [repos/cloud/toolforge/jobs-api] (dont_use_kind) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [16:27:35] 10VPS-project-Codesearch, 10MediaWiki-libs-utfnormal: Add utfnormal to Codesearch - https://phabricator.wikimedia.org/T430363#12067002 (10Dzahn) Is it working? I wanted to confirm that there was actually no human action needed beyond merging.. if we just wait for the timer to run that rewrites the config.... [16:28:56] 10Tool-translatetagger, 06Indic-TechCom: Infinite loop on unclosed tags - https://phabricator.wikimedia.org/T430565 (10Aafi) 03NEW [16:30:34] 10Tool-translatetagger, 06Indic-TechCom: Infinite loop on unclosed tags - https://phabricator.wikimedia.org/T430565#12067020 (10Aafi) p:05Triage→03Medium [16:31:16] (03CR) 10Dzahn: [C:03+1] "would have no problem to just merge and try it. What do others think?" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306099 (https://phabricator.wikimedia.org/T197802) (owner: 10Simon04) [16:35:13] 10VPS-project-Codesearch, 10MediaWiki-libs-utfnormal: Add utfnormal to Codesearch - https://phabricator.wikimedia.org/T430363#12067041 (10SomeRandomDeveloper) >>! In T430363#12067002, @Dzahn wrote: > Is it working? I wanted to confirm that there was actually no human action needed beyond merging.. if we ju... [16:35:23] 06cloud-services-team, 06tools-infrastructure-team: Patch demo resource check - https://phabricator.wikimedia.org/T429990#12067042 (10Andrew) These numbers are never full stable due to ongoing decoms and new hardware, but here's a snapshot of cloud resources: CPUs: 4277 scheduled out of 3632 available (134%... [16:37:41] 10VPS-project-Codesearch, 10MediaWiki-libs-utfnormal: Add utfnormal to Codesearch - https://phabricator.wikimedia.org/T430363#12067073 (10Dzahn) Cool, so if you did not have to ping anyone else, great :) Love it. [16:43:43] (03update) 10dcaro: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 [16:46:33] (03update) 10dcaro: refactor: rename user to tool_account where needed [repos/cloud/toolforge/jobs-api] (use_job_kind_for_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/329 [16:47:48] (03update) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] (rename_user_to_tool_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [17:23:11] (03update) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] (rename_user_to_tool_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [17:34:03] (03CR) 10Gergő Tisza: "I don't have a proper test environment, but just by live-hacking the datalist into the page in developer tools, or saving the HTML file lo" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306099 (https://phabricator.wikimedia.org/T197802) (owner: 10Simon04) [17:41:00] 06cloud-services-team, 10Toolforge: [builds-api] `step-fix-procfile-args-passing` causes signals (e.g. SIGTERM) to not be seen by application - https://phabricator.wikimedia.org/T430578 (10DamianZaremba) 03NEW [17:41:56] 06cloud-services-team, 10Cloud-VPS: TLS Certificate Provisioning for wikiba.se Subdomain - https://phabricator.wikimedia.org/T430057#12067409 (10Olea) At this moment https://explorer.wikiba.se redirect to https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_introduction FYI [17:43:55] 06cloud-services-team, 10Cloud-VPS: TLS Certificate Provisioning for wikiba.se Subdomain - https://phabricator.wikimedia.org/T430057#12067412 (10bd808) [17:43:57] 06cloud-services-team, 10Cloud-VPS: Create mechanism to allow the use of vanity domains by projects behind the Cloud VPS shared HTTP proxy - https://phabricator.wikimedia.org/T342398#12067413 (10bd808) [17:45:15] 06cloud-services-team, 10Cloud-VPS: TLS Certificate Provisioning for wikiba.se Subdomain - https://phabricator.wikimedia.org/T430057#12067421 (10bd808) The vanity domain needs to be setup directly in WMCS. See https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_VPS_servers_from_the_intern... [17:46:35] 06cloud-services-team, 10Cloud-VPS: TLS Certificate Provisioning for wikiba.se Subdomain - https://phabricator.wikimedia.org/T430057#12067423 (10bd808) [17:46:36] 06cloud-services-team, 10Cloud-VPS: Create mechanism to allow the use of vanity domains by projects behind the Cloud VPS shared HTTP proxy - https://phabricator.wikimedia.org/T342398#12067424 (10bd808) [17:46:47] 06cloud-services-team, 10Cloud-VPS: TLS Certificate Provisioning for wikiba.se Subdomain - https://phabricator.wikimedia.org/T430057#12067425 (10bd808) [17:46:49] 06cloud-services-team, 10Cloud-VPS: Create mechanism to allow the use of vanity domains by projects behind the Cloud VPS shared HTTP proxy - https://phabricator.wikimedia.org/T342398#12067426 (10bd808) [17:48:30] 06cloud-services-team, 10Toolforge: [jobs-api] `get_command_for_k8s` causes signals (e.g. SIGTERM) to not be seen by application - https://phabricator.wikimedia.org/T430578#12067428 (10DamianZaremba) [18:02:12] (03open) 10damian: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [18:02:26] (03CR) 10BryanDavis: [C:03+1] "I haven't tested this, but it seems like the right fix. As Andre noted in T430120 this seems like a copy-and-paste problem that turned int" [labs/striker] - 10https://gerrit.wikimedia.org/r/1305875 (https://phabricator.wikimedia.org/T430120) (owner: 10Aklapper) [18:11:16] (03update) 10damian: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [18:13:35] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [jobs-api] `get_command_for_k8s` causes signals (e.g. SIGTERM) to not be seen by application - https://phabricator.wikimedia.org/T430578#12067488 (10DamianZaremba) Example when command is not wrapped in `sh`; ` tools.cluebotng@tools-bastion-15:~$ kubect... [18:13:37] (03update) 10damian: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [18:18:17] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [jobs-api] `get_command_for_k8s` causes signals (e.g. SIGTERM) to not be seen by application - https://phabricator.wikimedia.org/T430578#12067496 (10DamianZaremba) Interestingly the tests in jobs-api are verifying behaviour when `command` + `args` is us... [18:20:38] (03update) 10damian: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [18:24:38] (03update) 10damian: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [18:26:07] 06cloud-services-team, 06tools-infrastructure-team: Patch demo resource check - https://phabricator.wikimedia.org/T429990#12067513 (10Andrew) a:05Andrew→03BLiviero-WMF Belinda -- assigning back to you to close or follow up as needed. [18:35:36] (03update) 10damian: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [18:40:13] (03update) 10damian: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [18:43:55] (03update) 10damian: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [18:45:20] (03update) 10damian: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [18:45:35] (03update) 10damian: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [18:47:25] (03update) 10damian: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [18:50:15] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [jobs-api] `get_command_for_k8s` causes signals (e.g. SIGTERM) to not be seen by application - https://phabricator.wikimedia.org/T430578#12067614 (10DamianZaremba) Initial proposal for un-breaking this (specifically for buildpack images), minimising the... [18:51:25] (03open) 10buxtang: update handler to return empty list when no deployments [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/174 [18:52:37] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [jobs-api] `get_command_for_k8s` causes signals (e.g. SIGTERM) to not be seen by application - https://phabricator.wikimedia.org/T430578#12067621 (10DamianZaremba) [18:53:37] (03update) 10damian: Draft: command - selectively wrap command [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333 (https://phabricator.wikimedia.org/T430578) [18:57:42] (03update) 10buxtang: update handler to return empty list when no deployments [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/174 [19:38:23] (03CR) 10Krinkle: "Note that the frontend is standalone and zero config. Just `composer serve`. See https://gerrit.wikimedia.org/g/labs/codesearch/+/master/f" [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306099 (https://phabricator.wikimedia.org/T197802) (owner: 10Simon04) [19:59:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [20:04:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [20:07:46] (03CR) 10Andrew Bogott: [C:03+2] wmcs.nfs.migrate_service: allow overriding the dest port name [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1305914 (owner: 10Andrew Bogott) [20:08:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:24:23] 06cloud-services-team: CVN: Replace Bullseye nfs server with a new Trixie server - https://phabricator.wikimedia.org/T430589 (10Andrew) 03NEW [20:25:13] 06cloud-services-team: Dumps: replace Bullseye nfs server with Trixie - https://phabricator.wikimedia.org/T430590 (10Andrew) 03NEW [20:47:29] 06cloud-services-team, 10Countervandalism-Network: CVN: Replace Bullseye nfs server with a new Trixie server - https://phabricator.wikimedia.org/T430589#12068077 (10Krinkle) [20:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:54:24] 06cloud-services-team, 10Countervandalism-Network: CVN: Replace Bullseye nfs server with a new Trixie server - https://phabricator.wikimedia.org/T430589#12068093 (10Krinkle) a:05Andrew→03Krinkle I'd prefer not having to think about what the processes are going to do with NFS absent. So maybe a clean shutdo... [21:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:51:58] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28404836827 (https://github.com/cluebotng/component-configs/commits/637e933186e3a7fedd0e7328d4c4e6035d12ec82) [21:52:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL