[01:38:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:08:20] (03update) 10raymond-ndibe: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 (owner: 10dcaro) [05:38:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:29:50] (03approved) 10raymond-ndibe: refactor: rename user to tool_account where needed [repos/cloud/toolforge/jobs-api] (use_job_kind_for_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/329 (owner: 10dcaro) [06:30:08] (03unapproved) 10raymond-ndibe: refactor: rename user to tool_account where needed [repos/cloud/toolforge/jobs-api] (use_job_kind_for_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/329 (owner: 10dcaro) [06:40:34] FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [06:41:43] (03update) 10raymond-ndibe: general: remove enable_storage config [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/332 (owner: 10dcaro) [06:41:48] (03approved) 10raymond-ndibe: general: remove enable_storage config [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/332 (owner: 10dcaro) [07:00:34] RESOLVED: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [07:42:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [07:47:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [08:02:33] (03update) 10dcaro: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 [08:32:59] (03merge) 10dcaro: general: remove enable_storage config [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/332 [08:36:11] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.520-20260630083312-c7b4a9df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1312 (https://phabricator.wikimedia.org/T359650) [08:36:14] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.520-20260630083312-c7b4a9df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1312 (https://phabricator.wikimedia.org/T359650) [08:39:17] (03update) 10dcaro: jobs-api: bump to 0.0.520-20260630083312-c7b4a9df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1312 (https://phabricator.wikimedia.org/T359650) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:43:11] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:43:42] (03update) 10dcaro: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 [08:52:23] (03CR) 10Gergő Tisza: "Oh right. I knew that at some point but forgot." [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306099 (https://phabricator.wikimedia.org/T197802) (owner: 10Simon04) [08:55:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [08:58:40] (03update) 10dcaro: refactor: rename user to tool_account where needed [repos/cloud/toolforge/jobs-api] (use_job_kind_for_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/329 [08:58:55] (03update) 10dcaro: refactor: rename user to tool_account where needed [repos/cloud/toolforge/jobs-api] (use_job_kind_for_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/329 [08:59:32] (03update) 10dcaro: refactor: rename user to tool_account where needed [repos/cloud/toolforge/jobs-api] (dont_use_kind) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/329 [09:11:45] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12069410 (10Marostegui) [09:12:14] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Data-Services, 06Data-Platform-SRE, and 4 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12069417 (10fgiunchedi) [09:20:52] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [09:33:21] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [09:38:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:01:55] (03approved) 10dcaro: jobs-api: bump to 0.0.520-20260630083312-c7b4a9df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1312 (https://phabricator.wikimedia.org/T359650) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:02:04] (03merge) 10dcaro: jobs-api: bump to 0.0.520-20260630083312-c7b4a9df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1312 (https://phabricator.wikimedia.org/T359650) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [10:04:36] (03approved) 10dcaro: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 [10:04:40] (03merge) 10dcaro: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 [10:04:47] (03update) 10dcaro: runtime, core: only fetch oneoffs from runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [10:04:52] (03update) 10dcaro: refactor: rename user to tool_account where needed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/329 [10:08:21] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.521-20260630100458-2ab4b76b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1313 (https://phabricator.wikimedia.org/T430071) [10:08:29] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.521-20260630100458-2ab4b76b [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1313 (https://phabricator.wikimedia.org/T430071) [10:35:21] (03update) 10dcaro: refactor: rename user to tool_account where needed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/329 [10:35:24] (03update) 10dcaro: refactor: rename user to tool_account where needed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/329 [10:35:26] (03open) 10dcaro: global: rename tool -> tool_name for consistency [repos/cloud/toolforge/jobs-api] (rename_user_to_tool_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/334 [10:36:39] (03update) 10dcaro: global: rename tool -> tool_name for consistency [repos/cloud/toolforge/jobs-api] (rename_user_to_tool_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/334 [10:37:25] (03merge) 10dcaro: refactor: rename user to tool_account where needed [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/329 [10:37:33] (03update) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [10:37:37] (03update) 10dcaro: global: rename tool -> tool_name for consistency [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/334 [10:40:36] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.522-20260630103743-7a5f747d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1313 (https://phabricator.wikimedia.org/T430071) [10:40:40] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.522-20260630103743-7a5f747d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1313 (https://phabricator.wikimedia.org/T430071) [10:50:16] (03CR) 10FNegri: [C:03+1] Remove bogus attachment parameter 'uris' from Phabricator API calls [labs/striker] - 10https://gerrit.wikimedia.org/r/1305875 (https://phabricator.wikimedia.org/T430120) (owner: 10Aklapper) [11:45:30] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#12070152 (10Marostegui) clouddb1027 (s2 and s7) is now up and catching up. [11:56:11] 06cloud-services-team (FY2025/2026-Q3-Q4), 06Infrastructure-Foundations, 10netops, 06Traffic: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651 (10fgiunchedi) 03NEW [11:57:55] 06cloud-services-team (FY2025/2026-Q3-Q4), 06Infrastructure-Foundations, 10netops, 06Traffic, 13Patch-For-Review: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12070204 (10fgiunchedi) https://gerrit.wikimedia.org/r/c/operations/puppet... [12:25:49] 06cloud-services-team (FY2025/2026-Q3-Q4), 06Infrastructure-Foundations, 10netops, 06Traffic, 13Patch-For-Review: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12070301 (10MoritzMuehlenhoff) All the IPs allowed by LOAD_BALANCER_HEALTH... [12:34:00] (03update) 10buxtang: update handler to return empty list when no deployments [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/174 [12:41:18] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops, 06SRE: Upgrade cloudsw1-e4-eqiad - https://phabricator.wikimedia.org/T429013#12070397 (10ayounsi) Unfortunately there is no easy way to perform the maintenance without impacting the devices connected to that switch. [12:52:38] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06tools-infrastructure-team: Put cloudvirt10[77-80] in service - https://phabricator.wikimedia.org/T429563#12070448 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi hosts are in service. I have updated https://wikitech.wikimedia.org/wiki/Portal... [12:59:27] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Data-Services, 06Data-Platform-SRE, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12070478 (10fgiunchedi) [13:06:38] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:09:31] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api [13:09:38] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Data-Services, 06Data-Platform-SRE, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12070530 (10fgiunchedi) I'll be starting the trials on `toolsbeta` and specifically `toolsbeta... [13:09:39] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: [jobs-api] Create storage layer, and save business models in persistent storage - https://phabricator.wikimedia.org/T359650#12070528 (10dcaro) 05In progress→03Resolved [13:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:25:25] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:33:03] 06cloud-services-team (FY2025/2026-Q3-Q4), 06Infrastructure-Foundations, 10netops, 06Traffic, 13Patch-For-Review: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12070626 (10cmooney) @fgiunchedi thanks for the task. I think @MoritzMueh... [13:37:12] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [13:38:15] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [13:46:42] (03CR) 10David Caro: "LGTM, I'm struggling to get the local env running to test it though, I'm happy if anyone can do before that, but might take me some time" [labs/striker] - 10https://gerrit.wikimedia.org/r/1305875 (https://phabricator.wikimedia.org/T430120) (owner: 10Aklapper) [13:51:15] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [14:11:19] (03approved) 10dcaro: jobs-api: bump to 0.0.522-20260630103743-7a5f747d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1313 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:11:28] (03merge) 10dcaro: jobs-api: bump to 0.0.522-20260630103743-7a5f747d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1313 (https://phabricator.wikimedia.org/T430071) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [14:11:33] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 90.55%) at host paws-nfs-2 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [15:37:56] (03PS2) 10Simon04: Add datalist suggestions to the file path filters [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306099 (https://phabricator.wikimedia.org/T197802) [15:38:58] (03CR) 10Simon04: "Err, I confused include and exclude while editing the mustache file. Fixed now." [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306099 (https://phabricator.wikimedia.org/T197802) (owner: 10Simon04) [16:15:42] 10Tool-wsindex, 10Wikisource Reader App (Android): Add Persian to Wsindex API and Wikisource reader app - https://phabricator.wikimedia.org/T430704 (10Bodhisattwa) 03NEW [16:15:45] 10Tool-wsindex, 10Wikisource Reader App (Android): Add name_native_language to People - https://phabricator.wikimedia.org/T430705 (10Saiphani02) 03NEW [16:16:07] 10Tool-wsindex, 10Wikisource Reader App (Android): Add Persian to Wsindex API and Wikisource reader app - https://phabricator.wikimedia.org/T430704#12071912 (10Bodhisattwa) a:03Bodhisattwa [16:17:36] 10Tool-wsindex, 10Wikisource Reader App (Android): Add name_native_language to People - https://phabricator.wikimedia.org/T430705#12071921 (10Saiphani02) [16:18:06] 10Tool-wsindex, 10Wikisource Reader App (Android): Add name_native_language to People - https://phabricator.wikimedia.org/T430705#12071923 (10Saiphani02) [16:20:45] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28459029072 (https://github.com/cluebotng/component-configs/commits/428e86ea3f09468d24a0ba98e75225c1c1698e53) [16:20:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [16:29:22] (03update) 10dcaro: global: rename tool -> tool_name for consistency [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/334 [16:31:00] 10Cloud-VPS (Project-requests), 10Maps: Check current usage of project maps on CloudVPS - https://phabricator.wikimedia.org/T429446#12072121 (10fnegri) 05Open→03Declined Closing as "Declined", there are still active users of this project as described above. [16:32:39] (03open) 10sarai-wmf: Add media and publications scenario [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/4 [16:32:43] (03update) 10dcaro: global: rename tool -> tool_name for consistency [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/334 [16:33:39] 10Tool-wsindex, 10Wikisource Reader App (Android): Add Persian to Wsindex API and Wikisource reader app - https://phabricator.wikimedia.org/T430704#12072147 (10Bodhisattwa) Persian is now in the API - https://wsindex.toolforge.org/books/?languages=fa [16:37:29] 06cloud-services-team, 10Cloud-VPS: Replace NFS in wikidumpparse with a standard cinder volume - https://phabricator.wikimedia.org/T430167#12072170 (10Andrew) a:03notconfusing [16:38:01] (03merge) 10sarai-wmf: Add media and publications scenario [toolforge-repos/wikimedia-attribution] - 10https://gitlab.wikimedia.org/toolforge-repos/wikimedia-attribution/-/merge_requests/4 [16:39:27] 06cloud-services-team: Update all cloud-vps Bullseye NFS servers - https://phabricator.wikimedia.org/T429793#12072178 (10Andrew) [16:40:35] 06cloud-services-team: Dumps: replace Bullseye nfs server with Trixie - https://phabricator.wikimedia.org/T430590#12072186 (10Andrew) I'll go ahead and run this update next week if I don't hear anything here in the meantime. [16:45:29] (03update) 10dcaro: runtime, core: only fetch oneoffs from runtime [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [16:46:01] (03update) 10dcaro: runtime, core: only fetch oneoffs from runtime [repos/cloud/toolforge/jobs-api] (rename_tool_name) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [16:46:11] (03update) 10dcaro: runtime, core: only fetch oneoffs from runtime [repos/cloud/toolforge/jobs-api] (rename_tool_name) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [16:48:22] (03update) 10dcaro: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] (rename_tool_name) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [16:48:29] (03update) 10dcaro: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] (rename_tool_name) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [16:59:23] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node toolsbeta-test-k8s-worker-nfs-8 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [17:07:33] (03update) 10dcaro: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] (rename_tool_name) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [17:07:47] (03update) 10dcaro: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] (rename_tool_name) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [17:10:32] FIRING: InstanceDown: Project toolsbeta instance toolsbeta-test-k8s-worker-nfs-8 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:14:10] 10Tool-wsindex, 10Wikisource Reader App (Android): Add Persian to Wsindex API and Wikisource reader app - https://phabricator.wikimedia.org/T430704#12072420 (10Bodhisattwa) 05Open→03Resolved Added fa to app repo too. [17:32:08] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 06MW-Interfaces-Team (MediaWiki-Interfaces-Sprints): Support requestBody content examples in MediaWiki REST Framework OAD generation - https://phabricator.wikimedia.org/T427360#12072551 (10AGhirelli-WMF) 05In progress→03Resolved [17:39:23] RESOLVED: ToolforgeKubernetesNodeNotReady: Kubernetes node toolsbeta-test-k8s-worker-nfs-8 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [17:46:33] 06cloud-services-team, 10PAWS, 06tools-platform-team, 13Patch-For-Review: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12072621 (10Andrew) I'm pretty sure that the cleanup script that copies things into /srv/paws/files-to-remove/ was lost when I rebuilt the NFS server. Going... [17:49:34] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance paws-nfs-1 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [17:51:32] (03update) 10dcaro: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] (rename_tool_name) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [17:53:23] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node toolsbeta-test-k8s-worker-nfs-8 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [17:53:58] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Allow cumin access to all Cloud VPS VMs - https://phabricator.wikimedia.org/T422801#12072648 (10Andrew) [20:00:00] (03open) 10zaidusyy: fix(a11y): add missing ARIA labels to interactive elements [toolforge-repos/wikisemanticimgsearch] - 10https://gitlab.wikimedia.org/toolforge-repos/wikisemanticimgsearch/-/merge_requests/1 [20:19:41] 10Tool-wmf-openapi-linter, 10MediaWiki-REST-API, 05MW-1.47-notes (1.47.0-wmf.10; 2026-07-07), 06MW-Interfaces-Team (MWI-Sprint-36 (2026-06-16 to 2026-06-30)): Generate a valid semver info.version for the default MediaWiki REST module - https://phabricator.wikimedia.org/T427359#12073509 (10MGoncalves-WMF) [22:09:31] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work: Deploy the OpenAPI linter in CI - https://phabricator.wikimedia.org/T422920#12073933 (10AGhirelli-WMF) a:03AGhirelli-WMF [22:10:18] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work: Publish the Wikimedia Spectral ruleset to NPM - https://phabricator.wikimedia.org/T425627#12073935 (10AGhirelli-WMF) 05Open→03In progress [22:10:20] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MediaWiki-Interfaces-Sprints), 07OKR-Work: Publish the Wikimedia Spectral ruleset to NPM - https://phabricator.wikimedia.org/T425627#12073937 (10AGhirelli-WMF) a:03AGhirelli-WMF [23:30:26] 10Toolforge, 06tools-platform-team, 07good first task: [builds-cli] does not output valid json when there's no builds - https://phabricator.wikimedia.org/T429229#12074072 (10Buxtang) 05Open→03In progress