[02:06:06] (03update) 10raymond-ndibe: get_job_from_k8s: stop using kind and use job_type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/320 (owner: 10dcaro) [02:14:25] (03update) 10raymond-ndibe: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 (owner: 10dcaro) [02:14:30] (03approved) 10raymond-ndibe: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 (owner: 10dcaro) [02:14:57] (03unapproved) 10raymond-ndibe: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 (owner: 10dcaro) [03:04:46] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [03:41:50] (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T388092) [05:23:34] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:23:45] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [05:28:34] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:28:45] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [05:30:34] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:30:45] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [05:35:34] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:35:45] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [05:53:32] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [05:58:32] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [06:33:52] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/25 [06:34:01] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/18 [06:34:01] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/25 [06:34:05] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/84 [06:34:14] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/18 [06:34:18] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/175 [06:34:20] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/84 [06:34:22] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/160 [06:34:41] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/175 [06:34:43] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/54 [06:34:46] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/160 [06:34:47] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/26 [06:34:55] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/85 [06:35:08] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/54 [06:35:19] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/26 [06:35:22] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/176 [06:35:23] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/99 [06:35:25] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/328 [06:35:35] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/85 [06:35:46] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/55 [06:35:47] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/141 [06:36:03] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/176 [06:36:08] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/113 [06:36:09] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/99 [06:36:09] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/335 [06:36:14] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/328 [06:36:20] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/100 [06:36:24] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/55 [06:36:27] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/141 [06:36:29] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/68 [06:36:36] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/335 [06:36:37] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/113 [06:36:44] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/100 [06:36:49] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: build: Upgrade Poetry dependencies [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/68 [07:01:13] (03open) 10raymond-ndibe: [values.yaml] add support for webservice-defaults [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/23 (https://phabricator.wikimedia.org/T348755) [07:01:48] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T348755) [07:02:46] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T348755) [07:05:20] (03open) 10govindlaltl: Add Wikimedia translation [toolforge-repos/tome] - 10https://gitlab.wikimedia.org/toolforge-repos/tome/-/merge_requests/2 [07:11:03] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T348755) [07:33:25] (03update) 10raymond-ndibe: add webservice endpoint [repos/cloud/toolforge/jobs-api] (add_ingress_option_to_continuous_job) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T348755) [08:17:55] 06cloud-services-team (FY2025/2026-Q3-Q4), 06Infrastructure-Foundations, 10netops, 06Traffic, 13Patch-For-Review: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651#12074704 (10ayounsi) This service is using LVS with IPIP so I don't unders... [08:22:36] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Data-Services, 06Data-Platform-SRE, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12074721 (10ayounsi) @fgiunchedi maybe silly question but to make sure we're preserving public... [08:26:00] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [builds-api] Allow queuing builds - https://phabricator.wikimedia.org/T401894#12074726 (10dcaro) p:05Medium→03Low This is not critical now that components-api will be the one waiting for the builds to finish {T402568} [08:27:34] 06cloud-services-team, 10Toolforge: [builds-api, maintain-harbor] fix build/image cleanup - https://phabricator.wikimedia.org/T404157#12074732 (10dcaro) [08:29:49] (03open) 10samwilson: Switch machine translation from checkbox to button [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/11 (https://phabricator.wikimedia.org/T425082) [08:37:48] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#12074753 (10dcaro) There's a couple things I want to get done here before closing: * Procfile upstream did not accept... [09:10:48] (03update) 10dcaro: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] (rename_tool_name) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [09:10:50] (03PS1) 10Elukey: role::cluster::management: add fake mgmt password [labs/private] - 10https://gerrit.wikimedia.org/r/1306875 (https://phabricator.wikimedia.org/T429699) [09:11:22] (03CR) 10Elukey: [V:03+2 C:03+2] role::cluster::management: add fake mgmt password [labs/private] - 10https://gerrit.wikimedia.org/r/1306875 (https://phabricator.wikimedia.org/T429699) (owner: 10Elukey) [09:31:27] (03open) 10dcaro: runtime: remove diff_with_running_job [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/336 [09:35:18] (03approved) 10psubhashish: Add Wikimedia translation [toolforge-repos/tome] - 10https://gitlab.wikimedia.org/toolforge-repos/tome/-/merge_requests/2 (owner: 10govindlaltl) [09:35:23] (03merge) 10psubhashish: Add Wikimedia translation [toolforge-repos/tome] - 10https://gitlab.wikimedia.org/toolforge-repos/tome/-/merge_requests/2 (owner: 10govindlaltl) [12:17:47] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28459029072 (https://github.com/cluebotng/component-configs/commits/428e86ea3f09468d24a0ba98e75225c1c1698e53) [12:17:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [13:10:28] 10Tool-translatetagger, 06Indic-TechCom, 10LPL sprints, 10MediaWiki-extensions-Translate: Integrate TranslateTagger functions/workflow into Special:PagePreparation of Translate Extension - https://phabricator.wikimedia.org/T430257#12075907 (10abi_) [13:10:47] 10Tool-translatetagger, 06Indic-TechCom, 10LPL sprints, 10MediaWiki-extensions-Translate: Integrate TranslateTagger functions/workflow into Special:PagePreparation of Translate Extension - https://phabricator.wikimedia.org/T430257#12075909 (10abi_) p:05Triage→03Medium [13:19:15] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506460 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) [13:19:16] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506734 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) [13:19:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [13:19:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [13:19:57] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28520506430 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) [13:19:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [13:20:05] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api] `get_command_for_k8s` causes signals (e.g. SIGTERM) to not be seen by application - https://phabricator.wikimedia.org/T430578#12075937 (10dcaro) [13:21:24] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506415 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) [13:21:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-functional-runner/SAL [13:21:39] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506454 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) [13:21:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [13:21:59] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506563 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) [13:22:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [13:22:48] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506690 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) [13:22:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [13:22:53] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506484 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) [13:22:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [13:24:37] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506476 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) [13:24:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [13:39:07] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Data-Services, 06Data-Platform-SRE, and 3 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12076020 (10fgiunchedi) >>! In T411248#12074721, @ayounsi wrote: > @fgiunchedi maybe silly que... [13:50:27] 06cloud-services-team, 10Horizon, 10Openstack-Magnum, 06tools-infrastructure-team, and 2 others: Can't retrieve Magnum cluster certificate and config via Horizon - https://phabricator.wikimedia.org/T430680#12076122 (10Andrew) 05Open→03Resolved [13:54:46] 10Toolforge, 06tools-platform-team: Unhandled ChunkedEncodingError crash from `toolforge jobs logs -f` - https://phabricator.wikimedia.org/T417225#12076194 (10aputhin) We'll change the timeout to 3h for now to unblock you, and then we'll work on a proper fix later on where we refresh the session (and then redu... [13:56:05] (03merge) 10andrew: update blubber buildkit image to v1.8.0 [repos/cloud/cloud-vps/horizon/deploy] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/merge_requests/7 (https://phabricator.wikimedia.org/T321316) (owner: 10jelto) [13:57:39] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [jobs-api] `get_command_for_k8s` causes signals (e.g. SIGTERM) to not be seen by application - https://phabricator.wikimedia.org/T430578#12076220 (10aputhin) p:05Triage→03High @DamianZaremba thank you for working on this. Pl... [14:11:33] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 87.77%) at host paws-nfs-2 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [14:56:00] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28520506734 (https://github.com/cluebotng/component-configs/commits/5d3fea141215e62d03aa2ecee86c6d2679cba955) [14:56:16] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28525308526 (https://github.com/cluebotng/component-configs/commits/4d35560ba5b135c6dff161ca5a7ed6c393edd180) [15:00:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [15:00:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [15:00:47] (03update) 10raymond-ndibe: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] (rename_tool_name) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 (owner: 10dcaro) [15:15:17] (03update) 10raymond-ndibe: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 (owner: 10dcaro) [15:18:18] (03approved) 10raymond-ndibe: global: rename tool -> tool_name for consistency [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/334 (owner: 10dcaro) [15:19:52] 06tools-infrastructure-team, 06tools-platform-team: [ceph] Stats not reporting slow operations - https://phabricator.wikimedia.org/T430846 (10dcaro) 03NEW [15:20:16] (03update) 10raymond-ndibe: global: rename tool -> tool_name for consistency [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/334 (owner: 10dcaro) [15:23:06] 06tools-infrastructure-team, 06tools-platform-team: [ceph] Stats not reporting slow operations - https://phabricator.wikimedia.org/T430846#12076879 (10dcaro) [15:23:45] 06tools-infrastructure-team, 06tools-platform-team: [ceph] Stats not reporting slow operations - https://phabricator.wikimedia.org/T430846#12076880 (10dcaro) [15:23:53] 06tools-infrastructure-team: [ceph] Stats not reporting slow operations - https://phabricator.wikimedia.org/T430846#12076881 (10dcaro) [15:24:39] 06tools-infrastructure-team: [ceph] Stats not reporting slow operations - https://phabricator.wikimedia.org/T430846#12076882 (10dcaro) [15:31:46] (03open) 10andrew: update requirements submodule [repos/cloud/cloud-vps/horizon/deploy] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/merge_requests/8 (https://phabricator.wikimedia.org/T321316) [15:35:21] (03update) 10andrew: update requirements submodule [repos/cloud/cloud-vps/horizon/deploy] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/merge_requests/8 [15:36:21] (03merge) 10andrew: update requirements submodule [repos/cloud/cloud-vps/horizon/deploy] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/merge_requests/8 [15:38:41] (03update) 10dcaro: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] (rename_tool_name) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [15:39:59] 06cloud-services-team, 10Data-Services, 06collaboration-services, 10Continuous-Integration-Infrastructure (Zuul upgrade): Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12076938 (10Dzahn) please try again now [16:03:41] 06tools-platform-team: How to Deploy on Toolforge Workshop - https://phabricator.wikimedia.org/T427961#12077043 (10dcaro) 05Open→03In progress [16:21:17] (03merge) 10dcaro: global: rename tool -> tool_name for consistency [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/334 [16:21:21] (03update) 10dcaro: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [16:21:41] (03update) 10dcaro: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [16:25:26] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.523-20260701162132-d3541ac2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1314 (https://phabricator.wikimedia.org/T430071) [16:25:35] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: jobs-api: bump to 0.0.523-20260701162132-d3541ac2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1314 (https://phabricator.wikimedia.org/T430071) [16:43:49] (03open) 10andrew: Update for 2026.1 [repos/cloud/cloud-vps/horizon/deploy] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/merge_requests/9 [16:49:16] (03merge) 10andrew: Update for 2026.1 [repos/cloud/cloud-vps/horizon/deploy] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/merge_requests/9 [16:50:56] (03update) 10dcaro: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [16:51:13] (03update) 10dcaro: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [16:51:58] (03open) 10arian-ar: Fix transparent bookmark drawer background [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/1 [16:52:14] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12077414 (10TLepage-WMF) a:03TLepage-WMF [16:58:03] 06cloud-services-team, 10Data-Services, 06collaboration-services, 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Zuul executor can not reach WMCS OpenStack API - https://phabricator.wikimedia.org/T430479#12077464 (10hashar) I have confirmed the port 25000 works! Unfortunatel... [16:59:00] (03open) 10arian-ar: Fix stale ToolsDB connections causing intermittent 500 errors [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/2 [16:59:17] (03merge) 10arian-ar: Fix stale ToolsDB connections causing intermittent 500 errors [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/2 [17:00:37] (03update) 10dcaro: runtime: remove diff_with_running_job [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/336 [17:07:47] (03open) 10arian-ar: Add GitLab MR template and fix Toolforge envvars command [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/3 [17:08:00] (03merge) 10arian-ar: Add GitLab MR template and fix Toolforge envvars command [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/3 [17:08:35] (03merge) 10arian-ar: Fix transparent bookmark drawer background [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/1 [17:25:17] (03update) 10dcaro: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 [17:27:15] (03update) 10dcaro: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 [17:27:16] (03update) 10dcaro: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 [17:29:14] (03update) 10dcaro: k8s: use job_type for label generating functions [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/328 [17:33:28] (03close) 10dcaro: k8s: move get k8s objects per job_type [repos/cloud/toolforge/jobs-api] (delete_all_per_job_type) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/326 [17:33:54] (03update) 10dcaro: runtime, core: split get_job/get_jobs [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/327 [17:42:27] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28536333606 (https://github.com/cluebotng/component-configs/commits/fc7f0d0647c84e0e5c9f02d279015dc3183f3151) [17:42:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [17:50:21] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#12077798 (10Osuji_pius) **Weekly Internship Report** **Week 4: June 16 – June 21** **Task Progress** * Continued the implementation of the redesigned CampWiz interface based on the... [17:57:02] (03update) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [17:59:45] 10Tools, 06WikiNYC Tech Committee: ITN/Wikipulse platform - https://phabricator.wikimedia.org/T427575#12077826 (10Pharos) Here is the very first prototype, by @Habst : https://wikinewsie.toolforge.org/ ! [18:04:17] (03open) 10dcaro: stop passing tool name [repos/cloud/toolforge/jobs-api] (fetch_only_oneoff_from_runtime) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/337 [18:05:33] (03update) 10dcaro: stop passing tool name [repos/cloud/toolforge/jobs-api] (use_job_kind_for_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/337 [18:06:58] (03update) 10dcaro: stop passing tool name [repos/cloud/toolforge/jobs-api] (use_job_kind_for_labels) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/337 [18:08:40] (03update) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [18:09:31] (03update) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] (stop_passing_tool_name) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [18:13:07] 06cloud-services-team, 06Trust-and-Safety: Account recovery help needed for Developer account TLepage-WMF - https://phabricator.wikimedia.org/T430872 (10TLepage-WMF) 03NEW [18:16:40] 06cloud-services-team, 06Trust-and-Safety: Account recovery help needed for Developer account TLepage-WMF - https://phabricator.wikimedia.org/T430872#12077876 (10Aklapper) Hi @TLepage-WMF and welcome! > setting up 2FA on Wikitech If this is about wikitech.wikimedia.org, then this is not a separate account an... [18:16:44] (03open) 10arian-ar: Prepare translatewiki.net i18n integration [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/4 [18:18:28] (03merge) 10arian-ar: Prepare translatewiki.net i18n integration [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/4 [18:20:05] 06cloud-services-team, 06Trust-and-Safety: Account recovery help needed for Developer account TLepage-WMF - https://phabricator.wikimedia.org/T430872#12077886 (10TLepage-WMF) I believe you are right, apologies for the lack of precision, still getting my footing among the various account types! I indeed have th... [18:24:05] (03update) 10dcaro: runtime: split create_job per-job type [repos/cloud/toolforge/jobs-api] (stop_passing_tool_name) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/330 [18:26:28] 06cloud-services-team, 06Trust-and-Safety: Account recovery help needed for Developer account TLepage-WMF - https://phabricator.wikimedia.org/T430872#12077909 (10jrbs) >>! In T430872#12077886, @TLepage-WMF wrote: > I believe you are right, apologies for the lack of precision, still getting my footing among the... [18:26:49] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28538748543 (https://github.com/cluebotng/component-configs/commits/6c7bcbaa2a10eb01c07a4df9fe5c1ea3b571b351) [18:26:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [18:33:03] 10Tools, 06WikiNYC Tech Committee: ITN/Wikipulse platform - https://phabricator.wikimedia.org/T427575#12077936 (10Pharos) About half of the 20 articles typically lack an image file. Maybe the file should be automatically drawn by Wikidata from the more generic thing it's part of. And/or it could be set manuall... [18:36:40] 06cloud-services-team, 06Trust-and-Safety: Account recovery help needed for Developer account TLepage-WMF - https://phabricator.wikimedia.org/T430872#12077941 (10TLepage-WMF) Thank you! I have followed your suggestion and sent an email, subject line "Account recovery help needed for SUL: TLepage-WMF". Happy to... [18:46:49] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12077956 (10TLepage-WMF) [18:53:12] (03open) 10arian-ar: Log expected EventStreams disconnects at info level [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/5 [18:53:21] (03merge) 10arian-ar: Log expected EventStreams disconnects at info level [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/5 [19:07:40] (03open) 10arian-ar: Fix uvicorn reload hanging on background task shutdown [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/6 [19:10:16] (03merge) 10arian-ar: Fix uvicorn reload hanging on background task shutdown [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/6 [19:19:47] 06tools-platform-team: Onboard Thibaut Le Page to Wikimedia Foundation as Staff SWE in Tools Platform - https://phabricator.wikimedia.org/T427914#12078052 (10TLepage-WMF) [19:35:29] 10Cloud-Services: PuppetStaleCertificates alert is too aggressive for deployment-prep - https://phabricator.wikimedia.org/T430875 (10dancy) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it wit... [19:35:51] 06cloud-services-team, 10Cloud-VPS: PuppetStaleCertificates alert is too aggressive for deployment-prep - https://phabricator.wikimedia.org/T430875#12078092 (10dancy) [20:12:01] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/28544657461 (https://github.com/cluebotng/component-configs/commits/5bba455f1ea62dc871234aa761ab283bfbaab861) [20:12:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [20:15:01] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/28544657044 (https://github.com/cluebotng/component-configs/commits/5bba455f1ea62dc871234aa761ab283bfbaab861) [20:15:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL