[00:49:16] (03open) 10jaredblumer: Add License Name and URL Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/1 [00:50:56] 10Tool-paulina: Implement breadcrumb navigation for improved contextual navigation across Paulina pages - https://phabricator.wikimedia.org/T409868#11364974 (10Reedy) [01:00:13] (03update) 10jaredblumer: Add License Name and URL Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/1 [01:00:40] (03update) 10jaredblumer: Add License Name and URL Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/1 [01:48:40] (03open) 10raymond-ndibe: Draft: add endpoint to get all available images [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T409726) [01:48:54] (03update) 10raymond-ndibe: Draft: add endpoint to get all available images [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T409726) [01:50:14] (03update) 10raymond-ndibe: Draft: add endpoint to get all available images [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T409726) [02:01:01] 10Toolforge (Toolforge iteration 25): [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images - https://phabricator.wikimedia.org/T409191#11365010 (10Raymond_Ndibe) **quick throw-away script for simple deployments in lima-kilo using the web images:** ` #!/usr/bin/env python3... [02:31:29] FIRING: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [02:41:29] RESOLVED: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [04:36:29] FIRING: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [04:46:29] RESOLVED: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [05:18:05] 06cloud-services-team, 10Toolforge: [toolsdb] Automatically terminate long transactions - https://phabricator.wikimedia.org/T409857#11365064 (10Cw95yt5) a:03Cw95yt5 Automatic [06:06:57] 06cloud-services-team, 10Toolforge: [toolsdb] Automatically terminate long transactions - https://phabricator.wikimedia.org/T409857#11365081 (10JJMC89) a:05Cw95yt5→03None [06:36:05] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19288653346 (https://github.com/cluebotng/component-configs/commits/11a18e10782c595ec28a055c4d1f4289f2276be9) [06:36:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [06:36:18] 10Tool-wsindex, 10Wikisource Reader App: Add deletion logic to WSIndex API - https://phabricator.wikimedia.org/T408588#11365088 (10Saiphani02) If we do "--languages bn", it should only update Bengali books and leave other languages as it is. But right now, all books in the db are removed and only bn books... [06:36:27] 10Tool-wsindex, 10Wikisource Reader App: Add deletion logic to WSIndex API - https://phabricator.wikimedia.org/T408588#11365089 (10Saiphani02) 05Resolved→03Open [07:03:12] 06cloud-services-team, 10Toolforge: Make toolsdb pt-heartbeat service automatically follow the primary - https://phabricator.wikimedia.org/T409890 (10fgiunchedi) 03NEW [07:11:38] !log tools.cluebot3 Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289349325 (https://github.com/cluebotng/component-configs/commits/ac67bbd04ad7d30de28f9a7daecf5a7d136f62ef) [07:11:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [07:29:30] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/19289723947 (https://github.com/cluebotng/component-configs/commits/6ad3fbf7ef1281dfed5868d2596d347e01131d18) [07:29:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [07:33:33] (03approved) 10wikigit: Add License Name and URL Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/1 (owner: 10jaredblumer) [07:35:19] (03merge) 10wikigit: Add License Name and URL Rulesets and Corresponding Specs [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/1 (owner: 10jaredblumer) [07:43:29] FIRING: ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [07:48:29] FIRING: [2x] ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [07:53:29] FIRING: [2x] ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [08:03:29] RESOLVED: [2x] ToolforgeToolviewsStale: Toolviews data is stale - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeToolviewsStale - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeToolviewsStale [08:04:12] (03approved) 10dcaro: [dev] CONTRIBUTING.md [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/297 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [08:04:22] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/80 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [08:04:27] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/68 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [08:04:32] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/101 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [08:04:37] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/84 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [08:04:42] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/125 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [08:04:47] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/90 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [08:04:53] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/58 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [08:04:59] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/140 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [08:05:05] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/7 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [08:05:09] (03approved) 10dcaro: [dev] add CONTRIBUTING.md [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/33 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [08:06:38] (03update) 10dcaro: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/buildpacks/clojure-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/clojure-buildpack/-/merge_requests/1 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [08:06:42] (03update) 10dcaro: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/buildpacks/locale-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/locale-buildpack/-/merge_requests/1 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [08:07:17] (03open) 10oluwatumininu: feat(ui): implement breadcrumb navigation for improved contextual navigation [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/169 (https://phabricator.wikimedia.org/T409868) [08:09:33] (03update) 10dcaro: [dev] CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/buildpacks/rust-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/rust-buildpack/-/merge_requests/1 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [08:10:05] (03update) 10oluwatumininu: feat(ui): implement breadcrumb navigation for improved contextual navigation [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/169 [08:12:41] (03approved) 10dcaro: disable_tool.py: fix mypy error [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/25 (owner: 10raymond-ndibe) [08:12:43] (03update) 10dcaro: disable_tool.py: fix mypy error [repos/cloud/toolforge/disable-tool] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/25 (owner: 10raymond-ndibe) [08:14:09] (03update) 10dcaro: tests: use harbor prebuilt images [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/248 (https://phabricator.wikimedia.org/T409727) (owner: 10raymond-ndibe) [08:14:33] (03update) 10dcaro: images: support harbor-based pre-built images [repos/cloud/toolforge/jobs-api] (resolve_harbor_images_every_time) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/252 (https://phabricator.wikimedia.org/T409727) [08:15:19] (03update) 10dcaro: images: support harbor-based pre-built images [repos/cloud/toolforge/jobs-api] (resolve_harbor_images_every_time) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/252 (https://phabricator.wikimedia.org/T409727) [08:15:31] (03update) 10dcaro: images: resolve the image every time [repos/cloud/toolforge/jobs-api] (use_cache_for_harbor) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 [08:16:20] (03update) 10dcaro: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/buildpacks/cmake-buildpack] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/cmake-buildpack/-/merge_requests/1 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [08:16:47] (03update) 10dcaro: d/changelog: bump to 0.0.16 [repos/cloud/toolforge/components-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-cli/-/merge_requests/65 (https://phabricator.wikimedia.org/T400064) (owner: 10raymond-ndibe) [08:17:54] (03update) 10dcaro: values.yml: use harbor images [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/17 (https://phabricator.wikimedia.org/T409727) (owner: 10raymond-ndibe) [08:18:17] (03update) 10dcaro: [dev] add CONTRIBUTING.md, LICENSE [repos/cloud/toolforge/buildpacks/dotnetcore-buildpack] (move_to_api_0.10) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpacks/dotnetcore-buildpack/-/merge_requests/1 (https://phabricator.wikimedia.org/T408783) (owner: 10raymond-ndibe) [08:39:18] (03update) 10dcaro: [maintain-harbor] set quota for toolforge-pre-built [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1076 (https://phabricator.wikimedia.org/T409727) (owner: 10raymond-ndibe) [08:41:25] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [image-config] deprecate and move all data to builds-api - https://phabricator.wikimedia.org/T409728#11365301 (10dcaro) p:05Triage→03Medium [08:41:32] 06cloud-services-team, 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [builds-api,harbor,image-config] Move pre-built images to harbor - https://phabricator.wikimedia.org/T409727#11365302 (10dcaro) p:05Triage→03Medium [08:41:38] 06cloud-services-team, 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [builds-api] Add an endpoint to get all available images - https://phabricator.wikimedia.org/T409726#11365303 (10dcaro) p:05Triage→03Medium [08:41:42] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [jobs-api,webservice] Fetch images from builds-api - https://phabricator.wikimedia.org/T409725#11365304 (10dcaro) p:05Triage→03Medium [09:25:36] (03open) 10dcaro: toolviews: allow it to fail once before alerting [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/48 [09:26:26] (03update) 10dcaro: toolviews: allow it to fail once before alerting [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/48 [09:34:41] 10Toolforge (Toolforge iteration 25): [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images - https://phabricator.wikimedia.org/T409191#11365509 (10dcaro) > quick throw-away script for simple deployments in lima-kilo using the web images: That's ok, but can you test if we c... [09:36:09] (03update) 10dcaro: toolviews: allow it to fail once before alerting [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/48 [10:07:34] 10Tool-wsindex, 10Wikisource Reader App: Add deletion logic to WSIndex API - https://phabricator.wikimedia.org/T408588#11365598 (10System625) Apologies for not seeing that, I have made a fix: https://codeberg.org/ph4ni/wsindex/pulls/8 Please kindly review, let me know if it works well [10:14:33] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [jobs-api,image-config] Deprecate/update the list of supported pre-built images - https://phabricator.wikimedia.org/T409900 (10dcaro) 03NEW [10:14:46] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [jobs-api,image-config] Deprecate/update the list of supported pre-built images - https://phabricator.wikimedia.org/T409900#11365649 (10dcaro) Maybe @komla can help here too [10:15:15] (03open) 10arthurtaylor: Add phabricator session authentication [toolforge-repos/wmde-phabricator-charts] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-phabricator-charts/-/merge_requests/2 [10:15:33] (03approved) 10dcaro: toolviews: allow it to fail once before alerting [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/48 [10:15:36] (03merge) 10dcaro: toolviews: allow it to fail once before alerting [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/48 [10:16:51] (03update) 10dcaro: Apply nodeAffinity when mount=none [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/38 (https://phabricator.wikimedia.org/T408707) (owner: 10damian) [10:21:33] (03approved) 10dcaro: Apply nodeAffinity when mount=none [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/38 (https://phabricator.wikimedia.org/T408707) (owner: 10damian) [10:24:22] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.77-20251112102146-fea77a31 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1077 [10:26:46] (03open) 10damian: server_test - invert nodeSelector check [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/41 [10:32:22] (03update) 10arthurtaylor: Add phabricator session authentication [toolforge-repos/wmde-phabricator-charts] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-phabricator-charts/-/merge_requests/2 [10:33:55] (03merge) 10arthurtaylor: Add phabricator session authentication [toolforge-repos/wmde-phabricator-charts] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-phabricator-charts/-/merge_requests/2 [10:39:33] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [maintain-kubeusers,maintain-dbusers] user homes are not readable by replica_cnf so it fails to create replica.my.cnf files - https://phabricator.wikimedia.org/T409847#11365713 (10dcaro) Merged the above patch, an... [10:40:26] (03approved) 10dcaro: server_test - invert nodeSelector check [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/41 (owner: 10damian) [10:40:34] (03merge) 10dcaro: server_test - invert nodeSelector check [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/41 (owner: 10damian) [10:43:29] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.78-20251112104046-d4fa0766 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1077 [10:43:32] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.78-20251112104046-d4fa0766 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1077 [10:47:24] 06cloud-services-team, 10Toolforge: [toolforge_run_functional_tests] Doesn't support alternate (fork) repo urls, unexpectedly continues on missing branch - https://phabricator.wikimedia.org/T408766#11365752 (10DamianZaremba) 05Open→03Resolved p:05Medium→03Low a:03DamianZaremba Verified this is wo... [10:51:40] (03open) 10arthurtaylor: Fixes [toolforge-repos/wmde-phabricator-charts] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-phabricator-charts/-/merge_requests/3 [10:51:44] (03merge) 10arthurtaylor: Fixes [toolforge-repos/wmde-phabricator-charts] - 10https://gitlab.wikimedia.org/toolforge-repos/wmde-phabricator-charts/-/merge_requests/3 [10:52:59] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11365796 (10fnegri) innochecksum did complete eventually with some information: `lang=shell-session,lines=10 fnegri@tools-db-4... [11:03:57] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [maintain-kubeusers,maintain-dbusers] user homes are not readable by replica_cnf so it fails to create replica.my.cnf files - https://phabricator.wikimedia.org/T409847#11365835 (10fnegri) > the error counter might... [11:12:43] 06cloud-services-team (FY2025/26-Q1), 10Toolforge: [toolsdb] Add users and grants for IPv6, remove obsolete ones - https://phabricator.wikimedia.org/T409563#11365867 (10taavi) >>! In T409563#11363899, @fnegri wrote: > @taavi do you agree that the security group rule with `208.80.154.149/32` can be deleted? Was... [11:15:40] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11365878 (10fnegri) The output from innochecksum is not very easy to read, but I think it confirms the theory that the increase... [11:53:06] 06cloud-services-team, 10Toolforge: [toolsdb] pt-heartbeat service should automatically follow the primary - https://phabricator.wikimedia.org/T409890#11366022 (10fnegri) p:05Triage→03Medium [12:02:53] 10Tool-paulina: Fix error handling and add request timeouts to prevent indefinite hangs - https://phabricator.wikimedia.org/T409914 (10System625) 03NEW [12:04:10] 10Tool-paulina, 13Patch-For-Review: Fix error handling and add request timeouts to prevent indefinite hangs - https://phabricator.wikimedia.org/T409914#11366082 (10System625) a:03System625 [12:04:32] (03update) 10dcaro: [jobs-cli] refactor job payload [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [12:05:09] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25): [toolsdb] Add users and grants for IPv6, remove obsolete ones - https://phabricator.wikimedia.org/T409563#11366085 (10fnegri) 05In progress→03Resolved I deleted the security group rule with `208.80.154.149/32` using Horizon. C... [12:09:59] 06cloud-services-team, 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [builds-api,harbor,image-config] Move pre-built images to harbor - https://phabricator.wikimedia.org/T409727#11366121 (10dcaro) [12:11:12] 06cloud-services-team, 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [builds-api,harbor,image-config] Move pre-built images to harbor - https://phabricator.wikimedia.org/T409727#11366146 (10dcaro) [12:16:54] (03open) 10system625: Add comprehensive error handling and HTTP request timeouts [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/170 (https://phabricator.wikimedia.org/T409914) [12:37:37] 06cloud-services-team (FY2025/26-Q1), 10Toolforge: [toolsdb] crash recovery can fail because of insufficient innodb_log_file_size - https://phabricator.wikimedia.org/T409922 (10fnegri) 03NEW [12:40:54] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] crash recovery can fail because of insufficient innodb_log_file_size - https://phabricator.wikimedia.org/T409922#11366254 (10fnegri) 05Open→03In progress p:05Triage→03High a:03fnegri [12:59:17] FIRING: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:13:14] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 13Patch-For-Review, 07Sustainability (Incident Followup): [toolsdb] crash recovery can fail because of insufficient innodb_log_file_size - https://phabricator.wikimedia.org/T409922#11366428 (10fnegri) This blog post has a lot of info on how to pick a good... [13:19:47] RESOLVED: JobUnavailable: Reduced availability for job maintain_dbusers_eqiad in cloud@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:43:25] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [13:45:42] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [14:44:36] 06cloud-services-team, 10Toolforge, 06Infrastructure-Foundations, 10netops: Create new VRF and networks for Toolforge-on-Metal - https://phabricator.wikimedia.org/T409309#11366850 (10cmooney) So a few things emerged after the call today: * We need a device that will provide NAT (IPv4) and firewalling (IPv... [15:03:12] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [jobs-api,image-config] Deprecate/update the list of supported pre-built images - https://phabricator.wikimedia.org/T409900#11366974 (10taavi) p:05Triage→03Medium [15:03:30] 06cloud-services-team, 10Toolforge: Track global and per-tool concurrent requests and 503 rate limitiing responses from Toolforge CDN edge - https://phabricator.wikimedia.org/T409794#11366976 (10taavi) p:05Triage→03High [15:04:57] 06cloud-services-team, 10Toolforge: See if we can borrow parts of the wikiprod WAF for Toolforge - https://phabricator.wikimedia.org/T409759#11366981 (10taavi) p:05Triage→03High [15:06:10] 06cloud-services-team, 10Tool-iw, 10Toolforge: Toolforge interwiki link handling no longer strips URL-encoding before redirecting when it previously did, breaking existing on-wiki links - https://phabricator.wikimedia.org/T409493#11366985 (10taavi) p:05Triage→03High [15:07:25] 06cloud-services-team, 10Tool-iw, 10Toolforge: iw.toolforge.org does not support URL-encoded query parameters ([[toolforge:foo?bar]]) - https://phabricator.wikimedia.org/T345783#11367001 (10taavi) p:05Triage→03Medium [15:07:48] 06cloud-services-team, 10Cloud-VPS, 10CAS-SSO, 06Infrastructure-Foundations: sso failure in codfw1dev (labtesthorizon.wikimedia.org) - https://phabricator.wikimedia.org/T409328#11367003 (10taavi) [15:08:54] 06cloud-services-team, 10Toolforge: [functional tests] Consider adding tool outputs on failure - https://phabricator.wikimedia.org/T409280#11367007 (10taavi) p:05Triage→03Low [15:09:57] 06cloud-services-team, 10wikitech.wikimedia.org: Flapping wikitech-static icinga alert - https://phabricator.wikimedia.org/T409029#11367018 (10taavi) p:05Triage→03High a:03Andrew [15:11:34] 06cloud-services-team, 10Toolforge: [jobs-api] failed to create job from components - https://phabricator.wikimedia.org/T409007#11367039 (10taavi) p:05Triage→03Medium a:03dcaro [15:13:08] 06cloud-services-team, 10Toolforge, 06Release-Engineering-Team: [toolforge_deploy_mr] Branches can only be tested for 24 hours, otherwise fail with CI too old. - https://phabricator.wikimedia.org/T408999#11367047 (10taavi) Dear #together, do you have opinions about raising reggie retention time? [15:15:02] 06cloud-services-team, 10Cloud-VPS: `logging` project missing normal DNS zone delegation - https://phabricator.wikimedia.org/T409361#11367060 (10taavi) The script to fix this is https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Maintenance#wmcs-makedomain. [15:17:09] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission [15:19:47] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission [15:25:05] (03approved) 10dcaro: volume-admission: bump to 0.0.78-20251112104046-d4fa0766 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1077 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:25:09] (03merge) 10dcaro: volume-admission: bump to 0.0.78-20251112104046-d4fa0766 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1077 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:26:04] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [jobs-api] failed to create job from components - https://phabricator.wikimedia.org/T409007#11367158 (10dcaro) [15:26:23] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [jobs-api] failed to create job from components - https://phabricator.wikimedia.org/T409007#11367161 (10dcaro) 05Open→03Resolved [15:33:26] 06cloud-services-team (FY2025/26-Q1), 10Toolforge, 07Sustainability (Incident Followup): [toolsdb] Destroy tools-db-4 and create new host - https://phabricator.wikimedia.org/T409287#11367207 (10fnegri) [15:50:22] 06cloud-services-team, 10Cloud-VPS: Tell Neutron the physical network has jumbo frames enabled - https://phabricator.wikimedia.org/T409544#11367305 (10taavi) Setting the network MTU does update the Neutron router ports, but not the guest VMs interfaces. I found related https://bugs.launchpad.net/neutron/+bug/2... [16:08:51] 06cloud-services-team, 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team (Priority Backlog 📥): gitlab: consider enabling docker container registry - https://phabricator.wikimedia.org/T304845#11367372 (10DPogorzelski-WMF) The ML team needs a place where to store large LLM docker contain... [16:21:40] 06cloud-services-team, 10Cloud-VPS: Tell Neutron the physical network has jumbo frames enabled - https://phabricator.wikimedia.org/T409544#11367432 (10taavi) A `os server stop $id ; sleep N ; os server start $id` cycle is enough to update the MTU. [16:21:49] 06cloud-services-team, 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [jobs-api] handle qualified image names - https://phabricator.wikimedia.org/T408574#11367433 (10dcaro) 05Open→03Resolved a:03dcaro [16:24:15] 10Toolforge (Toolforge iteration 25): [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images - https://phabricator.wikimedia.org/T409191#11367444 (10Raymond_Ndibe) >>! In T409191#11365509, @dcaro wrote: >> quick throw-away script for simple deployments in lima-kilo using the... [16:42:57] (03update) 10dcaro: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [16:43:07] (03update) 10dcaro: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) (owner: 10raymond-ndibe) [16:46:46] 10Toolforge (Toolforge iteration 25): [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images - https://phabricator.wikimedia.org/T409191#11367577 (10dcaro) >>! In T409191#11367444, @Raymond_Ndibe wrote: >>>! In T409191#11365509, @dcaro wrote: >>> quick throw-away script for s... [16:47:44] (03update) 10dcaro: toolforge_depoy: removed extension [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/296 [17:26:00] (03update) 10raymond-ndibe: Draft: add endpoint to get all available images [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T409726) [17:26:51] (03update) 10raymond-ndibe: add endpoint to get all available images [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T409726) [17:59:42] 06cloud-services-team, 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team (Priority Backlog 📥): gitlab: consider enabling docker container registry - https://phabricator.wikimedia.org/T304845#11367940 (10thcipriani) >>! In T304845#11367372, @DPogorzelski-WMF wrote: > The ML team needs a... [18:27:21] 06cloud-services-team, 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team (Priority Backlog 📥): gitlab: consider enabling docker container registry - https://phabricator.wikimedia.org/T304845#11368054 (10thcipriani) 05Stalled→03Declined Declining this task to be clear we have no... [19:39:58] 06cloud-services-team, 10Cloud-VPS (Project-requests): CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387#11368280 (10Andrew) >>! In T408387#11357872, @Odinaldo wrote: > Hi Andrew, > > Many thanks for your response. We are trying to find out whom to contact to move this forward and unders... [20:23:37] 06cloud-services-team, 06SRE: latest Trixie image (as of 2025-10-16) grub failure on R450 hardware - https://phabricator.wikimedia.org/T407586#11368447 (10Andrew) I just ran a couple more tests: 1) Installed host, paused at the end of install 2) Installed new grub packages (grub-common, grub2-common, grub-pc-... [21:02:26] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [21:03:24] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30031 bytes in 6.200 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [21:33:12] 10Cloud-VPS (Quota-requests): Increase volume storage on project analytics - https://phabricator.wikimedia.org/T409970 (10xcollazo) 03NEW [21:40:33] 06cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q2:rack/setup/install clouddb1026-1033 - https://phabricator.wikimedia.org/T409162#11368668 (10Andrew) a:05Andrew→03None [21:44:52] (03PS1) 10Eevans: Add (fake) revise_tone_task_generator password [labs/private] - 10https://gerrit.wikimedia.org/r/1204682 [21:46:41] (03CR) 10Eevans: [V:03+2 C:03+2] Add (fake) revise_tone_task_generator password [labs/private] - 10https://gerrit.wikimedia.org/r/1204682 (owner: 10Eevans) [21:47:30] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [21:48:30] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30037 bytes in 8.920 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [21:51:30] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [21:52:22] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30039 bytes in 2.352 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [22:04:14] PROBLEM - HTTPS-wikitech-static on wikitech-static.wikimedia.org is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/Wikitech-static [22:04:20] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: connect to address wikitech-static.wikimedia.org and port 443: Connection refused https://wikitech.wikimedia.org/wiki/Wikitech-static [22:05:14] RECOVERY - HTTPS-wikitech-static on wikitech-static.wikimedia.org is OK: SSL OK - Certificate status.wikimedia.org valid until 2025-12-28 19:04:50 +0000 (expires in 45 days) https://wikitech.wikimedia.org/wiki/Wikitech-static [22:05:20] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30037 bytes in 0.201 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [22:37:29] 10Toolforge (Quota-requests): Request increased quota for MilHistBot Toolforge tool - https://phabricator.wikimedia.org/T409981 (10Hawkeye7) 03NEW [22:39:27] 10Toolforge (Quota-requests): Request increased build quota for MilHistBot Toolforge tool - https://phabricator.wikimedia.org/T409981#11368876 (10Hawkeye7) [23:39:42] 10VPS-project-Phabricator, 06collaboration-services, 06Release-Engineering-Team (Radar): 'Fulltext' searches fail on test Phab instance due to ElasticSearch default config (PhutilAggregateException: All Fulltext Search hosts failed / CURLE_COULDNT_CONNECT) - https://phabricator.wikimedia.org/T403948#11368980 (... [23:48:31] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [23:56:25] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30031 bytes in 3.241 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static