[00:16:07] 06cloud-services-team, 10Toolforge: [toolforge_run_functional_tests] branch doesn't work on forks - https://phabricator.wikimedia.org/T408766#11325867 (10DamianZaremba) [00:16:37] 06cloud-services-team, 10Toolforge: [toolforge_run_functional_tests] copy/paste format doesn't work - https://phabricator.wikimedia.org/T408767#11325870 (10DamianZaremba) [00:18:19] (03open) 10damian: Draft: Add helper function for deployment status [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1027 (https://phabricator.wikimedia.org/T408002) [00:20:49] (03update) 10damian: Draft: Add helper function for deployment status [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1027 (https://phabricator.wikimedia.org/T408002) [00:31:30] (03update) 10damian: Draft: Add helper function for deployment status [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1027 (https://phabricator.wikimedia.org/T408002) [00:39:08] (03update) 10damian: Draft: Add helper function for deployment status [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1027 (https://phabricator.wikimedia.org/T408002) [00:45:53] (03update) 10damian: Draft: Add helper function for deployment status [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1027 (https://phabricator.wikimedia.org/T408002) [00:50:26] (03update) 10damian: Draft: Add helper function for deployment status [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1027 (https://phabricator.wikimedia.org/T408002) [01:03:15] (03update) 10damian: Draft: runtime - configure topologySpreadConstraints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235 (https://phabricator.wikimedia.org/T408707) [01:20:27] (03update) 10damian: Draft: runtime - configure topologySpreadConstraints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235 (https://phabricator.wikimedia.org/T408707) [01:23:08] (03update) 10damian: Draft: runtime - configure topologySpreadConstraints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235 (https://phabricator.wikimedia.org/T408707) [01:25:33] (03update) 10damian: Draft: runtime - configure topologySpreadConstraints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235 (https://phabricator.wikimedia.org/T408707) [01:31:47] (03update) 10damian: runtime - configure topologySpreadConstraints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235 (https://phabricator.wikimedia.org/T408707) [01:31:54] (03update) 10damian: runtime - configure topologySpreadConstraints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235 (https://phabricator.wikimedia.org/T408707) [01:43:54] (03update) 10damian: runtime - configure topologySpreadConstraints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235 (https://phabricator.wikimedia.org/T408707) [02:41:03] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudservices2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [02:54:19] (03PS1) 10Wandji collins: Landing page loads dynamic data, update display of actor detail page. [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1199912 [02:57:32] (03PS2) 10Wandji collins: Landing page loads dynamic data, update display of actor detail page. [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1199912 [06:41:03] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudservices2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [07:30:59] 06cloud-services-team, 10Cloud-VPS: MTU setting in IPv6 VMs causes issues with Docker - https://phabricator.wikimedia.org/T408543#11326217 (10fgiunchedi) >>! In T408543#11323453, @taavi wrote: > A major downside of doing it via Puppet is that that won't have any effect on instances with non-Puppetized services... [08:42:15] 10Toolforge (Toolforge iteration 25): [docs] update all readmes with the same deployment docs - https://phabricator.wikimedia.org/T407477#11326342 (10dcaro) [08:42:16] 10Toolforge (Toolforge iteration 25): [tools,infra,k8s] scale up the cluster, specifically CPU - https://phabricator.wikimedia.org/T404726#11326343 (10dcaro) [08:42:18] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): toolforge logs appears to suffer from intermittent latency - https://phabricator.wikimedia.org/T402736#11326344 (10dcaro) [08:42:24] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#11326345 (10dcaro) [08:42:26] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25), 07Epic: [KR] WE6.3 Introduce a sustainability scoring system for the Toolforge platform - https://phabricator.wikimedia.org/T368600#11326348 (10dcaro) [08:42:34] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [jobs-api] when running a command with wrong quoting, no logs nor useful feedback is given to the user - https://phabricator.wikimedia.org/T356267#11326346 (10dcaro) [08:42:35] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#11326347 (10dcaro) [08:42:37] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25): [builds-builder] Upgrade python buildpack to v0.17.0 or newer for Poetry support - https://phabricator.wikimedia.org/T374056#11326349 (10dcaro) [08:42:39] 10Toolforge (Toolforge iteration 25): [clis] standardize the package names - https://phabricator.wikimedia.org/T399080#11326351 (10dcaro) [08:42:41] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25): [components-api,beta] Config not updated from remote source - https://phabricator.wikimedia.org/T401868#11326352 (10dcaro) [08:42:43] 10Cloud Services Proposals, 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25), 05Cloud-Services-Origin-Team, and 3 others: [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-d... - https://phabricator.wikimedia.org/T194332#11326350 [08:42:44] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25): [lima-kilo] toolforge components are reported as changed on every ansible run - https://phabricator.wikimedia.org/T402689#11326353 (10dcaro) [08:42:47] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25): [lima-kilo] Only download artefacts if target binary checksum does not match - https://phabricator.wikimedia.org/T402684#11326354 (10dcaro) [08:43:32] 10Toolforge (Toolforge iteration 25): [api-gateway] can't retrieve openapi.json - https://phabricator.wikimedia.org/T407957#11326368 (10dcaro) [08:43:34] 10Toolforge (Toolforge iteration 25): [prometheus,infra] 2025-09-10 tools-prometheus-9 down - https://phabricator.wikimedia.org/T404199#11326371 (10dcaro) [08:43:37] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): Update maintain_kubeusers to use the toolstate database - https://phabricator.wikimedia.org/T334629#11326370 (10dcaro) [08:43:40] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [jobs-api] Allow customizing time to request Loki logs for - https://phabricator.wikimedia.org/T400917#11326372 (10dcaro) [08:43:41] 10Toolforge (Toolforge iteration 25): [components-api,beta] Image should only be build once when re-used in components - https://phabricator.wikimedia.org/T401851#11326374 (10dcaro) [08:43:43] 10Toolforge (Toolforge iteration 25): [components-api] bump the openapi version on every change - https://phabricator.wikimedia.org/T401374#11326375 (10dcaro) [08:43:46] 06cloud-services-team, 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [components-api] allow specifying `source_repo`+`ref` for the config - https://phabricator.wikimedia.org/T402764#11326373 (10dcaro) [08:43:48] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [builds-builder] review and potentially add cmake buildpack - https://phabricator.wikimedia.org/T401169#11326376 (10dcaro) [08:43:49] 10Toolforge (Toolforge iteration 25): [foxtrot-ldap] publish image in harbor repos - https://phabricator.wikimedia.org/T400167#11326377 (10dcaro) [08:43:52] 10Toolforge (Toolforge iteration 25): [docs] enable docs linter in one of the repos - https://phabricator.wikimedia.org/T397949#11326378 (10dcaro) [08:43:56] 10Toolforge (Toolforge iteration 25): [builds-cli] add resolved reference when showing a build - https://phabricator.wikimedia.org/T394300#11326379 (10dcaro) [08:44:00] 06cloud-services-team, 10Toolforge (Toolforge iteration 25): [components-api] optionally log deployments to SAL automatically - https://phabricator.wikimedia.org/T393169#11326380 (10dcaro) [08:44:04] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25), 07Epic: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice - https://phabricator.wikimedia.org/T388092#11326381 (10dcaro) [09:04:13] 10Cloud Services Proposals, 06cloud-services-team: Decision request template - Update and expand on python coding standards for toolforge - https://phabricator.wikimedia.org/T408782 (10dcaro) 03NEW [09:04:24] 10Cloud Services Proposals, 06cloud-services-team: Decision request template - Update and expand on python coding standards for toolforge - https://phabricator.wikimedia.org/T408782#11326428 (10dcaro) p:05Triage→03Medium [09:16:36] 10Toolforge (Toolforge iteration 25): [docs] Update all toolforge repos in gitlab with contribution guidelines and license - https://phabricator.wikimedia.org/T408783 (10dcaro) 03NEW [09:23:35] 10Cloud Services Proposals, 06cloud-services-team: Decision request - Update and expand on python coding standards for toolforge - https://phabricator.wikimedia.org/T408782#11326468 (10dcaro) [09:24:31] 10Cloud Services Proposals, 06cloud-services-team: Decision request - Update and expand on python coding standards for toolforge - https://phabricator.wikimedia.org/T408782#11326488 (10dcaro) [09:26:27] 10Toolforge (Toolforge iteration 25): [docs] Update all toolforge repos in gitlab with contribution guidelines and license - https://phabricator.wikimedia.org/T408783#11326496 (10dcaro) [09:33:24] 10Toolforge (Toolforge iteration 25): [docs] Update all toolforge repos in gitlab with contribution guidelines and license - https://phabricator.wikimedia.org/T408783#11326510 (10dcaro) [09:46:45] 06cloud-services-team, 10Toolforge, 07Sustainability (Incident Followup): [k8s,infra] scale up coredns replicas - https://phabricator.wikimedia.org/T333934#11326520 (10taavi) Kubernetes 1.31 adds support for applying patches to the CoreDNS deployment that could be used for this: https://github.com/kubernetes... [09:53:10] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.33 - https://phabricator.wikimedia.org/T408785 (10taavi) 03NEW p:05Triage→03Medium [09:53:18] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.33 - https://phabricator.wikimedia.org/T408785#11326549 (10taavi) [09:53:19] 06cloud-services-team, 10Toolforge: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.32 - https://phabricator.wikimedia.org/T379047#11326550 (10taavi) [09:55:12] 06cloud-services-team, 10Toolforge: Upgrade cert-manager past 1.15 - https://phabricator.wikimedia.org/T408786 (10taavi) 03NEW [09:55:13] 06cloud-services-team, 10Toolforge: [striker] tool creation validation is not applying project prefix consistently - https://phabricator.wikimedia.org/T408787 (10DamianZaremba) 03NEW [09:55:15] 10Cloud Services Proposals, 06cloud-services-team: Decision request - Update and expand on coding standards for toolforge - https://phabricator.wikimedia.org/T408782#11326573 (10dcaro) [09:56:10] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [infra,k8s] Upgrade Toolforge Kubernetes to version 1.31 - https://phabricator.wikimedia.org/T372697#11326575 (10taavi) [09:57:43] 06cloud-services-team, 10Striker: [striker] tool creation validation is not applying project prefix consistently - https://phabricator.wikimedia.org/T408787#11326579 (10taavi) [10:00:14] (03PS1) 10DamianZaremba: check_toolname_create - use prefixed name [labs/striker] - 10https://gerrit.wikimedia.org/r/1200014 (https://phabricator.wikimedia.org/T408787) [10:00:56] 06cloud-services-team, 10Striker: Add 3 maintainer to tool lists - https://phabricator.wikimedia.org/T326552#11326582 (10taavi) 05Open→03Declined Per Bryan. [10:03:44] (03close) 10damian: Draft: Retry deployments for 10min [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1006 (https://phabricator.wikimedia.org/T408002) [10:03:45] 06cloud-services-team, 10Striker: [striker] tool creation validation is not applying project prefix consistently - https://phabricator.wikimedia.org/T408787#11326586 (10fgiunchedi) p:05Triage→03Medium [10:03:48] 06cloud-services-team, 10Toolforge: Upgrade cert-manager past 1.15 - https://phabricator.wikimedia.org/T408786#11326587 (10fgiunchedi) p:05Triage→03Medium [10:04:18] 06cloud-services-team, 10Toolforge: [toolforge_run_functional_tests] copy/paste format doesn't work - https://phabricator.wikimedia.org/T408767#11326589 (10fgiunchedi) p:05Triage→03Medium [10:04:28] 06cloud-services-team, 10Toolforge: [toolforge_run_functional_tests] branch doesn't work on forks - https://phabricator.wikimedia.org/T408766#11326590 (10fgiunchedi) p:05Triage→03Medium [10:04:31] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [jobs-api] apply topology constraints - https://phabricator.wikimedia.org/T408707#11326591 (10fgiunchedi) p:05Triage→03Medium [10:10:57] 10Toolforge (Toolforge iteration 25): [docs] Update all toolforge repos in gitlab with contribution guidelines and license - https://phabricator.wikimedia.org/T408783#11326602 (10dcaro) p:05Triage→03Medium [10:12:15] (03CR) 10Majavah: [C:03+2] Always expose observed external URI when creating Diffusion repos [labs/striker] - 10https://gerrit.wikimedia.org/r/1196901 (https://phabricator.wikimedia.org/T407624) (owner: 10Aklapper) [10:15:13] (03CR) 10Majavah: [C:03+2] Set http IO to none when creating Diffusion repositories [labs/striker] - 10https://gerrit.wikimedia.org/r/1197035 (https://phabricator.wikimedia.org/T407705) (owner: 10Aklapper) [10:15:35] (03Merged) 10jenkins-bot: Always expose observed external URI when creating Diffusion repos [labs/striker] - 10https://gerrit.wikimedia.org/r/1196901 (https://phabricator.wikimedia.org/T407624) (owner: 10Aklapper) [10:17:07] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [jobs-api] apply topology constraints - https://phabricator.wikimedia.org/T408707#11326613 (10dcaro) Note that we have not had any worker get stuck anymore since the last OS upgrade of the NFS servers, so the differentiation of nfs/non-nfs might be less... [10:17:17] (03update) 10dcaro: runtime - configure topologySpreadConstraints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235 (https://phabricator.wikimedia.org/T408707) (owner: 10damian) [10:17:59] (03update) 10dcaro: runtime - configure topologySpreadConstraints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235 (https://phabricator.wikimedia.org/T408707) (owner: 10damian) [10:18:04] (03Merged) 10jenkins-bot: Set http IO to none when creating Diffusion repositories [labs/striker] - 10https://gerrit.wikimedia.org/r/1197035 (https://phabricator.wikimedia.org/T407705) (owner: 10Aklapper) [10:18:42] (03open) 10damian: components-api - Use helper for status checks [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1028 (https://phabricator.wikimedia.org/T408002) [10:21:19] (03update) 10dcaro: components-api - Use helper for status checks [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1028 (https://phabricator.wikimedia.org/T408002) (owner: 10damian) [10:22:50] (03open) 10damian: components-api - drop redundant config calls [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1029 [10:23:08] (03update) 10damian: components-api - drop redundant config calls [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1029 (https://phabricator.wikimedia.org/T408002) [10:23:45] (03close) 10damian: Draft: Add helper function for deployment status [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1027 (https://phabricator.wikimedia.org/T408002) [10:26:35] (03approved) 10dcaro: components-api - Use helper for status checks [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1028 (https://phabricator.wikimedia.org/T408002) (owner: 10damian) [10:35:10] (03CR) 10Majavah: "Did you test this locally? To me it seems like Bitu's username normalization logic is breaking this, for example https://toolsadmin.wikime" [labs/striker] - 10https://gerrit.wikimedia.org/r/1200014 (https://phabricator.wikimedia.org/T408787) (owner: 10DamianZaremba) [10:35:13] (03CR) 10Majavah: "recheck" [labs/striker] - 10https://gerrit.wikimedia.org/r/1200014 (https://phabricator.wikimedia.org/T408787) (owner: 10DamianZaremba) [10:37:51] (03update) 10dcaro: requirements - use pyuwsgi [toolforge-repos/sample-static-buildpack-app] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-static-buildpack-app/-/merge_requests/1 (https://phabricator.wikimedia.org/T408002) (owner: 10damian) [10:41:03] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudservices2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [10:46:11] (03PS2) 10DamianZaremba: check_toolname_create - use prefixed name [labs/striker] - 10https://gerrit.wikimedia.org/r/1200014 (https://phabricator.wikimedia.org/T408787) [10:48:46] (03CR) 10DamianZaremba: "Not yet - I don't have docker installed locally so the compose flow in the docs doesn't work and running `tox` results in almost every fil" [labs/striker] - 10https://gerrit.wikimedia.org/r/1200014 (https://phabricator.wikimedia.org/T408787) (owner: 10DamianZaremba) [10:49:41] 10Toolforge (Toolforge iteration 25): [maintain-harbor] Failing to cleanup stale artifacts - https://phabricator.wikimedia.org/T407496#11326708 (10dcaro) 05In progress→03Resolved [10:55:41] (03approved) 10fnegri: requirements - use pyuwsgi [toolforge-repos/sample-static-buildpack-app] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-static-buildpack-app/-/merge_requests/1 (https://phabricator.wikimedia.org/T408002) (owner: 10damian) [10:56:35] (03approved) 10dcaro: components-api - drop redundant config calls [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1029 (https://phabricator.wikimedia.org/T408002) (owner: 10damian) [10:56:58] (03merge) 10dcaro: components-api - drop redundant config calls [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1029 (https://phabricator.wikimedia.org/T408002) (owner: 10damian) [10:57:14] (03merge) 10dcaro: requirements - use pyuwsgi [toolforge-repos/sample-static-buildpack-app] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-static-buildpack-app/-/merge_requests/1 (https://phabricator.wikimedia.org/T408002) (owner: 10damian) [10:57:19] (03update) 10dcaro: requirements - use pyuwsgi [toolforge-repos/sample-static-buildpack-app] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-static-buildpack-app/-/merge_requests/1 (https://phabricator.wikimedia.org/T408002) (owner: 10damian) [10:57:29] (03open) 10damian: components-api - use $TOOLFORGE_API_URL [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1030 (https://phabricator.wikimedia.org/T408002) [11:12:54] (03update) 10dcaro: update deployment via template hash change [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/220 (https://phabricator.wikimedia.org/T403321) (owner: 10damian) [11:13:18] (03approved) 10dcaro: update deployment via template hash change [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/220 (https://phabricator.wikimedia.org/T403321) (owner: 10damian) [11:14:33] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component logs-api [11:14:49] (03merge) 10dcaro: update deployment via template hash change [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/220 (https://phabricator.wikimedia.org/T403321) (owner: 10damian) [11:16:26] (03update) 10dcaro: logs-api: bump to 0.0.4-20251029150446-a8982601 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1024 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [11:17:25] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.433-20251030111500-709a8c88 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1031 (https://phabricator.wikimedia.org/T403321) [11:18:29] (03update) 10damian: components-api - Use helper for status checks [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1028 (https://phabricator.wikimedia.org/T408002) [11:21:14] (03update) 10damian: components-api - Use helper for status checks [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1028 (https://phabricator.wikimedia.org/T408002) [11:22:11] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [11:22:20] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [11:24:05] (03update) 10damian: runtime - configure topologySpreadConstraints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235 (https://phabricator.wikimedia.org/T408707) [11:40:50] 06cloud-services-team (FY2025/26-Q1), 10Data-Services: [wikireplicas] Create views for new wiki minwikisource - https://phabricator.wikimedia.org/T408346#11326815 (10fnegri) 05Open→03In progress [11:40:57] 06cloud-services-team (FY2025/26-Q1), 10Data-Services: [wikireplicas] Create views for new wiki pcmwikiquote - https://phabricator.wikimedia.org/T408354#11326817 (10fnegri) 05Open→03In progress [11:44:01] 06cloud-services-team (FY2025/26-Q1), 10Toolforge: jobs-api limits quota in restrictive ways - https://phabricator.wikimedia.org/T403962#11326841 (10fnegri) [11:44:10] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25): toolforge logs appears to suffer from intermittent latency - https://phabricator.wikimedia.org/T402736#11326845 (10fnegri) [11:50:21] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logs-api [11:58:18] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api [12:30:34] (03open) 10taavi: Add persistent volume for LDAP data [repos/cloud/toolforge/foxtrot-ldap] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/foxtrot-ldap/-/merge_requests/11 [12:30:39] (03update) 10taavi: Add persistent volume for LDAP data [repos/cloud/toolforge/foxtrot-ldap] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/foxtrot-ldap/-/merge_requests/11 [12:31:01] (03update) 10taavi: Add persistent volume for LDAP data [repos/cloud/toolforge/foxtrot-ldap] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/foxtrot-ldap/-/merge_requests/11 [12:33:31] (03update) 10damian: components-api - Use helper for status checks [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1028 (https://phabricator.wikimedia.org/T408002) [12:36:08] (03update) 10damian: components-api - Use helper for status checks [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1028 (https://phabricator.wikimedia.org/T408002) [12:37:19] (03update) 10damian: components-api - Use helper for status checks [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1028 (https://phabricator.wikimedia.org/T408002) [12:38:06] (03update) 10damian: components-api - Use helper for status checks [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1028 (https://phabricator.wikimedia.org/T408002) [12:42:17] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/50 [12:45:52] (03update) 10damian: components-api - Use helper for status checks [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1028 (https://phabricator.wikimedia.org/T408002) [12:54:10] (03open) 10damian: copy/paste - don't quote extra_args [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1032 (https://phabricator.wikimedia.org/T408767) [12:57:02] (03open) 10damian: deploy-tests - no flush if skipping [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1033 [12:57:30] (03update) 10damian: components-api - Use helper for status checks [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1028 (https://phabricator.wikimedia.org/T408002) [12:59:03] (03update) 10damian: deploy-tests - no flush if skipping [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1033 [13:01:17] (03approved) 10dcaro: copy/paste - don't quote extra_args [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1032 (https://phabricator.wikimedia.org/T408767) (owner: 10damian) [13:01:21] (03merge) 10dcaro: copy/paste - don't quote extra_args [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1032 (https://phabricator.wikimedia.org/T408767) (owner: 10damian) [13:02:03] (03approved) 10dcaro: logs-api: bump to 0.0.4-20251029150446-a8982601 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1024 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:02:09] (03update) 10dcaro: logs-api: bump to 0.0.4-20251029150446-a8982601 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1024 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:02:31] (03merge) 10dcaro: logs-api: bump to 0.0.4-20251029150446-a8982601 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1024 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:03:09] (03approved) 10dcaro: deploy-tests - no flush if skipping [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1033 (owner: 10damian) [13:03:17] (03update) 10dcaro: deploy-tests - no flush if skipping [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1033 (owner: 10damian) [13:03:41] (03merge) 10dcaro: deploy-tests - no flush if skipping [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1033 (owner: 10damian) [13:04:37] (03update) 10dcaro: components-api - Use helper for status checks [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1028 (https://phabricator.wikimedia.org/T408002) (owner: 10damian) [13:06:30] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge_run_functional_tests] copy/paste format doesn't work - https://phabricator.wikimedia.org/T408767#11327041 (10DamianZaremba) 05Open→03Resolved a:03DamianZaremba [13:13:55] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [jobs-api] apply topology constraints - https://phabricator.wikimedia.org/T408707#11327056 (10DamianZaremba) >>! In T408707#11326613, @dcaro wrote: > Note that we have not had any worker get stuck anymore since the last OS upgrade of the NFS servers, so... [13:17:52] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [13:18:37] (03update) 10dcaro: builds-builder: bump to 0.0.136-20251029145852-cb090eff [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1023 (https://phabricator.wikimedia.org/T408140) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:22:14] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [13:45:44] (03approved) 10fnegri: Add persistent volume for LDAP data [repos/cloud/toolforge/foxtrot-ldap] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/foxtrot-ldap/-/merge_requests/11 (owner: 10taavi) [14:34:41] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-builder [14:37:46] (03merge) 10taavi: Add persistent volume for LDAP data [repos/cloud/toolforge/foxtrot-ldap] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/foxtrot-ldap/-/merge_requests/11 [14:39:19] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder [14:39:19] (03update) 10dcaro: components-api - use $TOOLFORGE_API_URL [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1030 (https://phabricator.wikimedia.org/T408002) (owner: 10damian) [14:39:24] 10Tool-wsindex, 10Wikisource Reader App: Add deletion logic to WSIndex API - https://phabricator.wikimedia.org/T408588#11327547 (10System625) Hello! Can I take this task? [14:43:51] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudservices2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [14:51:02] 06cloud-services-team, 10Cloud-VPS, 07Epic: Cloud VPS: extend tofu-infra coverage - https://phabricator.wikimedia.org/T370037#11327601 (10Andrew) > "admin-defined" resources I think that admin-defined just means 'things in cloud-vps managed and supported by staff rather than by random users'. [14:59:01] (03approved) 10dcaro: builds-builder: bump to 0.0.136-20251029145852-cb090eff [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1023 (https://phabricator.wikimedia.org/T408140) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:59:07] (03merge) 10dcaro: builds-builder: bump to 0.0.136-20251029145852-cb090eff [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1023 (https://phabricator.wikimedia.org/T408140) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:00:16] 06cloud-services-team (FY2025/26-Q1), 10Data-Services: [wikireplicas] Create views for new wiki pcmwikiquote - https://phabricator.wikimedia.org/T408354#11327674 (10fnegri) 05In progress→03Resolved [15:02:00] 06cloud-services-team (FY2025/26-Q1), 10Data-Services: [wikireplicas] Create views for new wiki minwikisource - https://phabricator.wikimedia.org/T408346#11327686 (10fnegri) 05In progress→03Resolved Verified this is working: `lang=shell-session fnegri@tools-bastion-14:~$ sql minwikisource [...] Mari... [15:08:01] 10Cloud Services Proposals, 06cloud-services-team: Decision request - Update and expand on coding standards for toolforge - https://phabricator.wikimedia.org/T408782#11327713 (10dcaro) [15:16:11] 06cloud-services-team (FY2025/26-Q1), 10Data-Services: [wikireplicas] Create views for new wiki pcmwikiquote - https://phabricator.wikimedia.org/T408354#11327771 (10fnegri) Verified this is working: `lang=shell-session fnegri@tools-bastion-14:~$ sql pcmwikiquote [...] MariaDB [pcmwikiquote_p]> select co... [15:41:40] 06cloud-services-team, 10Cloud-VPS (Project-requests): CloudVPS instance for ProVe - https://phabricator.wikimedia.org/T408387#11328001 (10Andrew) > we're told Gadgets shouldn't call external services Can you elaborate on this? Is there a written policy you're following, or a particular team or engineer who'... [15:48:49] (03update) 10dcaro: toolforge_deploy_mr.py: show the commands being run [repos/cloud/toolforge/lima-kilo] (move_local_deploy_to_deploy_mr) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/291 [15:55:40] 10Toolforge (Toolforge iteration 25): [builds-builder] Host all the build-time dependencies on gitlab - https://phabricator.wikimedia.org/T408140#11328090 (10dcaro) 05Open→03Resolved [16:08:51] (03merge) 10dcaro: logs: use logs-api for logs [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/121 [16:10:11] (03open) 10dcaro: d/changelog: bump to 16.1.24 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/136 (https://phabricator.wikimedia.org/T127367) [16:17:43] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [16:26:10] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [16:26:48] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [16:28:02] 06cloud-services-team, 10Toolforge: [toolforge_run_functional_tests] copy/paste format doesn't work - https://phabricator.wikimedia.org/T408767#11328342 (10dcaro) [16:28:17] 06cloud-services-team, 10Toolforge: [toolforge_run_functional_tests] copy/paste format doesn't work - https://phabricator.wikimedia.org/T408767#11328343 (10dcaro) 05Resolved→03Open [16:29:06] 06cloud-services-team, 10Toolforge: [toolforge_run_functional_tests] copy/paste format doesn't work - https://phabricator.wikimedia.org/T408767#11328349 (10dcaro) I think it's just `TEST_TOOL_UID="$test_tool_uid" ` that is needed, but have to double check [16:33:53] (03open) 10dcaro: run_functional_tests.sh: show the exported vars too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1034 (https://phabricator.wikimedia.org/T408767) [16:34:20] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge_run_functional_tests] copy/paste format doesn't work - https://phabricator.wikimedia.org/T408767#11328375 (10dcaro) >>! In T408767#11328343, @dcaro wrote: > I think it's just `TEST_TOOL_UID="$test_tool_uid" ` that is needed, but have to doubl... [16:35:21] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [16:37:21] (03approved) 10dcaro: d/changelog: bump to 16.1.24 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/136 (https://phabricator.wikimedia.org/T127367) [16:37:27] (03merge) 10dcaro: d/changelog: bump to 16.1.24 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/136 (https://phabricator.wikimedia.org/T127367) [16:40:56] (03open) 10dcaro: ldap_users: use statefulset to get foxtrot-ldap pods [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/292 [16:45:25] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25): toolforge logs appears to suffer from intermittent latency - https://phabricator.wikimedia.org/T402736#11328509 (10dcaro) 05In progress→03Resolved I think we can declare this fixed, we'll open a task if anyone finds issues agai... [16:47:13] (03approved) 10taavi: ldap_users: use statefulset to get foxtrot-ldap pods [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/292 (owner: 10dcaro) [16:49:40] (03merge) 10dcaro: ldap_users: use statefulset to get foxtrot-ldap pods [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/292 [16:53:11] (03open) 10dcaro: api-gateway: display logs-api openapi spec too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1035 (https://phabricator.wikimedia.org/T127367) [16:57:05] (03approved) 10taavi: api-gateway: display logs-api openapi spec too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1035 (https://phabricator.wikimedia.org/T127367) (owner: 10dcaro) [16:58:10] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component api-gateway [17:01:46] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway [17:02:22] (03approved) 10dcaro: api-gateway: display logs-api openapi spec too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1035 (https://phabricator.wikimedia.org/T127367) [17:02:28] (03merge) 10dcaro: api-gateway: display logs-api openapi spec too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1035 (https://phabricator.wikimedia.org/T127367) [17:10:20] (03update) 10dcaro: toolforge_deploy: support deploying local code [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/290 [17:11:48] (03update) 10dcaro: toolforge_deploy: support deploying local code [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/290 [17:12:43] (03approved) 10dcaro: toolforge_deploy: support deploying local code [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/290 [17:12:52] (03merge) 10dcaro: toolforge_deploy: support deploying local code [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/290 [17:12:54] (03update) 10dcaro: toolforge_deploy_mr.py: show the commands being run [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/291 [17:13:33] (03open) 10taavi: Add volans to -cloud channels [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/31 [17:13:38] (03update) 10taavi: Add volans to -cloud channels [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/31 [17:13:44] (03update) 10taavi: Add volans to -cloud channels [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/31 [17:14:21] (03approved) 10fnegri: Add volans to -cloud channels [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/31 (owner: 10taavi) [17:17:33] (03merge) 10taavi: Add volans to -cloud channels [toolforge-repos/ircservserv-config] - 10https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config/-/merge_requests/31 [17:18:17] !issync [17:18:18] Syncing #wikimedia-cloud-feed (requested by Majavah) [17:18:19] Set /cs flags #wikimedia-cloud-feed volans +Afiortv [17:20:32] (03update) 10dcaro: toolforge_deploy_mr.py: show the commands being run [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/291 [17:21:40] (03merge) 10dcaro: toolforge_deploy_mr.py: show the commands being run [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/291 [17:23:07] (03update) 10dcaro: run_functional_tests.sh: show the exported vars too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1034 (https://phabricator.wikimedia.org/T408767) [17:28:23] (03update) 10dcaro: run_functional_tests.sh: show the exported vars too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1034 (https://phabricator.wikimedia.org/T408767) [17:30:17] (03update) 10dcaro: run_functional_tests.sh: show the exported vars too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1034 (https://phabricator.wikimedia.org/T408767) [17:32:22] (03update) 10dcaro: run_functional_tests.sh: show the exported vars too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1034 (https://phabricator.wikimedia.org/T408767) [17:33:10] (03update) 10dcaro: run_functional_tests.sh: show the exported vars too [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1034 (https://phabricator.wikimedia.org/T408767) [17:33:57] 06cloud-services-team, 10Striker: Add 3 maintainer to tool lists - https://phabricator.wikimedia.org/T326552#11328824 (10Yug) >>! In T326552#8509845, @bd808 wrote: > The complete list of maintainers is available on each tool's detail screen. For example https://toolsadmin.wikimedia.org/tools/id/admin. I am... [17:41:55] 06cloud-services-team, 10Cloud-VPS: MTU setting in IPv6 VMs causes issues with Docker - https://phabricator.wikimedia.org/T408543#11328875 (10cmooney) I think this also applies to IPv4 on those networks? The problem isn't IPv6 as such, but due to the overhead the VXLAN encapsulation when that is used to tunne... [17:44:39] (03update) 10dcaro: README: update deployment instructions [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/230 [17:45:33] RESOLVED: PuppetConstantChange: Puppet performing a change on every puppet run on cloudservices2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [17:51:33] (03approved) 10dcaro: runtime - configure topologySpreadConstraints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235 (https://phabricator.wikimedia.org/T408707) (owner: 10damian) [17:52:52] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [17:54:40] (03update) 10damian: Apply nodeAffinity when mount=none [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/38 (https://phabricator.wikimedia.org/T408707) [17:56:16] 06cloud-services-team, 10Striker: Add 3 maintainer to tool lists - https://phabricator.wikimedia.org/T326552#11328914 (10bd808) >>! In T326552#11328824, @Yug wrote: > This gives a one-tool-at-a-time previous. I believe we have no bird view at the moment, where we can see 20 tools and quickly notice the wea... [17:57:38] (03update) 10dcaro: jobs-api: bump to 0.0.433-20251030111500-709a8c88 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1031 (https://phabricator.wikimedia.org/T403321) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:00:17] FIRING: JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:05:10] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [18:06:37] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [18:19:13] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [18:20:15] (03approved) 10dcaro: jobs-api: bump to 0.0.433-20251030111500-709a8c88 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1031 (https://phabricator.wikimedia.org/T403321) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:20:20] (03merge) 10dcaro: jobs-api: bump to 0.0.433-20251030111500-709a8c88 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1031 (https://phabricator.wikimedia.org/T403321) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [18:27:05] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [toolforge_run_functional_tests] copy/paste format doesn't work - https://phabricator.wikimedia.org/T408767#11329118 (10DamianZaremba) >>! In T408767#11328343, @dcaro wrote: > I think it's just `TEST_TOOL_UID="$test_tool_uid" ` that is needed, but have... [18:29:37] 06cloud-services-team, 10Toolforge: [toolforge_run_functional_tests] branch doesn't work on forks - https://phabricator.wikimedia.org/T408766#11329143 (10DamianZaremba) So the problem is the repo is cloned under the test user, which uses the upstream repo. Cloning the repo by hand, then running the script fro... [18:32:12] 06cloud-services-team, 10Toolforge: [build service] failure due to transient issue - https://phabricator.wikimedia.org/T401917#11329168 (10DamianZaremba) 05Open→03Resolved a:03DamianZaremba No longer using Github for the tool downloads. Closing for now, the other ticket about Gitlab might change thi... [18:34:55] 06cloud-services-team, 10Toolforge: Loki usage - https://phabricator.wikimedia.org/T401151#11329176 (10DamianZaremba) 05Open→03Resolved a:03DamianZaremba This has been migrated over for a while (https://phabricator.wikimedia.org/T401151#11084273), some logs have been reduced (debug -> info), still so... [18:36:25] 06cloud-services-team, 10Toolforge: Allow limiting exposed port access - https://phabricator.wikimedia.org/T400940#11329182 (10DamianZaremba) Currently handling this in my tools as part of the deployment (network policy / ingress being the only 'unsupported' objects managed (including cross-tool access), every... [18:45:17] RESOLVED: JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:32:31] 06cloud-services-team, 10Striker: Add 3 maintainer to tool lists - https://phabricator.wikimedia.org/T326552#11329327 (10Yug) Hello @bd808 , Yes, all good. I rose an idea and you both brought relevant view points. This task has played its role, we can now close it imo. [20:04:59] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/50 (owner: 10l10n-bot) [20:05:05] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/50 (owner: 10l10n-bot) [21:28:17] FIRING: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:33:51] FIRING: PuppetFailure: Puppet has failed on cloudbackup1001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [21:44:31] (03open) 10chumzine: Feature: Added results pagination [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/160 [21:48:42] 10Tool-paulina: Results pagination - https://phabricator.wikimedia.org/T399974#11329864 (10Chumzine) Hi @Pepe_piton @Nat_WDU @DidiCoronel I just worked on this task and would love a feedback if possible. Here's my [[ ttps://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/160 | merge request ]].... [21:48:51] RESOLVED: PuppetFailure: Puppet has failed on cloudbackup1001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [22:48:17] FIRING: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [22:53:17] RESOLVED: JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:10:48] PROBLEM - Host cloudbackup1001-dev is DOWN: PING CRITICAL - Packet loss = 100% [23:15:47] FIRING: NodeDown: Node cloudbackup1001-dev is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudbackup1001-dev - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [23:15:50] RECOVERY - Host cloudbackup1001-dev is UP: PING OK - Packet loss = 0%, RTA = 0.80 ms [23:17:14] PROBLEM - SSH on cloudbackup1001-dev is CRITICAL: connect to address 10.64.16.58 and port 22: Connection refused https://wikitech.wikimedia.org/wiki/SSH/monitoring [23:56:37] (03update) 10don-vip: Draft: Add NIH BioArt [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/10