[00:06:32] FIRING: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:36:39] (03update) 10raymond-ndibe: get_job_logs: return a message when no logs gotten [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/3 (owner: 10dcaro) [00:36:58] (03approved) 10raymond-ndibe: get_job_logs: return a message when no logs gotten [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/3 (owner: 10dcaro) [00:50:38] (03update) 10raymond-ndibe: inject_buildpacks: use gitlab-hosted cached files [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/76 (https://phabricator.wikimedia.org/T408140) (owner: 10dcaro) [00:50:39] (03approved) 10raymond-ndibe: inject_buildpacks: use gitlab-hosted cached files [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/76 (https://phabricator.wikimedia.org/T408140) (owner: 10dcaro) [01:01:34] 10Tool-paulina: Proof of concept for adding new statements - https://phabricator.wikimedia.org/T406989#11321822 (10Afanyulionel) Thanks for the feedback @Pepe_piton . Well noted, will update that as well. Regards, Lionel [01:08:14] 10VPS-project-Wikistats: Add pcmwikiquote to wikistats - https://phabricator.wikimedia.org/T408356#11321824 (10Dzahn) 05Open→03Resolved ` MariaDB [wikistats]> insert into wikiquotes (prefix, lang, loclang, method) select prefix,lang,loclang,method from wikipedias where prefix="pcm"; .. root@wikistats-b... [01:10:16] 10VPS-project-Wikistats: Add minwikisource to wikistats - https://phabricator.wikimedia.org/T408348#11321835 (10Dzahn) 05Open→03Resolved ` MariaDB [wikistats]> insert into wikisources (prefix, lang, loclang, method) select prefix,lang,loclang,method from wikipedias where prefix="min"; .. /usr/bin/php... [01:51:10] 10Tool-paulina: Create new generic page - https://phabricator.wikimedia.org/T407385#11321870 (10Afanyulionel) >>! In T407385#11304937, @Pepe_piton wrote: > @System625 Perfect! And thanks for detecting and coding a fix for the JavaScript errors! > > @shreya-bhagat @Blegodwin @Atlas0007 @Raboyusuf2024 Excellent!... [01:52:03] FIRING: PuppetFailure: Puppet has failed on cloudservices2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [01:53:05] FIRING: [2x] HostBGPDown: BGP session for cloudservices2004-dev (172.20.5.8) is down - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DHostBGPDown [06:23:27] 06cloud-services-team (FY2025/26-Q1), 10Data-Services, 06Data-Persistence, 06Data-Platform-SRE: Decide how to use the new clouddb hosts (clouddb102[2-5]) - https://phabricator.wikimedia.org/T401295#11322121 (10Marostegui) >>! In T401295#11303153, @fnegri wrote: > Are you leaving s1 alone based on its size,... [07:20:18] 06cloud-services-team, 10Toolforge: Access to grafana.wmcloud.org - https://phabricator.wikimedia.org/T408633#11322137 (10fgiunchedi) p:05Triage→03Medium [07:20:59] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [jobs-api] handle qualified image names - https://phabricator.wikimedia.org/T408574#11322138 (10fgiunchedi) p:05Triage→03Medium [07:35:23] 06cloud-services-team, 10Cloud-VPS: MTU setting in IPv6 VMs causes issues with Docker - https://phabricator.wikimedia.org/T408543#11322146 (10fgiunchedi) I'm +1 on getting docker's puppetization to do the right thing, in other words detect the default route's mtu and set said value as docker's default. Docker... [08:52:38] 10Tool-iw: IW ingress config does not work with nginx-ingress-controller:v1.13.3 - https://phabricator.wikimedia.org/T408570#11322262 (10taavi) Agreed that we should not disable the upstream validation entirely. Out of those two options, I have a slight preference for moving it to HAProxy, just for the sake of... [09:03:08] 10Tool-iw: IW ingress config does not work with nginx-ingress-controller:v1.13.3 - https://phabricator.wikimedia.org/T408570#11322278 (10dcaro) Both options look good to me yep, the difference I see is with a webapp it can stay as an independent tool, so it can be managed by volunteers, otherwise it will become... [09:04:28] 06cloud-services-team, 10Toolforge: Access to grafana.wmcloud.org - https://phabricator.wikimedia.org/T408633#11322284 (10dcaro) >>! In T408633#11321110, @DamianZaremba wrote: > That's helpful, I have a grafana instance on toolforge that I can use for now also. > > I've found prometheus.tookforge.org before (... [09:15:02] 06cloud-services-team (FY2025/26-Q1), 10Data-Services, 06Data-Persistence, 06Data-Platform-SRE: Decide how to use the new clouddb hosts (clouddb102[2-5]) - https://phabricator.wikimedia.org/T401295#11322301 (10fnegri) > s1 is the busiest section in production and may have replication issues if not isolated... [09:16:59] 06cloud-services-team, 10Cloud-VPS: MTU setting in IPv6 VMs causes issues with Docker - https://phabricator.wikimedia.org/T408543#11322318 (10fnegri) cc-ing @cmooney for networking considerations [09:28:15] 06cloud-services-team, 10Cloud-VPS: MTU setting in IPv6 VMs causes issues with Docker - https://phabricator.wikimedia.org/T408543#11322331 (10fnegri) > Docker actually used to auto-detect default route MTU, though that was removed in https://github.com/moby/moby/pull/18108 It's interesting that in that issue... [09:49:14] 06cloud-services-team, 10Toolforge: Upgrade Calico to 3.29 - https://phabricator.wikimedia.org/T408669 (10taavi) 03NEW [09:57:40] 06cloud-services-team (FY2025/26-Q1), 10Data-Services, 06Data-Persistence, 06Data-Platform-SRE: Decide how to use the new clouddb hosts (clouddb102[2-5]) - https://phabricator.wikimedia.org/T401295#11322434 (10Ladsgroup) >>! In T401295#11322301, @fnegri wrote: >> s1 is the busiest section in production and... [10:17:54] (03update) 10taavi: Add scripts to update Calico manifests [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/17 [10:17:54] (03open) 10taavi: Add scripts to update Calico manifests [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/17 [10:17:54] (03update) 10taavi: Update manifests to match currently deployed version [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/18 [10:17:55] (03update) 10taavi: Update Calico to 3.29.6 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/19 (https://phabricator.wikimedia.org/T408669) [10:17:56] (03open) 10taavi: Update manifests to match currently deployed version [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/18 [10:18:00] (03open) 10taavi: Update Calico to 3.29.6 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/19 (https://phabricator.wikimedia.org/T408669) [10:18:04] (03update) 10taavi: Add scripts to update Calico manifests [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/17 [10:18:07] (03update) 10taavi: Update manifests to match currently deployed version [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/18 [10:18:10] (03update) 10taavi: Update Calico to 3.29.6 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/19 (https://phabricator.wikimedia.org/T408669) [10:23:10] (03update) 10taavi: Update manifests to match currently deployed version [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/18 [10:23:11] (03update) 10taavi: Update Calico to 3.29.6 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/19 (https://phabricator.wikimedia.org/T408669) [10:23:11] (03update) 10taavi: Add scripts to update Calico manifests [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/17 [10:24:06] (03update) 10taavi: Update manifests to match currently deployed version [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/18 [10:24:10] (03update) 10taavi: Add scripts to update Calico manifests [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/17 [10:24:15] (03update) 10taavi: Update Calico to 3.29.6 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/19 (https://phabricator.wikimedia.org/T408669) [10:38:18] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18904984410 (https://github.com/cluebotng/component-configs/commits/6011377571043753ed9ceb9ca00a17bae6883d2a) [10:38:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [10:47:04] (03PS1) 10Majavah: toolforge: k8s: Add cookbook to mirror Calico images [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1199738 [10:48:33] (03update) 10damian: update deployment via template hash change [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/220 (https://phabricator.wikimedia.org/T403321) [11:00:33] (03approved) 10dcaro: Add scripts to update Calico manifests [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/17 (owner: 10taavi) [11:07:00] (03update) 10taavi: Update manifests to match currently deployed version [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/18 [11:07:01] (03update) 10taavi: Update Calico to 3.29.6 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/19 (https://phabricator.wikimedia.org/T408669) [11:07:02] (03update) 10taavi: Add scripts to update Calico manifests [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/17 [11:07:15] (03update) 10taavi: Add scripts to update Calico manifests [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/17 [11:07:30] 10VPS-project-Codesearch, 06collaboration-services, 07Sustainability (Incident Followup): Codesearch: Ensure logrotate for /var/log/account/pacct - https://phabricator.wikimedia.org/T408234#11322699 (10Ladsgroup) FWIW, this was source another outage of codesearch too and I had to truncate the logs to bring i... [11:09:29] (03update) 10taavi: Update Calico to 3.29.6 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/19 (https://phabricator.wikimedia.org/T408669) [11:09:29] (03update) 10taavi: Update manifests to match currently deployed version [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/18 [11:09:29] (03update) 10taavi: Add scripts to update Calico manifests [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/17 [11:27:11] (03PS1) 10Majavah: perl540: Install libnet-idn-encode-perl [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1199750 (https://phabricator.wikimedia.org/T407707) [11:27:39] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [pre-built images] Install libnet-idn-encode-perl in the perl\* pre-built images - https://phabricator.wikimedia.org/T407707#11322733 (10taavi) a:03taavi [11:40:18] (03approved) 10dcaro: Update manifests to match currently deployed version [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/18 (owner: 10taavi) [11:40:54] (03merge) 10taavi: Add scripts to update Calico manifests [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/17 [11:40:57] (03update) 10taavi: Update manifests to match currently deployed version [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/18 [11:42:22] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: calico: bump to 0.0.17-20251029114108-636fbfa4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1019 [11:42:37] (03merge) 10taavi: Update manifests to match currently deployed version [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/18 [11:42:40] (03update) 10taavi: Update Calico to 3.29.6 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/19 (https://phabricator.wikimedia.org/T408669) [11:43:48] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: calico: bump to 0.0.18-20251029114245-20cc47fb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1019 [11:43:51] (03update) 10taavi: Update Calico to 3.29.6 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/19 (https://phabricator.wikimedia.org/T408669) [11:43:51] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: calico: bump to 0.0.18-20251029114245-20cc47fb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1019 [11:44:44] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico [11:49:30] (03CR) 10David Caro: [C:03+1] "Small comment, the rest LGTM, have you run it yet?" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1199738 (owner: 10Majavah) [11:50:21] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico [11:53:42] (03CR) 10Volans: toolforge: k8s: Add cookbook to mirror Calico images (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1199738 (owner: 10Majavah) [11:58:03] (03CR) 10David Caro: [C:03+1] toolforge: k8s: Add cookbook to mirror Calico images (031 comment) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1199738 (owner: 10Majavah) [12:01:28] 06cloud-services-team, 10Toolforge: Command given to run failing functional tests manually does not function - https://phabricator.wikimedia.org/T408679 (10taavi) 03NEW [12:04:42] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [12:15:09] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [12:16:56] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component calico [12:20:35] 06cloud-services-team (FY2025/26-Q1), 10Data-Services, 06Data-Persistence, 06Data-Platform-SRE: Decide how to use the new clouddb hosts (clouddb102[2-5]) - https://phabricator.wikimedia.org/T401295#11322885 (10Marostegui) >>! In T401295#11322301, @fnegri wrote: >> s1 is the busiest section in production an... [12:22:12] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico [12:29:45] (03PS1) 10Majavah: toolforge: deploy: Show output of deploy command [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1199760 [12:30:34] (03CR) 10David Caro: [C:03+1] toolforge: deploy: Show output of deploy command [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1199760 (owner: 10Majavah) [12:32:17] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component calico [12:34:38] 06cloud-services-team, 10Toolforge: [toolforge-deploy] Command given to run failing functional tests manually does not function out of the box for some tests - https://phabricator.wikimedia.org/T408679#11322921 (10dcaro) [12:34:50] (03CR) 10Eugene233: [C:03+2] Test and fix all page designs and styles [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1199335 (owner: 10Wandji collins) [12:35:35] (03Merged) 10jenkins-bot: Test and fix all page designs and styles [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1199335 (owner: 10Wandji collins) [12:37:32] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico [12:40:01] (03merge) 10taavi: calico: bump to 0.0.18-20251029114245-20cc47fb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1019 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [12:41:02] (03open) 10taavi: calico: Increment Helm timeout [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1020 [12:41:07] (03update) 10taavi: calico: Increment Helm timeout [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1020 [12:41:15] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component calico [12:41:57] (03approved) 10dcaro: calico: Increment Helm timeout [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1020 (owner: 10taavi) [12:42:06] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico [12:42:54] (03merge) 10taavi: calico: Increment Helm timeout [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1020 [12:43:04] (03update) 10taavi: Use docker-registry.svc.toolforge.org [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/20 (https://phabricator.wikimedia.org/T394902) [12:43:05] (03CR) 10Majavah: [C:03+2] toolforge: deploy: Show output of deploy command [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1199760 (owner: 10Majavah) [12:43:05] (03update) 10taavi: Update Calico to 3.29.6 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/19 (https://phabricator.wikimedia.org/T408669) [12:43:06] (03open) 10taavi: Use docker-registry.svc.toolforge.org [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/20 (https://phabricator.wikimedia.org/T394902) [12:43:14] (03update) 10taavi: Update Calico to 3.29.6 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/19 (https://phabricator.wikimedia.org/T408669) [12:43:14] (03update) 10taavi: Use docker-registry.svc.toolforge.org [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/20 (https://phabricator.wikimedia.org/T394902) [12:44:32] (03PS2) 10Majavah: toolforge: k8s: Add cookbook to mirror Calico images [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1199738 [12:44:49] (03update) 10taavi: Use docker-registry.svc.toolforge.org [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/20 (https://phabricator.wikimedia.org/T394902) [12:46:09] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.calico.copy_images_to_registry for Calico v3.29.6 [12:46:09] !log taavi@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/calico/cni:v3.29.6 [12:46:45] !log taavi@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/calico/ctl:v3.29.6 [12:46:58] !log taavi@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/calico/kube-controllers:v3.29.6 [12:47:16] !log taavi@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/calico/node:v3.29.6 [12:47:16] (03Merged) 10jenkins-bot: toolforge: deploy: Show output of deploy command [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1199760 (owner: 10Majavah) [12:48:01] (03CR) 10Majavah: "yes, just tested and works fine." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1199738 (owner: 10Majavah) [12:48:11] !log taavi@cloudcumin1001 tools Updating container image docker-registry.svc.toolforge.org/calico/typha:v3.29.6 [12:48:24] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.calico.copy_images_to_registry (exit_code=0) for Calico v3.29.6 [12:57:03] (03approved) 10dcaro: Use docker-registry.svc.toolforge.org [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/20 (https://phabricator.wikimedia.org/T394902) (owner: 10taavi) [13:24:29] (03merge) 10taavi: Use docker-registry.svc.toolforge.org [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/20 (https://phabricator.wikimedia.org/T394902) [13:24:30] (03update) 10taavi: Update Calico to 3.29.6 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/19 (https://phabricator.wikimedia.org/T408669) [13:25:53] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: calico: bump to 0.0.19-20251029132436-f7c8ac19 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1021 (https://phabricator.wikimedia.org/T394902) [13:27:10] (03merge) 10taavi: Update Calico to 3.29.6 [repos/cloud/toolforge/calico] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/19 (https://phabricator.wikimedia.org/T408669) [13:28:28] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: calico: bump to 0.0.20-20251029132720-eeb72cb1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1021 (https://phabricator.wikimedia.org/T394902 https://phabricator.wikimedia.org/T408669) [13:28:30] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: calico: bump to 0.0.20-20251029132720-eeb72cb1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1021 (https://phabricator.wikimedia.org/T394902 https://phabricator.wikimedia.org/T408669) [13:28:54] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component calico (T408669) [13:29:01] T408669: Upgrade Calico to 3.29 - https://phabricator.wikimedia.org/T408669 [13:31:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-test-k8s-control-10 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:35:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tools-checker-5 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:35:29] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico (T408669) [13:35:35] T408669: Upgrade Calico to 3.29 - https://phabricator.wikimedia.org/T408669 [13:36:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-bastion-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:45:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance tools-bastion-15 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:46:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-bastion-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:50:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance tools-bastion-15 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [13:51:28] RESOLVED: [3x] PuppetAgentNoResources: No Puppet resources found on instance toolsbeta-bastion-6 on project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:00:28] RESOLVED: [2x] PuppetAgentNoResources: No Puppet resources found on instance tools-bastion-15 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [14:05:14] 06cloud-services-team, 06Data-Persistence: Set up replication on new hosts clouddb102[2-5] - https://phabricator.wikimedia.org/T408692#11323200 (10fnegri) [14:05:55] 06cloud-services-team (FY2025/26-Q1), 10Data-Services, 06Data-Persistence, 06Data-Platform-SRE: Decide how to use the new clouddb hosts (clouddb102[2-5]) - https://phabricator.wikimedia.org/T401295#11323204 (10fnegri) 05In progress→03Resolved > in wmcs it's constantly getting a lot of reads Some s... [14:06:07] 06cloud-services-team, 06Data-Persistence: Set up replication on new hosts clouddb102[2-5] - https://phabricator.wikimedia.org/T408692#11323212 (10fnegri) [14:06:13] 06cloud-services-team (FY2025/26-Q1), 10Data-Services, 06Data-Persistence, 06Data-Platform-SRE: Decide how to use the new clouddb hosts (clouddb102[2-5]) - https://phabricator.wikimedia.org/T401295#11323211 (10fnegri) [14:09:26] 06cloud-services-team, 10Data-Services, 06Data-Persistence: Set up replication on new hosts clouddb102[2-5] - https://phabricator.wikimedia.org/T408692#11323223 (10taavi) [14:15:45] (03open) 10raymond-ndibe: [jobs.py, test_jobs.py] replace gitlab api with git in get_tags_in_use_by_toolforge_deploy [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/56 (https://phabricator.wikimedia.org/T407496) [14:16:35] 06cloud-services-team (FY2025/26-Q1), 10Data-Services, 06Data-Persistence, 06Data-Platform-SRE: Decide how to use the new clouddb hosts (clouddb102[2-5]) - https://phabricator.wikimedia.org/T401295#11323260 (10Marostegui) >>! In T401295#11323204, @fnegri wrote: >> in wmcs it's constantly getting a lot... [14:24:58] (03approved) 10raymond-ndibe: [jobs.py, test_jobs.py] replace gitlab api with git in get_tags_in_use_by_toolforge_deploy [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/56 (https://phabricator.wikimedia.org/T407496) [14:25:04] (03merge) 10raymond-ndibe: [jobs.py, test_jobs.py] replace gitlab api with git in get_tags_in_use_by_toolforge_deploy [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/56 (https://phabricator.wikimedia.org/T407496) [14:27:26] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-harbor: bump to 0.0.63-20251029142515-eba80de2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1012 (https://phabricator.wikimedia.org/T407496) [14:27:30] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: maintain-harbor: bump to 0.0.63-20251029142515-eba80de2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1012 (https://phabricator.wikimedia.org/T407496) [14:28:58] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [14:29:56] 06cloud-services-team, 10VPS-Projects: wikidumpparse NFS volume filling up - https://phabricator.wikimedia.org/T402807#11323303 (10Andrew) p:05Triage→03High [14:31:21] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [14:31:36] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor [14:35:31] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor [14:35:54] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.63-20251029142515-eba80de2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1012 (https://phabricator.wikimedia.org/T407496) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:36:02] (03approved) 10raymond-ndibe: maintain-harbor: bump to 0.0.63-20251029142515-eba80de2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1012 (https://phabricator.wikimedia.org/T407496) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:36:08] (03merge) 10raymond-ndibe: maintain-harbor: bump to 0.0.63-20251029142515-eba80de2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1012 (https://phabricator.wikimedia.org/T407496) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:36:28] (03update) 10raymond-ndibe: maintain-harbor: bump to 0.0.63-20251029142515-eba80de2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1012 (https://phabricator.wikimedia.org/T407496) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [14:36:56] (03update) 10raymond-ndibe: [jobs.py, test_jobs.py] replace gitlab api with git in get_tags_in_use_by_toolforge_deploy [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/56 (https://phabricator.wikimedia.org/T407496) [14:36:57] (03update) 10raymond-ndibe: [jobs.py, test_jobs.py] replace gitlab api with git in get_tags_in_use_by_toolforge_deploy [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/56 (https://phabricator.wikimedia.org/T407496) [14:42:09] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): filelog in buildservice image is no longer relative to $TOOL_DATA_DIR on newly deployed tools - https://phabricator.wikimedia.org/T408583#11323377 (10dcaro) [14:42:25] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): filelog in buildservice image is no longer relative to $TOOL_DATA_DIR on newly deployed tools - https://phabricator.wikimedia.org/T408583#11323379 (10dcaro) 05In progress→03Resolved [14:42:40] (03open) 10dcaro: functional-tests: add mount tests to jobs-api and builds-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1022 (https://phabricator.wikimedia.org/T408583) [14:47:32] (03update) 10dcaro: functional-tests: add mount tests to jobs-api and builds-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1022 (https://phabricator.wikimedia.org/T408583) [14:47:50] 06cloud-services-team (FY2025/26-Q1), 10Data-Services, 06Data-Persistence, 06Data-Platform-SRE: Decide how to use the new clouddb hosts (clouddb102[2-5]) - https://phabricator.wikimedia.org/T401295#11323391 (10fnegri) > And just a quick look right now on the enwiki one Sorry, I did not explain myself... [14:53:43] (03update) 10dcaro: functional-tests: add mount tests to jobs-api and builds-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1022 (https://phabricator.wikimedia.org/T408583) [14:58:36] (03approved) 10dcaro: inject_buildpacks: use gitlab-hosted cached files [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/76 (https://phabricator.wikimedia.org/T408140) [14:58:41] (03merge) 10dcaro: inject_buildpacks: use gitlab-hosted cached files [repos/cloud/toolforge/builds-builder] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/76 (https://phabricator.wikimedia.org/T408140) [15:00:10] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-builder: bump to 0.0.136-20251029145852-cb090eff [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1023 (https://phabricator.wikimedia.org/T408140) [15:00:18] 06cloud-services-team, 10Huggle: huggle-nfs volume filling up - https://phabricator.wikimedia.org/T402806#11323451 (10Andrew) p:05Triage→03High [15:00:46] 06cloud-services-team, 10Cloud-VPS: MTU setting in IPv6 VMs causes issues with Docker - https://phabricator.wikimedia.org/T408543#11323453 (10taavi) A major downside of doing it via Puppet is that that won't have any effect on instances with non-Puppetized services (so basically anything not managed by WMF SREs). [15:04:12] (03update) 10dcaro: get_job_logs: return a message when no logs gotten [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/3 [15:04:34] (03merge) 10dcaro: get_job_logs: return a message when no logs gotten [repos/cloud/toolforge/logs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/3 [15:05:46] 06cloud-services-team, 10Cloud-VPS: MTU setting in IPv6 VMs causes issues with Docker - https://phabricator.wikimedia.org/T408543#11323503 (10fnegri) p:05Triage→03Medium [15:07:00] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: logs-api: bump to 0.0.4-20251029150446-a8982601 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1024 [15:09:11] 06cloud-services-team, 10Toolforge: [toolforge-deploy] Command given to run failing functional tests manually does not function out of the box for some tests - https://phabricator.wikimedia.org/T408679#11323566 (10fnegri) p:05Triage→03Low [15:15:29] 06cloud-services-team, 10Data-Services, 06Data-Persistence: Set up replication on new hosts clouddb102[2-5] - https://phabricator.wikimedia.org/T408692#11323603 (10fnegri) [15:15:37] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 06Data-Persistence, and 3 others: Set up x1 replication to Wiki Replicas - https://phabricator.wikimedia.org/T395881#11323602 (10fnegri) [15:19:23] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18912872623 (https://github.com/cluebotng/component-configs/commits/3281794d8d1d2e17d9e9859c6f6f7ae3c5216eda) [15:19:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [15:19:51] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/18912872633 (https://github.com/cluebotng/component-configs/commits/3281794d8d1d2e17d9e9859c6f6f7ae3c5216eda) [15:19:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [15:20:16] 06cloud-services-team, 10Toolforge: [jobs-api] apply topology constraints - https://phabricator.wikimedia.org/T408707#11323634 (10DamianZaremba) [15:45:17] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [15:49:55] (03CR) 10Majavah: [C:03+2] toolforge: k8s: Add cookbook to mirror Calico images [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1199738 (owner: 10Majavah) [15:54:01] (03Merged) 10jenkins-bot: toolforge: k8s: Add cookbook to mirror Calico images [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1199738 (owner: 10Majavah) [15:56:09] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [15:56:42] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component calico (T408669) [15:56:45] T408669: Upgrade Calico to 3.29 - https://phabricator.wikimedia.org/T408669 [16:19:13] (03update) 10dcaro: functional-tests: add mount tests to jobs-api and builds-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1022 (https://phabricator.wikimedia.org/T408583) [16:26:58] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico (T408669) [16:27:07] T408669: Upgrade Calico to 3.29 - https://phabricator.wikimedia.org/T408669 [16:29:14] (03update) 10dcaro: functional-tests: add mount tests to jobs-api and builds-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1022 (https://phabricator.wikimedia.org/T408583) [16:30:34] (03open) 10taavi: calico: Increment Helm timeout, take II [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1025 [16:30:46] (03update) 10taavi: calico: Increment Helm timeout, take II [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1025 [16:31:07] (03approved) 10dcaro: calico: Increment Helm timeout, take II [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1025 (owner: 10taavi) [16:31:26] (03merge) 10taavi: calico: Increment Helm timeout, take II [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1025 [16:31:45] (03update) 10taavi: calico: bump to 0.0.20-20251029132720-eeb72cb1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1021 (https://phabricator.wikimedia.org/T394902 https://phabricator.wikimedia.org/T408669) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:31:52] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component calico (T408669) [16:40:17] (03update) 10taavi: calico: Update kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1026 (https://phabricator.wikimedia.org/T408669) [16:40:17] (03open) 10taavi: calico: Update kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1026 (https://phabricator.wikimedia.org/T408669) [16:41:20] (03merge) 10taavi: calico: Update kubeVersion [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1026 (https://phabricator.wikimedia.org/T408669) [16:42:29] (03update) 10dcaro: functional-tests: add mount tests to jobs-api and builds-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1022 (https://phabricator.wikimedia.org/T408583) [16:56:09] (03update) 10dcaro: functional-tests: add mount tests to jobs-api and builds-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1022 (https://phabricator.wikimedia.org/T408583) [16:57:03] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico (T408669) [16:57:09] T408669: Upgrade Calico to 3.29 - https://phabricator.wikimedia.org/T408669 [16:58:12] (03update) 10dcaro: functional-tests: add mount tests to jobs-api and builds-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1022 (https://phabricator.wikimedia.org/T408583) [17:01:10] (03update) 10taavi: calico: bump to 0.0.20-20251029132720-eeb72cb1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1021 (https://phabricator.wikimedia.org/T394902 https://phabricator.wikimedia.org/T408669) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:01:24] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Upgrade Calico to 3.29 - https://phabricator.wikimedia.org/T408669#11324278 (10taavi) 05Open→03Resolved [17:01:44] (03merge) 10taavi: calico: bump to 0.0.20-20251029132720-eeb72cb1 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1021 (https://phabricator.wikimedia.org/T394902 https://phabricator.wikimedia.org/T408669) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:23:01] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [builds-builder] Host all the build-time dependencies on gitlab - https://phabricator.wikimedia.org/T408140#11324372 (10dcaro) [17:23:09] 10Toolforge (Toolforge iteration 25): [jobs-api] Create storage layer, and save business models in persistent storage - https://phabricator.wikimedia.org/T359650#11324374 (10dcaro) [17:23:16] 10Toolforge (Toolforge iteration 25): [maintain-harbor] Failing to cleanup stale artifacts - https://phabricator.wikimedia.org/T407496#11324376 (10dcaro) [17:23:29] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [builds-api, maintain-harbor] fix build/image cleanup - https://phabricator.wikimedia.org/T404157#11324378 (10dcaro) [17:23:42] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [jobs-api] make job status an enum, with clearly defined states - https://phabricator.wikimedia.org/T401172#11324380 (10dcaro) [17:23:45] (03approved) 10dcaro: functional-tests: add mount tests to jobs-api and builds-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1022 (https://phabricator.wikimedia.org/T408583) [17:23:55] (03update) 10dcaro: functional-tests: add mount tests to jobs-api and builds-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1022 (https://phabricator.wikimedia.org/T408583) [17:23:59] 06cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [harbor,infra] Find a way to manage toolforge project policies with code - https://phabricator.wikimedia.org/T360509#11324382 (10dcaro) [17:24:14] 10Toolforge (Toolforge iteration 25), 07Upstream: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#11324385 (10dcaro) [17:24:28] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [jobs-api] check for diff in services when running diff_with_running_job - https://phabricator.wikimedia.org/T392717#11324387 (10dcaro) [17:24:39] 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [components-api] Queue builds when the build queue is full - https://phabricator.wikimedia.org/T402568#11324389 (10dcaro) [17:24:51] 10Toolforge (Toolforge iteration 25): [toolforge] simplify calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377#11324391 (10dcaro) [17:24:59] 10Toolforge (Toolforge iteration 25), 07Upstream: [builds-builder,jobs-api,upstream] Calling nontrivial Procfile commands with arguments results in confusing error (“no such file or directory”) - https://phabricator.wikimedia.org/T356016#11324393 (10dcaro) [17:25:09] 06cloud-services-team, 10Toolforge (Toolforge iteration 25), 07Epic: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#11324395 (10dcaro) [17:33:06] (03CR) 10BryanDavis: [C:03+1] perl540: Install libnet-idn-encode-perl [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1199750 (https://phabricator.wikimedia.org/T407707) (owner: 10Majavah) [17:35:05] (03open) 10damian: Draft: runtime - configure topologySpreadConstraints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235 (https://phabricator.wikimedia.org/T408707) [17:40:26] (03update) 10dcaro: functional-tests: add mount tests to jobs-api and builds-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1022 (https://phabricator.wikimedia.org/T408583) [17:45:42] (03open) 10damian: Draft: Apply nodeAffinity when mount=none [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/38 [17:46:00] (03update) 10damian: Draft: runtime - configure topologySpreadConstraints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235 (https://phabricator.wikimedia.org/T408707) [17:47:40] (03update) 10damian: Draft: Apply nodeAffinity when mount=none [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/38 (https://phabricator.wikimedia.org/T408707) [17:52:39] (03update) 10dcaro: functional-tests: add mount tests to jobs-api and builds-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1022 (https://phabricator.wikimedia.org/T408583) [17:53:15] (03update) 10damian: Draft: Apply nodeAffinity when mount=none [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/38 (https://phabricator.wikimedia.org/T408707) [17:53:30] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.run_tests [17:54:41] 06cloud-services-team, 10Toolforge: Policy for (external) contributions / code style - https://phabricator.wikimedia.org/T408331#11324532 (10dcaro) Also related {T408722} [18:04:38] (03update) 10damian: Draft: Apply nodeAffinity when mount=none [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/38 (https://phabricator.wikimedia.org/T408707) [18:05:13] (03update) 10damian: Draft: Apply nodeAffinity when mount=none [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/38 (https://phabricator.wikimedia.org/T408707) [18:09:00] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) [18:09:06] (03update) 10damian: runtime - configure topologySpreadConstraints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235 (https://phabricator.wikimedia.org/T408707) [18:09:14] (03update) 10damian: Draft: runtime - configure topologySpreadConstraints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/235 (https://phabricator.wikimedia.org/T408707) [18:09:33] (03update) 10damian: Apply nodeAffinity when mount=none [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/38 (https://phabricator.wikimedia.org/T408707) [18:09:59] (03merge) 10dcaro: functional-tests: add mount tests to jobs-api and builds-api [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1022 (https://phabricator.wikimedia.org/T408583) [18:37:31] (03CR) 10Majavah: [C:03+2] perl540: Install libnet-idn-encode-perl [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1199750 (https://phabricator.wikimedia.org/T407707) (owner: 10Majavah) [18:38:05] (03Merged) 10jenkins-bot: perl540: Install libnet-idn-encode-perl [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1199750 (https://phabricator.wikimedia.org/T407707) (owner: 10Majavah) [19:18:15] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [pre-built images] Install libnet-idn-encode-perl in the perl\* pre-built images - https://phabricator.wikimedia.org/T407707#11324811 (10taavi) 05Open→03Resolved [20:02:21] (03update) 10damian: Apply nodeAffinity when mount=none [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/38 (https://phabricator.wikimedia.org/T408707) [20:06:16] (03update) 10damian: Apply nodeAffinity when mount=none [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/38 (https://phabricator.wikimedia.org/T408707) [20:32:35] 10Tool-paulina: Proof of concept: add work form, with autocomplete - https://phabricator.wikimedia.org/T407558#11325150 (10Pepe_piton) @Miiswom Good work! The solution you implemented works very well, congrats! I noticed that you implemented the feature using an API proxy, and this gives me the opportunity to ma... [21:14:32] (03open) 10don-vip: Draft: Add NIH BioArt [toolforge-repos/spacemedia] - 10https://gitlab.wikimedia.org/toolforge-repos/spacemedia/-/merge_requests/10 [21:34:27] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [21:35:17] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30029 bytes in 0.228 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [22:05:31] 06cloud-services-team, 10Toolforge: [build service] Python pack is outdated, does not support latest Python 3.14 stable release - https://phabricator.wikimedia.org/T408108#11325521 (10bd808) [22:40:48] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudservices2004-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [23:55:58] (03open) 10damian: requirements - use pyuwsgi [toolforge-repos/sample-static-buildpack-app] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-static-buildpack-app/-/merge_requests/1 [23:58:57] (03update) 10damian: requirements - use pyuwsgi [toolforge-repos/sample-static-buildpack-app] - 10https://gitlab.wikimedia.org/toolforge-repos/sample-static-buildpack-app/-/merge_requests/1 (https://phabricator.wikimedia.org/T408002)