[02:01:21] <wikibugs>	 (03update) 10ttaylor: Draft: Refactored project structure to add Python API to relay events [toolforge-repos/listen-to-wiki-changes] - 10https://gitlab.wikimedia.org/toolforge-repos/listen-to-wiki-changes/-/merge_requests/1
[02:32:45] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19
[02:36:47] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19
[03:07:04] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services
[03:15:53] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services
[03:23:03] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-19 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[08:00:53] <wikibugs>	 (03approved) 10dcaro: jobs-emailer: bump to 0.0.56-20250508170443-1a7a8bea [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/769 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:00:55] <wikibugs>	 (03merge) 10dcaro: jobs-emailer: bump to 0.0.56-20250508170443-1a7a8bea [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/769 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:12:27] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Toolforge bastion sssd/LDAP flakiness (May 2025) - https://phabricator.wikimedia.org/T393732#10810438 (10taavi) >>! In T393732#10809252, @taavi wrote: > * This is the query sssd does to find sudo rules: `'(&(objectClass=sudoRole)(|(&(!(sudoHost=*))(cn=d...
[08:21:33] <wmcs-alerts>	 FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate Puppet CA: project-proxy-puppetmaster-01.project-proxy.eqiad.wmflabs is about to expire in 13d 18h 14m 54s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire
[08:22:56] <wikibugs>	 (03update) 10dcaro: build.start: add use-latest-versions option [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/103 (https://phabricator.wikimedia.org/T380127)
[08:23:02] <wikibugs>	 (03update) 10dcaro: start: Add useLatestBuilder parameter and configs [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/124 (https://phabricator.wikimedia.org/T380127)
[08:26:36] <wikibugs>	 (03update) 10dcaro: start: Add useLatestBuilder parameter and configs [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/124 (https://phabricator.wikimedia.org/T380127)
[08:29:29] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/56 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:29:31] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/56 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:29:50] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/30 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:29:53] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/30 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:30:15] <wikibugs>	 (03update) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/20 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:30:24] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/20 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:30:28] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/20 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:31:30] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/24 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:34:10] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.67-20250512083005-a8783429 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/770
[08:34:12] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: volume-admission: bump to 0.0.67-20250512083005-a8783429 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/770
[08:34:31] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
[08:35:00] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: envvars-api: bump to 0.0.68-20250512082944-cf63e685 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/771
[08:35:01] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: registry-admission: bump to 0.0.61-20250512083144-8a2ebe74 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/772
[08:35:03] <wikibugs>	 (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.60-20250512083042-2bda0462 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/773
[08:35:06] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: registry-admission: bump to 0.0.61-20250512083144-8a2ebe74 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/772
[08:35:09] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.60-20250512083042-2bda0462 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/773
[08:35:15] <wikibugs>	 (03update) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/71 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:39:19] <wikibugs>	 (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/71 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:39:23] <wikibugs>	 (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/71 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:41:47] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.105-20250512083935-1bd485f6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/774
[08:45:41] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review, 07Security: Remove srv-networktests from tools.admin - https://phabricator.wikimedia.org/T393775#10810520 (10taavi) a:03taavi
[08:46:34] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
[08:46:59] <wikibugs>	 06cloud-services-team, 10Toolforge: [build-service] Document which versions of Node.js, PHP, Go, etc. are supported - https://phabricator.wikimedia.org/T393789#10810521 (10taavi)
[08:47:09] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component volume-admission
[08:59:04] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission
[08:59:38] <wikibugs>	 (03approved) 10dcaro: volume-admission: bump to 0.0.67-20250512083005-a8783429 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/770 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:59:40] <wikibugs>	 (03merge) 10dcaro: volume-admission: bump to 0.0.67-20250512083005-a8783429 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/770 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:59:51] <wikibugs>	 (03update) 10dcaro: envvars-api: bump to 0.0.68-20250512082944-cf63e685 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/771 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[08:59:57] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
[09:07:44] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review, 07Security: Remove srv-networktests from tools.admin - https://phabricator.wikimedia.org/T393775#10810613 (10taavi) 05Open→03Resolved The test suite passes now again.
[09:11:24] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
[09:17:14] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component envvars-api
[09:25:29] <wikibugs>	 (03update) 10dcaro: start: Add useLatestBuilder parameter and configs [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/124 (https://phabricator.wikimedia.org/T380127)
[09:27:55] <wikibugs>	 (03approved) 10dcaro: start: Add useLatestBuilder parameter and configs [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/124 (https://phabricator.wikimedia.org/T380127)
[09:29:05] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api
[09:30:06] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services, 06Data-Persistence, 13Patch-For-Review: wikireplicas: maintain-views should not create _p databases - https://phabricator.wikimedia.org/T392105#10810673 (10fnegri) 05In progress→03Resolved
[09:31:51] <wikibugs>	 (03merge) 10dcaro: start: Add useLatestBuilder parameter and configs [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/124 (https://phabricator.wikimedia.org/T380127)
[09:37:02] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: builds-api: bump to 0.0.190-20250512093109-f2cdc829 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/775 (https://phabricator.wikimedia.org/T380127)
[09:39:25] <wikibugs>	 (03approved) 10dcaro: envvars-api: bump to 0.0.68-20250512082944-cf63e685 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/771 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[09:39:28] <wikibugs>	 (03merge) 10dcaro: envvars-api: bump to 0.0.68-20250512082944-cf63e685 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/771 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[09:40:17] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[09:45:20] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
[09:46:23] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[09:57:46] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
[09:57:47] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[10:00:15] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
[10:01:53] <wikibugs>	 06cloud-services-team, 10Toolforge: [build-service] Document which versions of Node.js, PHP, Go, etc. are supported - https://phabricator.wikimedia.org/T393789#10810778 (10aborrero) p:05Triage→03Medium
[10:02:08] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Test (and implement?) Openstack Octavia lbaas - https://phabricator.wikimedia.org/T393783#10810779 (10aborrero) p:05Triage→03Medium
[10:02:42] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Investigate new Magnum drivers - https://phabricator.wikimedia.org/T393782#10810781 (10aborrero) p:05Triage→03Medium
[10:04:40] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: trove: Unable to create user with IPv6 address as host - https://phabricator.wikimedia.org/T393760#10810786 (10aborrero) Is this a tofu code making this call?
[10:04:50] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: trove: Unable to create user with IPv6 address as host - https://phabricator.wikimedia.org/T393760#10810787 (10aborrero) p:05Triage→03Medium
[10:11:57] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: trove: Unable to create user with IPv6 address as host - https://phabricator.wikimedia.org/T393760#10810801 (10aborrero) >>! In T393760#10810786, @aborrero wrote: > Is this a tofu code making this call?  Yes, see: https://gitlab.wikimedia.org/repos/cloud/metricsinfra/tofu-pr...
[10:21:03] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[10:21:05] <wikibugs>	 10Toolforge (Toolforge iteration 19): [functional-tests] maintain-harbor tests are a bit flaky - https://phabricator.wikimedia.org/T393878 (10dcaro) 03NEW
[10:22:45] <wikibugs>	 10Striker: 500 error on toolsadmin after successfully adding a maintainer - https://phabricator.wikimedia.org/T390516#10810823 (10taavi) 05Open→03Resolved >>! In T390516#10694486, @bd808 wrote: > The stacktrace from T390516#10692629 basically says "saving to LDAP failed". Should we call this "transient"...
[10:25:42] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: replication broken on cloudinfra-db04 - https://phabricator.wikimedia.org/T392889#10810829 (10taavi) a:03taavi
[10:32:36] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
[10:32:38] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[10:33:34] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Add replication alerting for cloudinfra-db - https://phabricator.wikimedia.org/T393881 (10taavi) 03NEW
[10:33:51] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: replication broken on cloudinfra-db04 - https://phabricator.wikimedia.org/T392889#10810875 (10taavi) 05Open→03Resolved Re-imported the dump to get it replicating again, and filed T393881 as a follow-up.
[10:39:28] <wmcs-alerts>	 FIRING: [2x] TargetDown: Job frontproxy-nginx is unreachable in project toolsbeta instance toolsbeta-proxy-5   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[10:44:05] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
[10:44:28] <wmcs-alerts>	 RESOLVED: [2x] TargetDown: Job frontproxy-nginx is unreachable in project toolsbeta instance toolsbeta-proxy-5   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown
[10:47:39] <wikibugs>	 06cloud-services-team, 10Toolforge, 07IPv6: Rebuild Toolforge Prometheus nodes in v6-dualstack network - https://phabricator.wikimedia.org/T393697#10810891 (10aborrero) p:05Triage→03Medium
[10:47:45] <wikibugs>	 06cloud-services-team: HighIOWaitStalling High iowait detected on clouddumps1002:9100. - https://phabricator.wikimedia.org/T393533#10810892 (10aborrero) 05Open→03Resolved a:03aborrero
[10:48:08] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Toolforge bastion sssd/LDAP flakiness (May 2025) - https://phabricator.wikimedia.org/T393732#10810894 (10aborrero) p:05Triage→03High
[10:49:26] <wikibugs>	 06cloud-services-team, 10Toolforge: mbh can't login to Toolforge - https://phabricator.wikimedia.org/T389704#10810896 (10aborrero) 05Open→03Resolved Please open another ticket in case of additional problems.
[10:49:33] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: tf-infra-test misbehavior in codfw1dev - https://phabricator.wikimedia.org/T391718#10810898 (10aborrero) p:05Triage→03Medium
[10:49:53] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: wmcs-cookbooks: write a cookbook to delete an openstack project - https://phabricator.wikimedia.org/T391836#10810899 (10aborrero) p:05Triage→03Low
[10:50:02] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Deleting a project does not release floating IPs for that project - https://phabricator.wikimedia.org/T392680#10810900 (10aborrero) p:05Triage→03Low
[10:50:21] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07Documentation: Consolidate and deduplicate docs about generating SSH keys - https://phabricator.wikimedia.org/T391989#10810901 (10aborrero) p:05Triage→03Medium
[10:51:06] <wikibugs>	 06cloud-services-team, 10Toolforge: Toolforge OpenTofu support - https://phabricator.wikimedia.org/T329425#10810903 (10aborrero) p:05Triage→03Low
[10:51:22] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Cloud VPS mail servers should drop mail sent from non-supported domains - https://phabricator.wikimedia.org/T366935#10810904 (10aborrero) p:05Triage→03Medium
[10:51:55] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Reject outbound traffic to port 25 (SMTP) from instances without public IPs - https://phabricator.wikimedia.org/T366936#10810907 (10aborrero) 05Open→03In progress p:05Triage→03Medium
[10:54:19] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Add replication alerting for cloudinfra-db - https://phabricator.wikimedia.org/T393881#10810909 (10aborrero) p:05Triage→03Medium
[10:54:22] <wikibugs>	 06cloud-services-team: PuppetFailure Puppet has failed on cloudrabbit2003-dev:9100 - https://phabricator.wikimedia.org/T393529#10810910 (10aborrero) 05Open→03Resolved a:03aborrero
[10:54:32] <wikibugs>	 06cloud-services-team: PuppetFailure Puppet has failed on cloudrabbit2002-dev:9100 - https://phabricator.wikimedia.org/T393528#10810912 (10aborrero) 05Open→03Resolved a:03aborrero
[10:54:47] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Alert when views are out of sync - https://phabricator.wikimedia.org/T393388#10810916 (10aborrero) p:05Triage→03Medium
[10:54:56] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Add an option to cookbooks to specify which hosts should be targeted - https://phabricator.wikimedia.org/T393387#10810917 (10aborrero) p:05Triage→03Medium
[10:55:19] <wikibugs>	 06cloud-services-team, 10Toolforge: push-to-deploy: optionally log deployments to SAL automatically - https://phabricator.wikimedia.org/T393169#10810918 (10aborrero) p:05Triage→03Medium
[10:55:45] <wikibugs>	 10Cloud Services Proposals, 06cloud-services-team, 10Toolforge: [DRAFT] Decision Request - Initial product approach to integrate Toolforge UI functionality with Toolsadmin - https://phabricator.wikimedia.org/T393010#10810921 (10aborrero) p:05Triage→03Medium
[10:57:00] <wikibugs>	 06cloud-services-team, 10Data-Services: maintain-dbusers: Use cloud-private to talk to NFS servers instead of proxies - https://phabricator.wikimedia.org/T392794#10810928 (10aborrero) p:05Triage→03Medium
[10:57:03] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07IPv6: Enable IPv6 on the Cloud VPS bastion - https://phabricator.wikimedia.org/T392689#10810931 (10aborrero) p:05Triage→03Medium
[10:57:19] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 07IPv6: Enable IPv6 on Cloud VPS infrastructure services - https://phabricator.wikimedia.org/T392688#10810932 (10aborrero) p:05Triage→03Medium
[10:57:27] <wikibugs>	 06cloud-services-team: PuppetFailure Puppet has failed on cloudcontrol1011:9100 - https://phabricator.wikimedia.org/T392603#10810934 (10aborrero) 05Open→03Resolved a:03aborrero
[10:57:57] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: metricsinfra: Alert on SD failures - https://phabricator.wikimedia.org/T392568#10810936 (10aborrero) p:05Triage→03Medium
[10:59:11] <wikibugs>	 (03approved) 10dcaro: builds-api: bump to 0.0.190-20250512093109-f2cdc829 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/775 (https://phabricator.wikimedia.org/T380127) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[10:59:16] <wikibugs>	 (03update) 10dcaro: builds-api: bump to 0.0.190-20250512093109-f2cdc829 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/775 (https://phabricator.wikimedia.org/T380127) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[11:02:42] <wikibugs>	 (03merge) 10dcaro: builds-api: bump to 0.0.190-20250512093109-f2cdc829 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/775 (https://phabricator.wikimedia.org/T380127) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[11:02:58] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
[11:14:30] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
[11:14:31] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission
[11:22:40] <wikibugs>	 (03update) 10dcaro: ingress-admission: bump to 0.0.60-20250512083042-2bda0462 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/773 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[11:26:15] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission
[11:28:15] <wikibugs>	 (03approved) 10dcaro: ingress-admission: bump to 0.0.60-20250512083042-2bda0462 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/773 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[11:28:18] <wikibugs>	 (03merge) 10dcaro: ingress-admission: bump to 0.0.60-20250512083042-2bda0462 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/773 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[11:28:30] <wikibugs>	 (03update) 10dcaro: registry-admission: bump to 0.0.61-20250512083144-8a2ebe74 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/772 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[11:28:32] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
[11:39:18] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
[11:39:19] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component registry-admission
[11:40:08] <wikibugs>	 06cloud-services-team: PuppetFailure Puppet has failed on cloudcumin1001:9100 - https://phabricator.wikimedia.org/T393047#10810993 (10aborrero) 05Open→03Resolved a:03aborrero
[11:40:25] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: metricsinfra: maintain-projects should not crash when a project with alerts is deleted - https://phabricator.wikimedia.org/T392560#10810996 (10aborrero) p:05Triage→03Medium
[11:40:43] <wikibugs>	 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T392547#10810997 (10aborrero) 05Open→03Resolved a:03aborrero
[11:41:00] <wikibugs>	 06cloud-services-team, 10Toolforge, 07IPv6: Enable IPv6 for Toolforge mail server - https://phabricator.wikimedia.org/T392511#10810999 (10aborrero) p:05Triage→03Medium
[11:41:06] <wikibugs>	 06cloud-services-team, 10Toolforge, 07IPv6: Enable IPv6 on the Toolforge bastion - https://phabricator.wikimedia.org/T392510#10811000 (10aborrero) p:05Triage→03Medium
[11:41:15] <wikibugs>	 06cloud-services-team, 10Toolforge, 07IPv6: Enable IPv6 for Toolforge services - https://phabricator.wikimedia.org/T392509#10811001 (10aborrero) p:05Triage→03Medium
[11:41:23] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki rkiwiki - https://phabricator.wikimedia.org/T392502#10811002 (10aborrero) p:05Triage→03Medium
[11:42:05] <wikibugs>	 06cloud-services-team, 10Horizon, 10Striker, 06serviceops, 06SRE: Move cloudweb to Ganeti VMs and repurpose the servers as wikikube nodes - https://phabricator.wikimedia.org/T392478#10811004 (10aborrero) p:05Triage→03Low
[11:42:35] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Data-Engineering (Q4 2025 April 1st - June 30th), 07IPv6, 13Patch-For-Review: Add new WMCS IP ranges to analytics - https://phabricator.wikimedia.org/T392468#10811005 (10aborrero) p:05Triage→03High
[11:42:45] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: KernelErrors Server cloudcephmon1004 logged kernel errors - https://phabricator.wikimedia.org/T392423#10811006 (10aborrero) 05Open→03Resolved a:03aborrero
[11:43:01] <wikibugs>	 06cloud-services-team, 10Toolforge: toolforge: Investigate ingress-nginx replacements - https://phabricator.wikimedia.org/T392356#10811009 (10aborrero) p:05Triage→03Medium
[11:43:18] <wikibugs>	 06cloud-services-team, 10Quarry, 07Documentation: [[wikitech:Portal:Data Services/Admin/Quarry]] documents legacy Quarry setup - https://phabricator.wikimedia.org/T392181#10811022 (10aborrero) p:05Triage→03Medium
[11:51:05] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission
[11:59:19] <wikibugs>	 (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686)
[12:00:55] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge, 05Cloud-Services-Origin-Team, and 3 others: [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features - https://phabricator.wikimedia.org/T393564#10811077 (10dcaro)
[12:01:51] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge, 05Cloud-Services-Origin-Team, and 3 others: [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features - https://phabricator.wikimedia.org/T393564#10811083 (10dcaro)
[12:02:08] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge, 05Cloud-Services-Origin-Team, and 3 others: [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features - https://phabricator.wikimedia.org/T393564#10811086 (10dcaro)
[12:04:28] <wikibugs>	 (03approved) 10dcaro: registry-admission: bump to 0.0.61-20250512083144-8a2ebe74 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/772 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[12:04:30] <wikibugs>	 (03merge) 10dcaro: registry-admission: bump to 0.0.61-20250512083144-8a2ebe74 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/772 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[12:05:10] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api
[12:08:06] <wikibugs>	 06cloud-services-team, 10Toolforge: [gateway-api] something is caching the openapi docs - https://phabricator.wikimedia.org/T371033#10811105 (10dcaro) 05Open→03Resolved a:03dcaro I think this can be closed, I have not seen it happen in a while, we can reopen if it happens again.
[12:08:53] <wikibugs>	 06cloud-services-team, 10Toolforge: dev.toolforge.org unreachable - https://phabricator.wikimedia.org/T389717#10811112 (10taavi) →14Duplicate dup:03T393732
[12:08:55] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Toolforge bastion sssd/LDAP flakiness (May 2025) - https://phabricator.wikimedia.org/T393732#10811114 (10taavi)
[12:09:26] <wikibugs>	 06cloud-services-team, 10Toolforge: tofu-provisioning: factorize gitlab pipeline logic - https://phabricator.wikimedia.org/T393686#10811119 (10aborrero) pushed container image: docker-registry.tools.wmflabs.org/tofu-provisioning:20250512 from Dockerfile: https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/...
[12:14:46] <wikibugs>	 (03update) 10dcaro: components-api: bump to 0.0.105-20250512083935-1bd485f6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/774 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[12:15:29] <wikibugs>	 (03merge) 10dcaro: build.start: add use-latest-versions option [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/103 (https://phabricator.wikimedia.org/T380127)
[12:16:51] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api
[12:21:24] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Observability-Metrics, 10SRE Observability (FY2024/2025-Q4): Remove librenms -> graphite integration, replace with gnmi - https://phabricator.wikimedia.org/T372457#10811193 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi I'm boldly resolving this since AFAICT...
[12:23:22] <wikibugs>	 06cloud-services-team, 10Toolforge: [toolforge-prometheus] upgrade to bookworm - https://phabricator.wikimedia.org/T375523#10811200 (10taavi) a:05dcaro→03taavi stealing this to do it with {T393697}
[12:23:24] <wikibugs>	 06cloud-services-team, 10Toolforge: [toolforge-prometheus] upgrade to bookworm - https://phabricator.wikimedia.org/T375523#10811206 (10taavi)
[12:23:28] <wikibugs>	 06cloud-services-team, 10Toolforge, 07IPv6: Rebuild Toolforge Prometheus nodes in v6-dualstack network - https://phabricator.wikimedia.org/T393697#10811207 (10taavi)
[12:23:30] <wikibugs>	 06cloud-services-team, 10Toolforge, 07IPv6: Rebuild Toolforge Prometheus nodes in v6-dualstack network - https://phabricator.wikimedia.org/T393697#10811208 (10taavi) a:03taavi
[12:24:01] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717)
[12:25:09] <wikibugs>	 (03approved) 10dcaro: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118) (owner: 10raymond-ndibe)
[12:25:50] <wikibugs>	 (03open) 10dcaro: d/changelog: bump to 0.0.20 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/105 (https://phabricator.wikimedia.org/T380127)
[12:27:05] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/commons-mass-description] - 10https://gerrit.wikimedia.org/r/1144540 (owner: 10L10n-bot)
[12:34:42] <wikibugs>	 (03approved) 10dcaro: components-api: bump to 0.0.105-20250512083935-1bd485f6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/774 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[12:34:44] <wikibugs>	 (03merge) 10dcaro: components-api: bump to 0.0.105-20250512083935-1bd485f6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/774 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[12:35:10] <wikibugs>	 10Toolforge (Toolforge iteration 19): [components-api,buildsa-api] When building and deploying, if none of the settings changed, the jobs are not restarted - https://phabricator.wikimedia.org/T389044#10811289 (10dcaro)
[12:35:13] <wikibugs>	 10Toolforge (Toolforge iteration 19): [components-api] allow stopping a deployment that's running - https://phabricator.wikimedia.org/T388644#10811290 (10dcaro)
[12:35:15] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge, 05Cloud-Services-Origin-Team, and 3 others: [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features - https://phabricator.wikimedia.org/T393564#10811288 (10dcaro)
[12:35:17] <wikibugs>	 10Toolforge (Toolforge iteration 19), 13Patch-For-Review: [components-api] Rename the CRDs groups to be `components-api.toolforge.org` - https://phabricator.wikimedia.org/T386829#10811291 (10dcaro)
[12:35:17] <wikibugs>	 06cloud-services-team, 10Toolforge: [components-api] Add webservice support (to refine) - https://phabricator.wikimedia.org/T362077#10811292 (10dcaro)
[12:35:19] <wikibugs>	 06cloud-services-team, 10Toolforge: [components-api] add one-off, scheduled and continuous jobs support to the yaml + api - https://phabricator.wikimedia.org/T362075#10811293 (10dcaro)
[12:35:21] <wikibugs>	 06cloud-services-team, 10Toolforge: [components-api] Extend the list of build triggers (unrefined) - https://phabricator.wikimedia.org/T362071#10811294 (10dcaro)
[12:47:12] <wikibugs>	 06cloud-services-team, 10Toolforge: [components-api] Add source polling build trigger - https://phabricator.wikimedia.org/T362071#10811332 (10dcaro)
[12:53:05] <wikibugs>	 10Tool-campwiz-nxt, 06translatewiki.net, 10LPL Essential (LPL Essential 2025 Apr-Jun: CX), 07Unplanned-Sprint-Work: Add CampWiz NXT to translatewiki.net - https://phabricator.wikimedia.org/T393850#10811378 (10Nikerabbit) p:05Triage→03Medium I see you have put target branch as different from the source...
[12:54:51] <wikibugs>	 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Toolforge bastion sssd/LDAP flakiness (May 2025) - https://phabricator.wikimedia.org/T393732#10811408 (10Fnielsen) I have been able to login to Toolforge and do `become`
[12:54:52] <wikibugs>	 (03open) 10dcaro: builds-api: configure the builder/runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/776 (https://phabricator.wikimedia.org/T380127)
[12:57:31] <wikibugs>	 (03update) 10dcaro: builds-api: configure the builder/runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/776 (https://phabricator.wikimedia.org/T380127)
[12:57:50] <wikibugs>	 (03update) 10dcaro: builds-api: configure the builder/runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/776 (https://phabricator.wikimedia.org/T380127)
[12:59:48] <wikibugs>	 (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686)
[13:00:53] <wikibugs>	 (03update) 10aborrero: gitlab-ci: replace local logic with included one [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/23 (https://phabricator.wikimedia.org/T393686)
[13:03:40] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[13:04:18] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
[13:10:09] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[13:10:49] <wikibugs>	 (03PS1) 10Klausman: Revert "thanos/swift: at pseudo secrets for mint_ro" [labs/private] - 10https://gerrit.wikimedia.org/r/1144569
[13:11:37] <wikibugs>	 (03CR) 10Klausman: [V:03+2 C:03+2] Revert "thanos/swift: at pseudo secrets for mint_ro" [labs/private] - 10https://gerrit.wikimedia.org/r/1144569 (owner: 10Klausman)
[13:14:46] <wikibugs>	 (03update) 10chuckonwumelu: Importing: Floating IPs from Tools & Toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/20 (https://phabricator.wikimedia.org/T390056)
[13:17:28] <wmcs-alerts>	 FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure
[13:17:53] <wikibugs>	 (03open) 10taavi: service: New abstraction [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/24
[13:18:28] <wmcs-alerts>	 FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project gitlab-runners   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure
[13:18:28] <wmcs-alerts>	 FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure
[13:18:37] <wikibugs>	 (03update) 10chuckonwumelu: Importing: Floating IPs from Tools & Toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/20 (https://phabricator.wikimedia.org/T390056)
[13:19:28] <wmcs-alerts>	 FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure
[13:19:28] <wmcs-alerts>	 FIRING: WidespreadPuppetAgentFailure: Widespread puppet agent failures in project toolsbeta   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure
[13:21:35] <wikibugs>	 (03update) 10taavi: service: New abstraction [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/24
[13:21:45] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
[13:22:50] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[13:23:25] <wikibugs>	 (03update) 10taavi: service: New abstraction [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/24
[13:23:50] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api
[13:23:57] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: decommission cloudlb2001-dev.codfw.wmnet - https://phabricator.wikimedia.org/T392686#10811564 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm
[13:24:41] <wikibugs>	 (03update) 10dcaro: builds-api: configure the builder/runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/776 (https://phabricator.wikimedia.org/T380127)
[13:25:08] <wikibugs>	 (03update) 10taavi: service: New abstraction [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/24
[13:26:57] <wikibugs>	 (03update) 10taavi: service: New abstraction [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/24
[13:26:58] <wmcs-alerts>	 RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tools-bastion-13 in project tools   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[13:27:06] <wikibugs>	 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: decommission cloudlb2001-dev.codfw.wmnet - https://phabricator.wikimedia.org/T392686#10811587 (10cmooney) >>! In T392686#10786597, @Andrew wrote: > + @cmooney because I bet he can fix this in 5 seconds  Yeah manually delete...
[13:31:08] <wikibugs>	 (03update) 10taavi: service: New abstraction [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/24
[13:31:25] <wikibugs>	 (03open) 10dcaro: config: add use_latest_versions to the source build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/72 (https://phabricator.wikimedia.org/T380127)
[13:34:14] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api
[13:34:40] <wikibugs>	 (03approved) 10taavi: tools: dns: drop docker-registry.tools.wmcloud.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/21 (owner: 10aborrero)
[13:36:40] <wikibugs>	 (03update) 10taavi: service: New abstraction [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/24
[13:38:03] <wmcs-alerts>	 FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-36 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[13:43:03] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-36 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[13:46:33] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api
[13:48:28] <wikibugs>	 (03update) 10chuckonwumelu: Importing: Floating IPs from Tools & Toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/20 (https://phabricator.wikimedia.org/T390056)
[13:55:00] <wikibugs>	 (03approved) 10dcaro: builds-api: configure the builder/runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/776 (https://phabricator.wikimedia.org/T380127)
[13:55:04] <wikibugs>	 (03merge) 10dcaro: builds-api: configure the builder/runner images [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/776 (https://phabricator.wikimedia.org/T380127)
[13:59:18] <wikibugs>	 (03update) 10chuckonwumelu: Importing: Floating IPs from Tools & Toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/20 (https://phabricator.wikimedia.org/T390056)
[14:01:42] <wikibugs>	 (03update) 10chuckonwumelu: Importing: Floating IPs from Tools & Toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/20 (https://phabricator.wikimedia.org/T390056)
[14:14:21] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki rkiwiki - https://phabricator.wikimedia.org/T392502#10811816 (10taavi)
[14:14:34] <wikibugs>	 06cloud-services-team, 10Data-Services: [wikireplicas] Create views for new wiki rkiwiki - https://phabricator.wikimedia.org/T392502#10811819 (10taavi) 05Open→03Stalled stalled pending T392498
[14:22:03] <wikibugs>	 (03update) 10chuckonwumelu: Importing: Floating IPs from Tools & Toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/20 (https://phabricator.wikimedia.org/T390056)
[14:27:15] <wikibugs>	 (03update) 10chuckonwumelu: Importing: Floating IPs from Tools & Toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/20 (https://phabricator.wikimedia.org/T390056)
[14:29:57] <wikibugs>	 (03approved) 10aborrero: service: New abstraction [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/24 (owner: 10taavi)
[14:32:14] <wikibugs>	 10Striker: Use IDP for authentication in Striker - https://phabricator.wikimedia.org/T359554#10811895 (10joanna_borun)
[14:40:43] <wikibugs>	 (03update) 10taavi: Importing: Floating IPs from Tools & Toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/20 (https://phabricator.wikimedia.org/T390056) (owner: 10chuckonwumelu)
[14:45:30] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge, 05Cloud-Services-Origin-Team, and 3 others: [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features - https://phabricator.wikimedia.org/T393564#10811960 (10dcaro)
[14:56:17] <wikibugs>	 06cloud-services-team, 10Quarry, 07Documentation: [[wikitech:Portal:Data Services/Admin/Quarry]] update quarry docs to reflect the current setup - https://phabricator.wikimedia.org/T392181#10812019 (10dcaro)
[15:00:46] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
[15:12:25] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
[15:26:07] <wikibugs>	 (03approved) 10aborrero: Importing: Floating IPs from Tools & Toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/20 (https://phabricator.wikimedia.org/T390056) (owner: 10chuckonwumelu)
[15:28:03] <wmcs-alerts>	 FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-36 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[15:29:22] <wikibugs>	 (03merge) 10chuckonwumelu: Importing: Floating IPs from Tools & Toolsbeta [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/20 (https://phabricator.wikimedia.org/T390056)
[15:30:21] <wikibugs>	 (03approved) 10chuckonwumelu: tools: dns: drop docker-registry.tools.wmcloud.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/21 (owner: 10aborrero)
[15:32:58] <wikibugs>	 (03update) 10aborrero: tools: dns: drop docker-registry.tools.wmcloud.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/21
[15:33:00] <wikibugs>	 (03approved) 10chuckonwumelu: service: New abstraction [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/24 (owner: 10taavi)
[15:35:07] <wikibugs>	 (03merge) 10aborrero: tools: dns: drop docker-registry.tools.wmcloud.org [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/21
[15:39:50] <wikibugs>	 (03update) 10taavi: service: New abstraction [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/24
[15:41:42] <wikibugs>	 (03merge) 10taavi: service: New abstraction [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/24
[15:47:48] <wikibugs>	 (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686)
[15:48:03] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-36 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[15:48:26] <wikibugs>	 (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686)
[15:48:32] <wikibugs>	 (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686)
[15:53:20] <wikibugs>	 (03update) 10aborrero: gitlab-ci: replace local logic with included one [repos/cloud/cloud-vps/networktests-tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/networktests-tofu-provisioning/-/merge_requests/23 (https://phabricator.wikimedia.org/T393686)
[15:55:52] <wikibugs>	 (03open) 10aborrero: gitlab-ci: replace local logic with included one [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/25 (https://phabricator.wikimedia.org/T393686)
[16:00:29] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Reject outbound traffic to port 25 (SMTP) from instances without public IPs - https://phabricator.wikimedia.org/T366936#10812560 (10cmooney) It's not uncommon to block it yeah.  +1
[16:06:20] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-cli
[16:07:54] <wikibugs>	 (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686)
[16:09:06] <wikibugs>	 (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686)
[16:10:37] <wikibugs>	 (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686)
[16:11:32] <wikibugs>	 (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686)
[16:12:51] <wikibugs>	 (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686)
[16:14:09] <wikibugs>	 (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686)
[16:15:15] <wikibugs>	 (03update) 10aborrero: gitlab-ci: introduce tofu-provisioning code [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/51 (https://phabricator.wikimedia.org/T393686)
[16:17:37] <wikibugs>	 10Tool-gitlab-content: Add maxage/smaxage cache header controls to gilab-content proxy - https://phabricator.wikimedia.org/T393928 (10bd808) 03NEW
[16:17:47] <wikibugs>	 06cloud-services-team, 10Cloud-VPS: Reject outbound traffic to port 25 (SMTP) from instances without public IPs - https://phabricator.wikimedia.org/T366936#10812676 (10taavi) 05In progress→03Resolved
[16:18:10] <logmsgbot_cloud>	 !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli
[16:22:16] <wikibugs>	 (03approved) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/20 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[16:22:19] <wikibugs>	 (03merge) 10dcaro: build: Upgrade pre-commit dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/20 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[16:24:46] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.57-20250512162230-f6958e24 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/777
[16:29:47] <wikibugs>	 06cloud-services-team, 10Toolforge: Toolforge Build Service does not support .python-version - https://phabricator.wikimedia.org/T381923#10812732 (10dcaro) @LucasWerkmeister can you try now with `toolforge build start --use-latest-versions <url>`? That will be using the new runner (ubuntu24) and the newer buil...
[16:39:32] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19): Upgrade python buildpack to v0.17.0 or newer for Poetry support - https://phabricator.wikimedia.org/T374056#10812792 (10dcaro) @bd808 just released a new flag for the cli `toolforge build start --use-latest-versions` that will pull the latest buildpa...
[16:41:08] <wikibugs>	 06cloud-services-team, 10Toolforge (Toolforge iteration 19): [builds-builder] Golang buildpack does not allow using Procfiles so can't use custom scripts/entrypoints - https://phabricator.wikimedia.org/T390845#10812796 (10dcaro) @Nokib_Sarkar Hi! can you try using `toolforge build start --use-latest-versions <...
[16:42:27] <wikibugs>	 06cloud-services-team, 10Toolforge: [build-service] Document which versions of Node.js, PHP, Go, etc. are supported - https://phabricator.wikimedia.org/T393789#10812810 (10dcaro) A good example is the golang buildpack, this is the version we are shipping with the `--use-latest-versions` flag: https://github.co...
[16:49:05] <wikibugs>	 06cloud-services-team, 10Toolforge: [jobs-cli,components-api] Provide YAML schema file for toolforge-jobs definition files - https://phabricator.wikimedia.org/T314729#10812841 (10dcaro)
[16:49:08] <wikibugs>	 10Cloud Services Proposals, 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge, 05Cloud-Services-Origin-Team, and 3 others: [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features - https://phabricator.wikimedia.org/T393564#10812842 (10dcaro)
[16:51:26] <wikibugs>	 (03update) 10ttaylor: Draft: Refactored project structure to add Python API to relay events [toolforge-repos/listen-to-wiki-changes] - 10https://gitlab.wikimedia.org/toolforge-repos/listen-to-wiki-changes/-/merge_requests/1
[16:52:29] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10812890 (10bd808) This is still happening, and I'm currently struggling to figure out how to get verbose errors to explain t...
[17:00:16] <wikibugs>	 (03approved) 10dcaro: d/changelog: bump to 0.0.20 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/105 (https://phabricator.wikimedia.org/T380127)
[17:00:20] <wikibugs>	 (03merge) 10dcaro: d/changelog: bump to 0.0.20 [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/105 (https://phabricator.wikimedia.org/T380127)
[17:02:03] <wmcs-alerts>	 FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-36 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[17:03:07] <wikibugs>	 10Toolforge (Toolforge iteration 19): [builds-api] define a policy to update runtimes - https://phabricator.wikimedia.org/T393937 (10dcaro) 03NEW
[17:07:03] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-36 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[17:07:33] <wmcs-alerts>	 FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-36 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[17:13:07] <wikibugs>	 10Toolforge (Toolforge iteration 19): [functional-tests] maintain-harbor tests are a bit flaky - https://phabricator.wikimedia.org/T393878#10813056 (10dcaro) p:05Triage→03Low
[17:13:14] <wikibugs>	 10Toolforge (Toolforge iteration 19): [builds-api] define a policy to update runtimes - https://phabricator.wikimedia.org/T393937#10813057 (10dcaro) p:05Triage→03Medium
[17:17:18] <wmcs-alerts>	 RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-36 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[17:17:33] <wmcs-alerts>	 FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Node tools-k8s-worker-nfs-36 has at least 12 procs in D state, and may be having NFS/IO issues - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses
[17:39:52] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-deploy] run specific tests on deploy [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/755 (https://phabricator.wikimedia.org/T381011)
[17:44:10] <wikibugs>	 06cloud-services-team, 10Toolforge: Toolforge Build Service does not support .python-version - https://phabricator.wikimedia.org/T381923#10813247 (10LucasWerkmeister) Seems to be working \o/  I feel like the first build took longer to start showing output than usual, but that might just be random noise or even...
[17:52:16] <wikibugs>	 06cloud-services-team, 10Toolforge: Toolforge Build Service does not support .python-version - https://phabricator.wikimedia.org/T381923#10813281 (10dcaro) Nice :), it might be because it's the first time it uses the builder/runner/etc. images, so it has to pull them anew.  Feel free to close the task once you...
[17:53:56] <wikibugs>	 06cloud-services-team, 10Toolforge: Toolforge Build Service does not support .python-version - https://phabricator.wikimedia.org/T381923#10813287 (10LucasWerkmeister) I don’t think I have any more Python build service tools at the moment, so I’ll just go ahead and close this ^^
[17:54:04] <wikibugs>	 06cloud-services-team, 10Toolforge: Toolforge Build Service does not support .python-version - https://phabricator.wikimedia.org/T381923#10813288 (10LucasWerkmeister) 05Stalled→03Resolved a:03dcaro
[18:21:14] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10813400 (10Andrew) The issue is this bit in the post install script:   `         if ! getent passwd ${VAR_UG_PKG_NAME} > /de...
[18:23:36] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10813405 (10Andrew) > I'm currently struggling to figure out how to get verbose errors to explain the error exit status 6 res...
[18:24:48] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-deploy] run specific tests on deploy [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/755 (https://phabricator.wikimedia.org/T381011)
[18:30:38] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-deploy] run specific tests on deploy [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/755 (https://phabricator.wikimedia.org/T381011)
[18:32:41] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-deploy] run specific tests on deploy [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/755 (https://phabricator.wikimedia.org/T381011)
[18:37:13] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118)
[18:40:06] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] use pydantic for all models [repos/cloud/toolforge/jobs-api] (move_most_custom_validations_out_of_api_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T389118)
[18:43:20] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118)
[18:43:37] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] use pydantic for all models [repos/cloud/toolforge/jobs-api] (move_most_custom_validations_out_of_api_models) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T389118)
[18:43:46] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] (use_pydantic_for_core_job_model) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136)
[18:45:34] <wikibugs>	 (03approved) 10raymond-ndibe: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118)
[18:48:15] <wikibugs>	 (03merge) 10raymond-ndibe: [jobs-api] move custom validations out of api models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/150 (https://phabricator.wikimedia.org/T389118)
[18:48:17] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] use pydantic for all models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T389118)
[18:50:42] <wikibugs>	 (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.373-20250512184826-3b202d92 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/778 (https://phabricator.wikimedia.org/T389118)
[18:54:57] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[19:05:48] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
[19:28:27] <wikibugs>	 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Create views for new wiki nupwiki - https://phabricator.wikimedia.org/T390714#10813676 (10Pppery) It looks like the domain name `nupwiki.analytics.db.svc.wikimedia.cloud` wasn't created. Was that step missed, or an I doing something...
[19:35:57] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api
[19:48:10] <logmsgbot_cloud>	 !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api
[19:59:50] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for all services
[20:07:49] <logmsgbot_cloud>	 !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for all services
[20:24:34] <wikibugs>	 (03open) 10addshore: Draft: Components [repos/cloud/toolforge/toolforge-gen-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-gen-cli/-/merge_requests/2
[20:55:27] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10813948 (10bd808) >>! In T361381#10813400, @Andrew wrote: > At first blush, I think that usermod is just wrong -- since when...
[20:56:56] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] use pydantic for all models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T389118)
[21:04:22] <wikibugs>	 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10813975 (10bd808) 05Open→03Resolved I have a feeling that everything maps is still busted in deployment-prep, but Pu...
[21:08:49] <wikibugs>	 10Tool-campwiz-nxt, 06translatewiki.net, 10LPL Essential (LPL Essential 2025 Apr-Jun: CX), 07Unplanned-Sprint-Work: Add CampWiz NXT to translatewiki.net - https://phabricator.wikimedia.org/T393850#10813991 (10Nokib_Sarkar) I am not sure if my required JSON format is compatible with the formats available in...
[21:12:34] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] (use_pydantic_for_core_job_model) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136)
[21:13:59] <wikibugs>	 (03open) 10raymond-ndibe: [jobs-api] refactor quota models [repos/cloud/toolforge/jobs-api] (use_pydantic_for_core_job_model) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/164 (https://phabricator.wikimedia.org/T389118)
[21:22:59] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] use pydantic for all models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T389118)
[21:27:00] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] use pydantic for all models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T389118)
[21:29:59] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] use pydantic for all models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T389118)
[21:35:33] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] (use_pydantic_for_core_job_model) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136)
[21:41:59] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] use pydantic for all models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/139 (https://phabricator.wikimedia.org/T389118)
[21:42:13] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] refactor quota models [repos/cloud/toolforge/jobs-api] (use_pydantic_for_core_job_model) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/164 (https://phabricator.wikimedia.org/T389118)
[21:42:35] <wikibugs>	 (03update) 10raymond-ndibe: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] (use_pydantic_for_core_job_model) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136)
[21:46:23] <wikibugs>	 10Tool-gawa: [Code Contribution] Hébergement de l'outil GAWA sur Tooforge - https://phabricator.wikimedia.org/T393162#10814133 (10paulwiki) 05Open→03Resolved p:05Triage→03High L'outil GAWA a été hébergé sur Toolforge et est disponible à l'adresse suivante : https://gawa-ci.toolforge.org/
[21:49:35] <wikibugs>	 (03update) 10raymond-ndibe: jobs-api: bump to 0.0.373-20250512184826-3b202d92 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/778 (https://phabricator.wikimedia.org/T389118) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[21:49:36] <wikibugs>	 (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.373-20250512184826-3b202d92 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/778 (https://phabricator.wikimedia.org/T389118) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[21:49:42] <wikibugs>	 (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.373-20250512184826-3b202d92 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/778 (https://phabricator.wikimedia.org/T389118) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620)
[21:51:56] <wikibugs>	 (03update) 10raymond-ndibe: [toolforge-deploy] run specific tests on deploy [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/755 (https://phabricator.wikimedia.org/T381011)
[22:15:15] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Beta-Cluster-Infrastructure: Route deployment-prep Prometheus alerts to the betacluster-alerts@lists.wikimedia.org mailing list - https://phabricator.wikimedia.org/T393975 (10bd808) 03NEW
[22:25:22] <wikibugs>	 06cloud-services-team, 10Cloud-VPS, 10Beta-Cluster-Infrastructure: Route deployment-prep Prometheus alerts to the betacluster-alerts@lists.wikimedia.org mailing list - https://phabricator.wikimedia.org/T393975#10814317 (10bd808) I think what is needed here is a row poked into the prometheusconfig database as...