[00:16:35] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/30 [00:16:36] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/30 [00:16:39] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/30 [00:16:58] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/22 [00:16:59] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/22 [00:17:03] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/22 [00:18:07] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/14 [00:18:13] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/14 [00:18:15] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/14 [00:19:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/34 [00:19:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/34 [00:19:15] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/34 [00:19:49] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/15 [00:19:54] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/15 [00:20:06] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/22 [00:20:12] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/22 [00:20:21] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/25 [00:20:24] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/25 [00:20:29] (03open) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/25 [00:21:56] FIRING: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1002. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [01:14:44] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/30 [01:15:15] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/14 [01:15:39] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/34 [01:15:58] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/22 [01:16:02] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/25 [02:16:56] FIRING: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1002 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [06:17:11] FIRING: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1002 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [07:28:21] 10Tools: dimastbkbot tool is making very long db queries and crashing repeatedly - https://phabricator.wikimedia.org/T429027#12017235 (10fnegri) 05Open→03Resolved a:03fnegri @dima_st_bk thanks for fixing this! I can confirm no slow db queries were logged after you restarted the tool. Also, I'm no longe... [07:30:29] 10Tools: dimastbkbot tool is making very long db queries and crashing repeatedly - https://phabricator.wikimedia.org/T429027#12017240 (10fnegri) a:05fnegri→03dima_st_bk [07:36:10] 06cloud-services-team, 10Cloud-VPS: Openstack root disk backups log errors - https://phabricator.wikimedia.org/T428865#12017266 (10Volans) [07:36:12] 06cloud-services-team, 10Cloud-VPS: Openstack cinder volumes backups are broken - https://phabricator.wikimedia.org/T428867#12017267 (10Volans) [07:54:57] 10Toolforge, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#12017322 (10fgiunchedi) 05Stalled→03Declined I'm boldly declining the task, we can reopen as/if needed [07:56:11] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 07Epic: Toolforge on bare metal POC - https://phabricator.wikimedia.org/T407296#12017325 (10fgiunchedi) 05Stalled→03Declined I'm boldly declining the task, we can reopen as/if needed [07:56:24] 06cloud-services-team, 10Toolforge, 06Infrastructure-Foundations, 10netops: Create new VRF and networks for Toolforge-on-Metal - https://phabricator.wikimedia.org/T409309#12017328 (10fgiunchedi) 05Open→03Declined I'm boldly declining the task, we can reopen as/if needed [08:16:56] RESOLVED: SystemdUnitDown: The service unit kiwix-mirror-update.service is in failed status on host clouddumps1002. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:16:56] RESOLVED: SystemdUnitDown: The systemd unit kiwix-mirror-update.service on node clouddumps1002 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=clouddumps1002 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [08:44:26] 10Toolforge, 06tools-platform-team: Regression: Images built with the build service images do no longer contain/configure locales specified in .locales - https://phabricator.wikimedia.org/T428230#12017603 (10dcaro) Related previous work on this: {T362680} [09:13:08] (03open) 10filippo: Remove checker dns and floating IP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/105 [09:47:01] (03approved) 10taavi: Remove checker dns and floating IP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/105 (owner: 10filippo) [09:47:38] (03merge) 10filippo: Remove checker dns and floating IP [repos/cloud/toolforge/tofu-provisioning] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/105 [09:54:26] 06cloud-services-team, 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [builds-api] expose supported versions - https://phabricator.wikimedia.org/T422046#12017833 (10dcaro) Fyi. this info is https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/blob/main/components/builds-api/values... [09:54:57] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 13Patch-For-Review: [toolforge.infra] Replace Toolschecker alerts with Prometheus based ones - https://phabricator.wikimedia.org/T313030#12017834 (10fgiunchedi) [09:56:48] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139#12017842 (10fnegri) The longest transactions are now for user `s53685` ([editgroups](https://toolsadmin.wikimedia.org/tools/id/editgroups)): `lang=mysq... [09:58:34] FIRING: InstanceDown: Project tools instance tools-checker-5 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:59:39] 06cloud-services-team (FY2025/2026-Q3-Q4): Remove Icinga checks for Cloud VPS projects (not: infrastructure) - https://phabricator.wikimedia.org/T345983#12017848 (10taavi) >>! In T345983#12003748, @fgiunchedi wrote: > @taavi do you reckon there's anything in this task that's not covered by checks in {T328502} ?... [10:03:34] RESOLVED: InstanceDown: Project tools instance tools-checker-5 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:03:41] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [toolsdb] Automatically terminate long transactions - https://phabricator.wikimedia.org/T409857#12017891 (10fnegri) For reference, a useful query to find long transactions and their respective users: `lang=mysql MariaDB [(none)]> SELECT trx_id, trx_sta... [10:03:54] 10Tools, 06Commons, 07Community-Wishlist: Image Editor for Commons - https://phabricator.wikimedia.org/T426106#12017897 (10Doc_James) We built one here a while ago https://commons.wikimedia.org/wiki/Commons:ImageAnnotateTool [10:04:32] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on tools-puppetserver-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [10:14:32] RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on tools-puppetserver-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [10:20:10] 06cloud-services-team (FY2025/2026-Q3-Q4): Remove Icinga checks for Cloud VPS projects (not: infrastructure) - https://phabricator.wikimedia.org/T345983#12017932 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Sweet, resolving! [11:00:36] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.32.13 (T427919) [11:00:41] T427919: Upgrade tools cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427919 [11:20:00] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.32.13 (T427919) [11:20:06] T427919: Upgrade tools cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427919 [11:20:44] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.31.14 to 1.32.13 (T427919) [11:22:45] 10Toolforge, 06tools-infrastructure-team: Upgrade tools cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427919#12018149 (10taavi) `lines=10 ----- OUTPUT for command #1: 'sudo -i kubeadm ...ade plan 1.32.13' -----... [11:26:44] (03merge) 10fnegri: Replace only views that need updating [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/9 (https://phabricator.wikimedia.org/T351637) [11:26:45] (03update) 10fnegri: Add --diff-mode and remove --dry-run [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/10 (https://phabricator.wikimedia.org/T351637) [11:27:47] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.31.14 to 1.32.13 (T427919) [11:27:53] T427919: Upgrade tools cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427919 [11:27:55] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.31.14 to 1.32.13 (T427919) [11:30:20] (03merge) 10fnegri: Add --diff-mode and remove --dry-run [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/10 (https://phabricator.wikimedia.org/T351637) [11:30:21] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-8 from 1.31.14 to 1.32.13 (T427919) [11:30:21] (03update) 10fnegri: Add summary with counts [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/11 (https://phabricator.wikimedia.org/T351637) [11:31:14] (03merge) 10fnegri: Add summary with counts [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/11 (https://phabricator.wikimedia.org/T351637) [11:31:15] (03update) 10fnegri: Catch SQL errors [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/12 (https://phabricator.wikimedia.org/T351637) [11:31:40] (03merge) 10fnegri: Catch SQL errors [repos/cloud/wikireplicas-utils] - 10https://gitlab.wikimedia.org/repos/cloud/wikireplicas-utils/-/merge_requests/12 (https://phabricator.wikimedia.org/T351637) [11:33:06] 10Toolforge, 06tools-platform-team: toolforge worker upgrade cookbook sometimes fails when uncordoning - https://phabricator.wikimedia.org/T429157 (10taavi) 03NEW [11:33:18] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.31.14 to 1.32.13 (T427919) [11:33:19] 10Toolforge, 06tools-infrastructure-team: toolforge worker upgrade cookbook sometimes fails when uncordoning - https://phabricator.wikimedia.org/T429157#12018193 (10taavi) [11:33:24] T427919: Upgrade tools cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427919 [11:39:01] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.31.14 to 1.32.13 (T427919) [11:39:06] T427919: Upgrade tools cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427919 [11:40:14] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 (T427919) [11:40:51] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k [11:40:51] 8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8 [11:40:51] s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8s [11:40:51] -worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker-nfs-9 (T427919) [11:41:07] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k [11:41:07] 8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8 [11:41:07] s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8s [11:41:07] -worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker-nfs-9 (T427919) [11:42:17] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k [11:42:17] 8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8 [11:42:17] s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8s [11:42:17] -worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker-nfs-9 (T427919) [11:45:47] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo [11:45:47] rker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-wor [11:45:47] ker-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-work [11:45:48] er-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker [11:45:48] -nfs-9 (T427919) [11:45:52] T427919: Upgrade tools cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427919 [11:48:12] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k [11:48:12] 8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8 [11:48:12] s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-69, tools-k8s [11:48:12] -worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker-nfs-9 (T427919) [11:53:01] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 (T427919) [11:53:06] T427919: Upgrade tools cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427919 [11:54:40] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-1 from 1.31.14 to 1.32.13 (T427919) [11:55:34] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-1 from 1.31.14 to 1.32.13 (T427919) [11:56:24] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-2 from 1.31.14 to 1.32.13 (T427919) [11:57:17] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-2 from 1.31.14 to 1.32.13 (T427919) [11:58:15] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-3 from 1.31.14 to 1.32.13 (T427919) [11:58:20] T427919: Upgrade tools cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427919 [11:59:09] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-3 from 1.31.14 to 1.32.13 (T427919) [12:01:30] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud (T427919) [12:02:08] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud (T427919) [12:06:43] taavi@cloudcumin1001 upgrade_workers (PID 3351188) is awaiting input [12:06:55] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor [12:06:55] ker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-work [12:06:55] er-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worke [12:06:55] r-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker- [12:06:55] nfs-9 (T427919) [12:06:58] T427919: Upgrade tools cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427919 [12:16:54] (03PS1) 10Chuiimuii_ofc: Fix capitalization and consistency of UI strings; update messages.pot [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1302134 (https://phabricator.wikimedia.org/T354920) [12:17:03] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor [12:17:03] ker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-work [12:17:03] er-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worke [12:17:03] r-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker- [12:17:03] nfs-9 (T427919) [12:17:06] T427919: Upgrade tools cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427919 [12:17:57] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor [12:17:57] ker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-work [12:17:57] er-nfs-44, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-68, tools-k8s-worke [12:17:57] r-nfs-69, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-83, tools-k8s-worker- [12:17:57] nfs-9 (T427919) [12:18:09] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.run_tests [12:41:23] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) [12:43:46] 06cloud-services-team, 10Openstack-Magnum: Create two or three shared magnum templates - https://phabricator.wikimedia.org/T429164 (10Andrew) 03NEW [12:45:01] 10Toolforge, 06tools-infrastructure-team: Upgrade tools cluster to Kubernetes 1.32 - https://phabricator.wikimedia.org/T427919#12018473 (10taavi) 05Open→03Resolved [12:45:02] 10Toolforge, 06tools-platform-team: toolforge jobs logs: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url - https://phabricator.wikimedia.org/T413874#12018478 (10taavi) [12:45:06] 10Toolforge, 06tools-platform-team: toolforge jobs logs: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url - https://phabricator.wikimedia.org/T413874#12018479 (10taavi) Also seen during the upgrade to 1.32. [12:45:21] (03merge) 10taavi: kind: Upgrade Kubernetes to 1.32 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/325 (https://phabricator.wikimedia.org/T379047) [12:46:42] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.upgrade_mons (T428385) [12:51:56] PROBLEM - Host cloudcephmon1004 is DOWN: PING CRITICAL - Packet loss = 100% [12:53:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [12:53:44] RECOVERY - Host cloudcephmon1004 is UP: PING OK - Packet loss = 0%, RTA = 0.50 ms [12:53:57] 10Toolforge, 06tools-platform-team: toolforge jobs logs: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url - https://phabricator.wikimedia.org/T413874#12018560 (10dcaro) Seen locally also when recreating a fresh lima-kilo VM: ` /data/project/tf-test/venv/lib/python3.13/site-packages/bats_... [12:56:44] 06cloud-services-team, 10Cloud-VPS: Upgrade cloud-vps hosts to Debian Trixie - https://phabricator.wikimedia.org/T409579#12018571 (10Andrew) 05Open→03Resolved I'm going to close this task. The uefi images are resolved on existiing hosts, and everything except for ceph nodes is now running Trixie. The c... [12:59:40] PROBLEM - Host cloudcephmon1005 is DOWN: PING CRITICAL - Packet loss = 100% [13:00:41] 10Toolforge, 06tools-platform-team: toolforge webservice logs: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url - https://phabricator.wikimedia.org/T413874#12018582 (10dcaro) [13:01:04] RECOVERY - Host cloudcephmon1005 is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms [13:08:24] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.upgrade_mons (exit_code=99) [13:18:58] 10Toolforge, 06tools-platform-team: toolforge webservice logs: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url - https://phabricator.wikimedia.org/T413874#12018741 (10dcaro) Similar error (not sure it's a reproducer yet): ` local.tf-test@toolslocal:~/toolforge-deploy/functional-tests/tool... [13:21:44] 10Toolforge, 06tools-platform-team: toolforge webservice logs: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url - https://phabricator.wikimedia.org/T413874#12018748 (10dcaro) Probably, yep: ` (Pdb) print(e.response.text) { "kind": "Status", "apiVersion": "v1", "metadata": {}, "statu... [13:29:58] 10Toolforge, 06tools-platform-team: toolforge webservice logs: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url - https://phabricator.wikimedia.org/T413874#12018833 (10dcaro) I was able to reproduce (most times) by adding a `kubectl delete pods --all` to the test right before doing the webs... [13:38:39] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.upgrade_osds (T428385) [13:41:20] 10Tool-centralnotice-banner-editor: Change the templates to dark mode accepted colours as a default - https://phabricator.wikimedia.org/T429069#12018953 (10Oyelola_Victoria) a:03Oyelola_Victoria [13:42:42] (03open) 10vriaa: refactor: use Codex design tokens for template default colours [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/82 (https://phabricator.wikimedia.org/T429069) [13:43:36] 10Tool-centralnotice-banner-editor: Banner tool preview is not the same as on-wiki prview - https://phabricator.wikimedia.org/T429070#12018967 (10Oyelola_Victoria) a:03Oyelola_Victoria [13:44:22] 06cloud-services-team, 10Toolforge: [lima-kilo] SSL certificate errors after restarting the VM - https://phabricator.wikimedia.org/T427801#12018987 (10dcaro) We can implement a service to run the ldap script on reboot, similar to the one monitoring containerd: ` 03:41 PM ~/Work/wikimedia/lima-kilo (main|✔) dca... [13:48:27] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.upgrade_osds (exit_code=97) [13:50:37] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.upgrade_mons (T428385) [13:52:30] PROBLEM - Host cloudcephmon1004 is DOWN: PING CRITICAL - Packet loss = 100% [13:54:44] RECOVERY - Host cloudcephmon1004 is UP: PING OK - Packet loss = 0%, RTA = 0.21 ms [13:57:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [13:59:40] PROBLEM - Host cloudcephmon1005 is DOWN: PING CRITICAL - Packet loss = 100% [14:01:04] RECOVERY - Host cloudcephmon1005 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [14:06:40] PROBLEM - Host cloudcephmon1006 is DOWN: PING CRITICAL - Packet loss = 100% [14:07:10] (03open) 10dcaro: logs: handle the 400 returned by k8s [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/113 [14:07:42] 10Toolforge, 06tools-platform-team: toolforge webservice logs: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url - https://phabricator.wikimedia.org/T413874#12019173 (10dcaro) [14:07:55] 10Toolforge, 06tools-platform-team: toolforge webservice logs: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url - https://phabricator.wikimedia.org/T413874#12019177 (10dcaro) 05Open→03In progress [14:07:59] 10Toolforge, 06tools-platform-team: toolforge webservice logs: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url - https://phabricator.wikimedia.org/T413874#12019178 (10dcaro) a:03dcaro [14:08:08] RECOVERY - Host cloudcephmon1006 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms [14:08:59] (03update) 10dcaro: logs: handle the 400 returned by k8s [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/113 [14:10:10] (03update) 10dcaro: logs: handle the 400 returned by k8s [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/113 [14:10:12] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.upgrade_mons (exit_code=0) [14:12:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [14:15:24] 06cloud-services-team, 10Toolforge: toolforge webservice logs output has encoding issues and gets truncated compared to kubectl logs - https://phabricator.wikimedia.org/T429028#12019229 (10dcaro) @diegodlh How much does this impact your flow? We are working on moving the webservice cli to a completely differen... [14:15:38] (03update) 10renovatebot: Update Rust crate tower-http to 0.7.0 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/23 [14:15:42] (03open) 10renovatebot: Update Rust crate tower-http to 0.7.0 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/23 [14:16:24] (03update) 10dcaro: logs: handle the 400 returned by k8s [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/113 [14:18:30] 10Tools, 06Commons: locator-tool 401 "Error" - https://phabricator.wikimedia.org/T429192 (10kruusamagi) 03NEW [14:20:11] (03CR) 10Novem Linguae: "I don't have +2 on this repo. You'll probably want to add one of these people as reviewers: https://gerrit.wikimedia.org/r/admin/groups/b0" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1302134 (https://phabricator.wikimedia.org/T354920) (owner: 10Chuiimuii_ofc) [14:20:32] (03update) 10dcaro: logs: handle the 400 returned by k8s [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/113 [14:22:30] (03update) 10dcaro: logs: handle the 400 returned by k8s [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/113 [14:31:58] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.upgrade_osds (T428385) [14:36:08] PROBLEM - Host cloudcephosd1016 is DOWN: PING CRITICAL - Packet loss = 100% [14:36:38] RECOVERY - Host cloudcephosd1016 is UP: PING OK - Packet loss = 0%, RTA = 0.17 ms [14:37:10] (03close) 10dcaro: logs: handle the 400 returned by k8s [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/113 [14:37:11] (03open) 10dcaro: logs.kubernetes: retry when container not ready [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/94 [14:39:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [14:40:04] 10Tools: 'digero' tool uses an unreasonable amount of disk space - https://phabricator.wikimedia.org/T428430#12019587 (10taavi) [14:42:40] PROBLEM - Host cloudcephosd1017 is DOWN: PING CRITICAL - Packet loss = 100% [14:42:42] (03update) 10dcaro: logs.kubernetes: retry when container not ready [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/94 [14:44:08] RECOVERY - Host cloudcephosd1017 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [14:44:29] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27554179699 (https://github.com/cluebotng/component-configs/commits/a236330774424b9ce999258a01f924f1994594b1) [14:44:30] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27554179713 (https://github.com/cluebotng/component-configs/commits/a236330774424b9ce999258a01f924f1994594b1) [14:44:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [14:44:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [14:45:07] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27554179755 (https://github.com/cluebotng/component-configs/commits/a236330774424b9ce999258a01f924f1994594b1) [14:45:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [14:47:42] PROBLEM - Host cloudcephosd1018 is DOWN: PING CRITICAL - Packet loss = 100% [14:50:08] RECOVERY - Host cloudcephosd1018 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [14:52:51] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27554700598 (https://github.com/cluebotng/component-configs/commits/12298f8c7711b0dbc3ebe3196da055b62b307301) [14:52:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [14:55:20] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)), 07OKR-Work: Improve linting - detect examples in nested schema.properties - https://phabricator.wikimedia.org/T424002#12019760 (10KineticPelagic) [14:55:37] 10Tool-wmf-openapi-linter, 06MW-Interfaces-Team (MWI-Sprint-35 (2026-06-02 to 2026-06-16)), 07OKR-Work: Improve linting - detect examples in nested schema.properties - https://phabricator.wikimedia.org/T424002#12019766 (10KineticPelagic) 05In progress→03Resolved [14:55:40] PROBLEM - Host cloudcephosd1019 is DOWN: PING CRITICAL - Packet loss = 100% [14:56:08] RECOVERY - Host cloudcephosd1019 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [15:06:20] (03update) 10dcaro: logs.kubernetes: retry when container not ready [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/94 [15:11:24] (03update) 10dcaro: logs.kubernetes: retry when container not ready [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/94 [15:13:38] 10Toolforge, 06tools-platform-team: toolforge webservice logs: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url - https://phabricator.wikimedia.org/T413874#12019960 (10dcaro) okok, got a working MR in toolforge-weld, note that this will go away when we move to jobs-api + logs-api. https://... [15:14:15] 06cloud-services-team, 10Data-Services, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 13Patch-For-Review: Send JSON access logs for dumps.wikimedia.org to Kafka - https://phabricator.wikimedia.org/T425087#12019962 (10BTullis) Sorry, there are just too many different options here for me to be comfortable m... [15:15:19] 10Toolforge, 06tools-platform-team: Regression: Images built with the build service images do no longer contain/configure locales specified in .locales - https://phabricator.wikimedia.org/T428230#12019964 (10dcaro) p:05Triage→03Medium a:03dcaro [15:21:21] (03update) 10fnegri: logs.kubernetes: retry when container not ready [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/94 (owner: 10dcaro) [15:21:22] (03update) 10fnegri: logs.kubernetes: retry when container not ready [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/94 (owner: 10dcaro) [15:21:37] (03approved) 10fnegri: logs.kubernetes: retry when container not ready [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/94 (owner: 10dcaro) [15:24:36] 06tools-platform-team: [builds-api] Add support for using a commit hash as ref - https://phabricator.wikimedia.org/T429227 (10dcaro) 03NEW [15:26:30] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27556781475 (https://github.com/cluebotng/component-configs/commits/81d6af2f5fa912449070fe6ac104761023a9960b) [15:26:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [15:31:04] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27557053203 (https://github.com/cluebotng/component-configs/commits/80c3b871693d13296e1c4640105e4a8cabc9d5a4) [15:31:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [15:33:46] (03approved) 10dcaro: logs.kubernetes: retry when container not ready [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/94 [15:33:54] (03merge) 10dcaro: logs.kubernetes: retry when container not ready [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/94 [15:36:10] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 10Toolforge, 10Observability-Alerting, and 3 others: Move WMCS off of Icinga and introduce alertmanager - https://phabricator.wikimedia.org/T328502#12020109 (10Andrew) > 2. check-flavor_aggregates ? It's useful to prevent creation of broken flavors (... [15:42:21] (03open) 10dcaro: bump_version: copy from jobs-api [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/95 [15:44:31] 06cloud-services-team, 10Toolforge: [builds-cli] does not output valid json - https://phabricator.wikimedia.org/T429229 (10DamianZaremba) 03NEW [15:47:19] (03update) 10dcaro: bump_version: copy from jobs-api [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/95 [15:49:47] (03update) 10dcaro: bump_version: copy from jobs-api [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/95 [15:50:25] (03update) 10dcaro: bump_version: copy from jobs-api [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/95 [15:52:25] (03open) 10dcaro: bump toolforge weld [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/96 (https://phabricator.wikimedia.org/T413874) [15:53:23] (03update) 10dcaro: d/changelog: bump to 1.6.15 [repos/cloud/toolforge/toolforge-weld] (add_bump_branch) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/96 (https://phabricator.wikimedia.org/T413874) [15:53:48] (03update) 10dcaro: d/changelog: bump to 1.6.15 [repos/cloud/toolforge/toolforge-weld] (add_bump_branch) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/96 (https://phabricator.wikimedia.org/T413874) [15:53:55] (03update) 10dcaro: d/changelog: bump to 1.6.15 [repos/cloud/toolforge/toolforge-weld] (add_bump_branch) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/96 (https://phabricator.wikimedia.org/T413874) [15:54:12] (03update) 10dcaro: d/changelog: bump to 1.6.15 [repos/cloud/toolforge/toolforge-weld] (add_bump_branch) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/96 (https://phabricator.wikimedia.org/T413874) [15:54:27] (03update) 10dcaro: Draft: d/changelog: bump to 1.6.15 [repos/cloud/toolforge/toolforge-weld] (add_bump_branch) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/96 (https://phabricator.wikimedia.org/T413874) [15:55:46] (03update) 10dcaro: bump_version: copy from jobs-api [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/95 [15:57:05] 06cloud-services-team, 10Toolforge: [jobs-cli] emits a warning to re-create valid jobs - https://phabricator.wikimedia.org/T429231 (10DamianZaremba) 03NEW [15:57:32] 06cloud-services-team, 10Toolforge: [jobs-cli] emits a warning to re-create valid jobs - https://phabricator.wikimedia.org/T429231#12020231 (10DamianZaremba) [15:59:22] 06cloud-services-team, 10Toolforge: [builds-cli] does not output valid json when there's no builds - https://phabricator.wikimedia.org/T429229#12020235 (10dcaro) [16:02:43] 06tools-platform-team: [builds-api] Add support for using a commit hash as ref - https://phabricator.wikimedia.org/T429227#12020248 (10dcaro) As a workaround, in gitlab each MR stores a ref, like: ` refs/merge-requests/10/head ` [16:03:06] 06cloud-services-team, 10Toolforge: [jobs-cli] emits a warning to re-create valid jobs - https://phabricator.wikimedia.org/T429231#12020249 (10dcaro) p:05Triage→03High [16:06:52] 06tools-platform-team: [builds-api] Add support for using a commit hash as ref - https://phabricator.wikimedia.org/T429227#12020279 (10dcaro) p:05Triage→03Low [16:10:01] 06tools-platform-team: [toolforge-weld] update bump_version.sh - https://phabricator.wikimedia.org/T429235 (10dcaro) 03NEW [16:10:07] 06tools-platform-team: [toolforge-weld] update bump_version.sh - https://phabricator.wikimedia.org/T429235#12020336 (10dcaro) p:05Triage→03High [16:11:31] (03update) 10dcaro: bump_version: copy from jobs-api [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/95 [16:15:28] 06cloud-services-team, 10Toolforge, 07Documentation, 07good first task: Create a new doc about managing and sharing files in Toolforge - https://phabricator.wikimedia.org/T347753#12020363 (10apaskulin) a:03Tejinderk.2004 Hi @Tejinderk.2004, I've gone ahead and assigned the task to you. Thank you! [16:23:48] (03update) 10fnegri: bump_version: copy from jobs-api [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/95 (owner: 10dcaro) [16:23:49] (03update) 10fnegri: bump_version: copy from jobs-api [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/95 (owner: 10dcaro) [16:23:53] (03approved) 10fnegri: bump_version: copy from jobs-api [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/95 (owner: 10dcaro) [16:24:02] (03update) 10fnegri: bump_version: copy from jobs-api [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/95 (owner: 10dcaro) [16:24:22] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27560286658 (https://github.com/cluebotng/component-configs/commits/5c1faa1ebe1b269c9d3e13fc4d7c7a9fddf23bf2) [16:24:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [16:27:12] (03merge) 10dcaro: bump_version: copy from jobs-api [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/95 [16:27:16] (03update) 10dcaro: Draft: d/changelog: bump to 1.6.15 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/96 (https://phabricator.wikimedia.org/T413874) [16:27:50] (03close) 10dcaro: Draft: d/changelog: bump to 1.6.15 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/96 (https://phabricator.wikimedia.org/T413874) [16:28:59] 06tools-platform-team: [toolforge-weld] update bump_version.sh - https://phabricator.wikimedia.org/T429235#12020408 (10dcaro) 05Open→03Resolved [16:32:30] (03open) 10dcaro: d/changelog: bump to 1.6.15 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/97 (https://phabricator.wikimedia.org/T413874) [16:58:05] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [16:58:18] (03merge) 10vriaa: refactor: use Codex design tokens for template default colours [toolforge-repos/centralnotice-banner-editor] - 10https://gitlab.wikimedia.org/toolforge-repos/centralnotice-banner-editor/-/merge_requests/82 (https://phabricator.wikimedia.org/T429069) [17:03:56] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/27562697218 (https://github.com/cluebotng/component-configs/commits/5c1faa1ebe1b269c9d3e13fc4d7c7a9fddf23bf2) [17:03:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [17:07:30] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.upgrade_osds (exit_code=99) [17:09:51] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [17:11:24] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/27562889711 (https://github.com/cluebotng/component-configs/commits/b9c48365007a90ed1c240655306615542751ea6f) [17:11:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [17:13:19] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld [17:14:06] 10Cloud-VPS (Quota-requests): Quota increase request for project wlm-it-visual - https://phabricator.wikimedia.org/T427731#12020581 (10Andrew) +1 approved [17:19:58] !log volans@cloudcumin1001 wlm-it-visual START - Cookbook wmcs.openstack.quota_increase by 24 cores, 20 gigabytes, 32768 ram (T427731) [17:20:02] T427731: Quota increase request for project wlm-it-visual - https://phabricator.wikimedia.org/T427731 [17:20:06] !log volans@cloudcumin1001 wlm-it-visual END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 24 cores, 20 gigabytes, 32768 ram (T427731) [17:20:58] (03open) 10dcaro: deploy_task: wait for empty slot when starting build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/173 [17:23:03] 10Cloud-VPS (Quota-requests): Quota increase request for project wlm-it-visual - https://phabricator.wikimedia.org/T427731#12020626 (10Volans) 05Open→03Resolved a:03Volans Limit increased. Just a few suggestions: 1. Try to not create a full-size stating environment, it can surely have much less resour... [17:26:28] 10Cloud-VPS (Project-requests): Request creation of eduwikihubstaging VPS project - https://phabricator.wikimedia.org/T429032#12020645 (10Volans) @Ederporto will the admins of the projects be the same of the existing project (`globaleducation` if I'm not mistaken)? Because if they are the same, then there is no... [17:26:56] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld [17:30:59] (03update) 10dcaro: deploy_task: wait for empty slot when starting build [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/173 [17:31:08] (03approved) 10dcaro: d/changelog: bump to 1.6.15 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/97 (https://phabricator.wikimedia.org/T413874) [17:31:15] (03merge) 10dcaro: d/changelog: bump to 1.6.15 [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/97 (https://phabricator.wikimedia.org/T413874) [17:33:53] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: toolforge webservice logs: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url - https://phabricator.wikimedia.org/T413874#12020683 (10dcaro) 05In progress→03Resolved [17:35:35] 06tools-platform-team: [toolforge-weld] Fails to publish to pypi - https://phabricator.wikimedia.org/T429241 (10dcaro) 03NEW [17:35:41] 06tools-platform-team: [toolforge-weld] Fails to publish to pypi - https://phabricator.wikimedia.org/T429241#12020696 (10dcaro) p:05Triage→03Medium [18:27:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.upgrade_osds (T428385) [18:40:26] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.upgrade_osds (exit_code=99) [19:00:10] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.upgrade_osds (T428385) [19:32:43] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.upgrade_osds (exit_code=0) [19:46:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.upgrade_osds (T428385) [19:46:39] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [19:53:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [20:08:00] 10Cloud-VPS (Project-requests): Request creation of eduwikihubstaging VPS project - https://phabricator.wikimedia.org/T429032#12021219 (10Ragesoss) @Volans they will not be; we want a separate project so that @JGonzalez_EdWH can be an admin of the new one, without access to the production instances. [20:17:01] 10Cloud-VPS (Project-requests): Request creation of eduwikihubstaging VPS project - https://phabricator.wikimedia.org/T429032#12021246 (10Andrew) +1, a new project is a good way to provide human access separation [20:29:47] FIRING: NodeDown: Node cloudcephosd1044 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1044 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [20:59:17] RESOLVED: NodeDown: Node cloudcephosd1044 is down. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1044 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [21:02:57] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.upgrade_osds (exit_code=99) [21:04:23] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.upgrade_osds (T428385) [21:20:59] 10Tool-centralnotice-banner-editor: Banner tool preview is not the same as on-wiki prview - https://phabricator.wikimedia.org/T429070#12021467 (10Ciell) Fyi, screenshot is in Vector 2010, I didn't realize I had my meta settings set to the older skin. But same difference seems to occur in Monobook and Timeless.... [21:21:36] 10Tool-centralnotice-banner-editor: Banner tool preview is not the same as on-wiki preview in skins other than Vector 2022 - https://phabricator.wikimedia.org/T429070#12021469 (10Ciell) [21:27:37] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: Consider allowing cumin access to all Cloud VPS VMs - https://phabricator.wikimedia.org/T422801#12021474 (10Andrew) good news: despite being configured to use Ignition, magnum actually just uses cloud-init like any other VM. So we can wedge in the... [21:39:51] 06cloud-services-team, 10Toolforge: [logs-api] failing to return logs for job - https://phabricator.wikimedia.org/T429265 (10DamianZaremba) 03NEW [21:48:52] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.upgrade_osds (exit_code=0) [21:53:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [21:55:10] 06cloud-services-team, 10Cloud-VPS, 10Ceph: cloudcephosd1044 boot issues - https://phabricator.wikimedia.org/T429267 (10Andrew) 03NEW [23:21:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [23:41:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors