[13:10:06] Okay, I have not seen this before [13:10:10] amir@amir:~$ ssh worker01.huma.eqiad1.wikimedia.cloud [13:10:10] channel 0: open failed: connect failed: No route to host [13:14:30] Amir1: run with `ssh -v` please? [13:15:13] https://www.irccloud.com/pastebin/ktPNituE/ [13:15:27] of course the whole thing is way too long [13:38:34] I can ssh into another vm in the same group, I probably gonna force a reboot [13:42:00] The last Puppet run was at Fri Nov 14 05:06:06 UTC 2025 (515 minutes ago). [13:42:00] Last Puppet commit: [13:42:16] I think I destroyed the VM. Meh :D [14:50:10] quick review: https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1078 [15:14:46] I also need a +1 on https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/279 [15:26:51] andrewbogott: if you're around, do you remember how to add the "high-iops" volume type to a new project? [15:26:54] context is T409365 [15:26:54] T409365: Grant zuul project access to `fast-iops` volume type and `4xiops` instance flavor - https://phabricator.wikimedia.org/T409365 [15:27:20] I cannot find it either in wikitech or in the openstack docs. :/ [15:28:45] ah wait I think I found it, quite hidden :) https://docs.openstack.org/cinder/2025.2/admin/get-capabilities.html#usage-of-volume-type-access-extension [15:29:20] I think 'openstack volume type set' something something [15:29:58] hm or maybe not [15:30:21] oh yeah, it is, it's just that 'show' doesn't show the project list [15:32:44] dhinus: did that work? [15:32:54] yep! "show" does show the access list [15:33:04] but doesn't tell you how to change it :) [15:33:26] can we handle that in tofu? [15:33:29] "set --project" worked, I was afraid it would wipe out the others but it doesn't [15:33:35] taavi: just checked, and yes https://search.opentofu.org/provider/terraform-provider-openstack/openstack/latest/docs/resources/blockstorage_volume_type_access_v3 [15:33:39] I'll try to import it [15:33:56] andrewbogott: we have an orphan reference to "clouddb-services", a project that doesn't exist anymore [15:34:12] I tried clearing it but I get 'Failed to remove volume type access from project: No project with a name or ID of 'clouddb-services' exists.' [15:34:33] but it still shows in "openstack volume type show high-iops" [15:35:06] ok. We can ignore it or I can fix it in the db [15:38:10] I'm fine with ignoring, unless it breaks the tofu import [15:38:14] I created T410148 [15:38:14] T410148: tofu-infra: add cinder volume types - https://phabricator.wikimedia.org/T410148 [19:54:11] andrewbogott: found the immediate CAS issue, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1205208 [19:55:18] if you're curious, I took a packet capture of the communication between envoy and CAS so that I could reproduce the exact request with curl, after which it was a simple search to find out which exact header was changing things [19:55:33] any guess why it worked on bookworm? [19:56:16] not really [19:56:53] could've been a hotfix; I didn't try reimaging with bookworm [19:58:57] I still don't quite understand. If tomcat doesn't trust the 127. address why does it redirect rather than just failing? [20:01:56] tomcat doesn't trust the 208.80.153.41 address in our envoy config [20:02:17] so my best guess is that it ignores the x-forwaded-proto header and thinks it's a http request, and thus tries to redirect you to https [20:02:47] ah, ok, that makes sense. So the change could be due to the tomcat version change [20:23:19] thank you for working on that, taavi! If you want an even messier one, T407586 should be fun. [20:23:20] T407586: latest Trixie image (as of 2025-10-16) grub failure on R450 hardware - https://phabricator.wikimedia.org/T407586 [20:23:37] not today at least. [20:23:58] sensible