[06:15:20] greetings [07:49:24] morning! [08:06:51] morning [08:30:11] sigh, a problem I didn't consider with dumps-nfs and containers and mount propagation, we'll need to bind-mount the parent directory of the mountpoint for things to work as expected [08:30:18] I'm opening a task with more details, just wanted to share [08:35:09] :/ [08:37:43] oh, okok [08:40:39] I'm sure there was a joke there with 'share' and 'unshare' and containers [08:42:32] lol [08:42:52] 'bind-mount' might help too [08:43:29] yes indeed [08:56:59] hmm... I rebuilt my lima-kilo yesterday, and it seems it failed to apply the inotify watches limit correctly or something, as it was not set, rerunning ansible did apply it ok, maybe it disappears after reboot? [08:58:22] dcaro: did it disappear after rebuild, or after rebuild+reboot? [08:58:34] I think I did not reboot [08:58:40] so rebuild would be [08:58:56] I haven't tested reboots, but the last rebuild I tested I think it was working, let me try again [08:59:20] I only noticed a few full functional test runs later, when it got out of watches [08:59:48] this parameter is suspicious xd https://docs.ansible.com/projects/ansible/latest/collections/ansible/posix/sysctl_module.html#parameter-sysctl_set [09:04:12] in the logs of the rebuild I did, it says it ran it ok [09:04:13] https://www.irccloud.com/pastebin/aJyaYq4k/ [09:04:19] maybe you also need "reload: True"? [09:04:38] should be the default according to the docs [09:04:53] I think we can set both, reload: true, and sysctl_set: true, just in case [09:05:10] ah yeah you're right, i didn't check the default for "reload" [09:06:11] I read "sysctl_set" as "only modify the file if the current live value is different", but it's not super clear [09:06:54] it's been there since the initial commit, so there's no extra context in the code [09:08:03] just tested a full rebuild of lima-kilo, and the values are set correctly [09:08:22] (checked with "sudo sysctl -a |grep inotify") [09:08:34] let me try a VM reboot [09:09:31] what I understand it does is check that the operating value (`sysctl `) is the expected after modifying the file + reloading, and if not, then run `sysctl -w =` [09:09:32] ha, reboot does reset it! [09:09:37] my values are [09:09:39] https://www.irccloud.com/pastebin/YlK7xhxL/ [09:09:41] (were) [09:09:49] so we need to add it to bashrc or similar [09:09:52] ooohh, what do you have in `/etc/sysctl.conf [09:09:54] ? [09:10:06] it's there [09:10:12] :/ [09:11:19] " sudo sysctl -p" loads them [09:11:26] but it's not run automatically apparently [09:12:18] claude points me to https://bugs-devel.debian.org/cgi-bin/bugreport.cgi?bug=1077184 [09:16:30] Ohhhh, that seems to be true [09:17:40] https://www.irccloud.com/pastebin/28DBnDU6/ [09:17:59] that's what the systemd service is running [09:18:00] https://www.irccloud.com/pastebin/Lb5f9yVk/ [09:21:30] https://github.com/ansible-collections/ansible.posix/blob/ef1937b771670f104853d16888dd1083f983bc65/plugins/modules/sysctl.py#L18 [09:21:47] ` - If you are using C(systemd-sysctl(8)), note that it won't use the file C(/etc/sysctl.conf).` [09:21:50] xd [09:22:45] I'll send a quick patch [09:23:16] unless someone wants to fix it instead? thilp maybe you want to play with lima-kilo+ansible? T432323 [09:23:16] T432323: [lima-kilo] put sysctl config under /etc/sysctl.d/, as /etc/sysctl.conf is not being used anymore - https://phabricator.wikimedia.org/T432323 [09:24:19] hmm... that helpful message is nowere in the generated docs though: https://docs.ansible.com/projects/ansible/latest/collections/ansible/posix/sysctl_module.html#parameter-sysctl_set [09:24:23] *nowhere [09:32:44] * thilp is happy to take this one, you already figured everything out :D [10:24:07] the task I mentioned T432325 [10:24:08] T432325: Make sure dumps-nfs mount/umount is propagated inside containers - https://phabricator.wikimedia.org/T432325 [10:24:19] interested in your input when you get a chance [10:34:19] I'll have a look after lunch [10:34:22] * dcaro lunch [10:36:33] thank you ! [10:36:44] JFYI I'll be out this afternoon and tomorrow [12:33:07] godog: replied in task, essentially, I think it's ok to mount /mnt/nfs for now, in the future we might want to change it, but until we have a clearer idea on how it's hard to make it more generic [12:34:37] dcaro: nice! thank you, good point re: "all or nothing" and moving to mounting /mnt/nfs makes things easier! sweet I'll followup next week [12:39:51] I've created T432338 as there's some weird network/ip6 issue on toolsbeta it seems (my early checks), I've set it to tools-infrastructure-team, let me know if you think should be platform-side instead [12:39:53] T432338: [toolsbeta,infra,admin] toolsbeta-test-k8s-haproxy7 is flaky on ip6 when replying to admin - https://phabricator.wikimedia.org/T432338 [12:40:21] ack thank you [13:18:18] what was the magic command to make someone an admin in cloudvps to see all project? (for Raymond_Ndibe and thilp) [13:19:06] dcaro: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Projects_lifecycle#Working_with_projects [13:19:38] awesome, thanks, not sure how I missed it, I'll add it also to the onboarding template [13:21:18] beware the dropdown [13:21:54] (/me also wonders how much that is needed by the platform team SWEs these days if more and more of cloud vps specific stuff is handled by infra these days) [13:36:33] taavi, dcaro, David talked me through this a bit yesterday but I still can not at all follow what's happening with the 'modules' file in tofu-infra. At the moment my patch is mostly blind-copying/magical thinking; do you have the patience to talk me through what the bits actually are/should be? https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/331/diffs [13:38:12] sure, but first, where did 206.225.75.225 come from as the DNS server?? [13:39:58] huh... I don't know! Should be 172.20.254.1, I will fix [13:40:59] oh, I see where it came from, it's from 'cat /etc/resolv.conf' but in the wrong terminal :p [13:41:14] bah :D [13:41:46] anyway, I guess the 'validate' CI issue is the one you're confused about? [13:42:03] someone overcomplicated that repo while I wasn't looking so I need to do a bit of catch up to figure out what's going on there [13:43:41] It's not so much the specific error reports that concern me as me just literally not knowing what I'm doing with the module changes. For instance, in the paws tofu code we just say 'resource "openstack_containerinfra_clustertemplate_v1"' and it makes one, no need to define types at all... [13:44:06] In tofu-infra there's that extra layer of type definition which is fine but I don't really understand how to get it right. If it all looks right to you already that is entirely accidental [13:45:16] honestly I would not attempt to fit this into the existing per-project logic given that these are, well, not project-specific [13:46:15] But openstack will regard them as 'in' a project even if world-readable. Doesn't that mean they need to be created as part of a project? [13:46:28] Or you're thinking top-level in the tofu code and just hard-code the project id? [13:47:54] do we have a need, in this repo, to declare cluster templates for any arbitrary projects? or do we just have (again, for templates declared in this repository only) a generic set of templates that just so happen to live in a specific project? [13:48:32] As of today we don't need to do it for arbitrary projects. [13:48:49] Well, we could move the template definition from paws/quarry tofu into infra tofu but I don't know why we would do that. [13:49:03] as of today? is there a realistic scenario where that would change? [13:49:49] um... *waves hands* if toolforge moved to magnum maybe we'd tofu-ize the template. But that's not happening anytime soon and I can't think of other great examples. [13:50:13] I'm just wasting air, the answer is 'generic set of templates that just live in a project because that's how openstack works' [13:50:35] if toolforge moves to magnum they would live in the toolforge repo and not here? [13:50:51] yeah, that's certainly how paws and quarry work [13:51:54] but yeah, I would just treat those as generic things, so the simple but clean enough option would ihmo be to move what you currently have in `resources/codfw1dev-r/magnum/clustertemplates.tf` to a new module (so a new directory under modules/) and include it either on the top level or the deployment-specific resources directory, I would say [13:53:33] that confuses me even more :) Is there any easy way to explain what's in 'modules' vs whats in 'resources'? [13:53:55] Because I thought 'modules' was basically just type definitions but now you're suggesting I define the actual templates there... [13:54:09] would the new dir under modules still allow for datacenter distinctions? [13:55:47] modules are just generic re-usable chunks of manifests that can do whatever you want to do with them [13:55:55] (technically even the resources directory are just modules :D) [13:56:48] ok, so I would just put roughly the definitions I have now into modules/ and don't need to define any kind of intermediate resources? That would be nice! [13:57:16] you could of course do a similar wiring with all the data somewhere in resources/ and some wrapper taking those arrays and declaring actual resources in modules/, but again I think that's just unnecessary complexity [13:57:30] ok! I will take a stab at that and then you can have another round of trying to unconfuse me. [14:52:07] taavi: fmt still doesn't like me, but is this close to what you were suggesting? https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/333/diffs [14:56:25] andrewbogott: why have separate eqiad1-r/ and codfw1dev-r1/ modules in there with pretty much identical code instead of adjusting the variables where needed? [14:59:26] can you tell me where the deployment-specific vars would be defined? [15:00:03] where you're including the modules in resources/.../main.tf [15:00:14] ok, so just in line in main.tf. [15:00:25] when you declare a variable in a module, it's usually so that you can set it when including that module [15:00:47] ok [15:00:50] oops meeting time [15:58:39] also, s/public-resources/magnum_templates/, I would say, since that's the only thing actually in there [16:00:34] ok! [16:00:55] where do I define the variable that I want to 'pass' to the included file? [16:01:18] your existing `variable "..." {}` were doing exactly that already? [16:04:39] I mean, in resources/codfw1dev-r/main.tf how do I set 'datacenter=codfw1dev' so that the switches work in the included file? [16:05:19] add it as a value to the `module "magnum_templates" {}` block [16:05:38] also, passing in the datacenter and then having switches inside the module seems a lot more work and lot less flexible than passing just those values directly [16:06:23] huh, does it? [16:06:32] I'm ambivalent, happy to change [16:15:57] (the reason it's like that is because paws/magnum do it with one big dc switch, was following that pattern) [17:14:00] * dcaro off [17:14:03] cya on monday! [17:14:15] uuu, tofu stuff xd, scary [18:10:45] ^ after way too big of a distraction thanks to gitlab hostname changes and https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/5 [23:35:55] cteam: something to keep an eye out for -- I've had to hard reboot 2 vms, logging-logstash-04 in the logging project and toolhub-demo02 in the toolhub project, because they had completely fallen off the internal network. The commonality looks to have been that both vms were live migrated on 2026-07-06. [23:41:15] bd808: what were the signs that they were non-responsive? wondering if there are others in a similar predicament? [23:42:55] bliviero: for the logging one it was `scap` failing on deploment-prep with a "no route to host" message. For the toolhub demo server it was no route to host when I tried to ssh in to see why the webservice was not working. [23:44:13] If there is a log of what vms were migrated somewhere it might be enough just to try and ping them all. [23:44:43] it is also possible that the network issues are unrelated, but it seemed suspicious. [23:45:15] oof...i suspect that every VM has migrated around as security patches were getting deployed...repeatedly. would a ping to all VMs (or maybe a cumin run) help ID all affected? [23:45:36] i am not in a position to do so, but andrewbogott probably can [23:46:27] cumin might be a good check, yeah. Just a "/bin/true" across the fleet would give you a list of unreachables. [23:47:19] * bd808 runs away for dinner and a baseball game