[08:26:28] morning [08:26:44] greetings [09:26:03] dcaro: thank you for the quick reviews re: dumps-nfs, one additional question, I was thinking about the mount propagation on failover, and we'd need to set HostToContainer (https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation) is that part of volume-admission too if I'm not mistaken ? [09:28:05] godog: yep, should be somewhere here I think https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/blob/main/server/admission.go?ref_type=heads#L276 [09:28:23] dcaro: nice thank you, will start from there and report back [09:28:57] note that if/when we deprecate the other mountpoint, we will have to send first an email to cloud-announce and give them some time to change it as some people are creating their own manifests and such (maybe koml.a can help there) [09:29:03] I feel like when you start pulling a thread on a knitted sweater and end up destroying the sweater [09:29:30] dcaro: good point, I had not thought of that [09:29:58] I'm assuming you mean in other cloud vps projects ? [09:30:04] as opposed to tools that is [09:35:56] lol (re: I feel like when you start pulling a thread on a knitted sweater and end up destroying the sweater) [09:41:26] oh no, both, toolforge users historically have been creating their own pods and such, and some rely on the volume-admission, set me check if we ended up just not allowing to specify directly (and rely on the labels and volume-admission doing the volume addition), but I think we did not [09:42:08] ack thank you [09:44:03] we are not allowing it anymore, so that would be ok \o/, toolforge users should not be affected by the change (not even if they manually create their jobs/pods), https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/blob/main/server/admission.go?ref_type=heads#L191 [09:44:30] of course, it was me xd https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/commit/0ca3b79d5f325da09d41764f63d72df5964b709a [09:46:14] lol, sweet yeah makes sense as a policy [09:51:20] I noted though that the dumps mounts inside toolforge have soft links to paths that don't exist in the pods tree, like `lrwxrwxrwx 1 root root 37 Aug 6 2013 tools -> /data/xmldatadumps/public/other/tools` [09:51:43] might be hapenning also in VMs [09:51:47] *happening [09:56:54] mmhh interesting, ok if those are not working now then I'm not super concerned, what's the full path to that 'tools' symlink ? [10:01:59] So the main entry point is the symlink `/public/dumps/public`, that points currently to `/mnt/nfs/dumps-clouddumps1001.wikimedia.org/`, so it could be accessed both `/public/dumps/public/tools` and `/mnt/nfs/dumps-clouddumps1001.wikimedia.org/tools`, and with the new mount, it could be access also through `/mnt/nfs/dumps/tools` [10:02:58] there's a few more symlinks [10:05:24] yes that checks out, the symlinks in /public/dumps [10:06:33] but yeah the symlinks to /data/xmldatadumps are not working now either [10:06:59] i.e [10:07:02] filippo@toolsbeta-bastion-7:~$ ls -la /public/dumps/public/tools/ ls: cannot access '/public/dumps/public/tools/': No such file or directory [10:07:26] they do work in production though, puppet there does mount /data/xmldatadumps/public/ [10:10:02] * godog lunch [11:07:53] * dcaro lunch [13:26:37] there's a toolsdb alert firing (ToolsDBHistoryLengthGrowing). I silenced it for 2 days and reopened T428139 [13:26:38] T428139: [toolsdb] Transaction History Length growing too much - https://phabricator.wikimedia.org/T428139 [13:27:38] thanks [15:57:05] Maybe I'm just not fully awake yet, but... what is failing about this test? https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/jobs/899097 [15:58:05] andrewbogott: the files listed at 31-32 do not match what the autoformatter wants them to be [15:58:46] oh, ok [15:58:57] It doesn't say that, though, does it? You just know that from seeing that failure before? [16:01:17] it says that the "tofu fmt" command failed and I can't think of an alternative explanation for those output lines [16:01:46] * andrewbogott nods [17:43:31] andrewbogott: this might be interesting https://wiki.archlinux.org/title/Disk_quota [18:11:56] * dcaro off [18:11:58] cya tomorrow!