[08:00:37] morning [08:40:12] dcaro: when you have a moment, there are some harbor quota requests that I'd like your opinions on: T405643 T405644 T405645 [08:40:13] T405643: Request increased build quota for cluebotng-review Toolforge tool - https://phabricator.wikimedia.org/T405643 [08:40:13] T405644: Request increased build quota for cluebotng-monitoring Toolforge tool - https://phabricator.wikimedia.org/T405644 [08:40:14] T405645: Request increased build quota for cluebotng Toolforge tool - https://phabricator.wikimedia.org/T405645 [08:42:06] 👀 [08:43:12] taavi: they look ok to me, 1G is indeed kinda little if you are building more than one image [08:43:25] * dcaro double checks the current storage usage just in case [08:43:40] do you think the default quota needs adjusting or is handling these as individual requests the way to go? [08:45:03] We can bump it a bit yes, though we don't get too many requests either imo, up to you [08:45:32] I'll just implement these then [08:46:38] 👍 [08:47:19] we are at ~59% of object storage quota, grew 1% in the last couple weeks (it has bumps and valleys, so cleanup is probably working as expected) [08:47:40] https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/982 [08:47:56] (huh, almost at MR 1,000 there) [08:49:10] yep :), many deploys [09:18:21] hello! [09:57:13] hi :) [10:17:32] Something I noticed today about stuck nfs workers [10:17:35] https://usercontent.irccloud-cdn.com/file/zw2Va3Fq/image.png [10:17:42] a couple of them got unstuck by themselves [10:22:54] there's this on the dmesg, for k8s-worker-61 (the white one in that graph) [10:23:31] https://www.irccloud.com/pastebin/yXqsxlT2/ [10:24:05] * dcaro lunch [11:50:17] btw. quick review https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/145 [17:03:06] * dhinus off