[07:34:09] greetings [07:41:46] morning [07:58:24] unless there are objections I'll go ahead with cloudnet/cloudgw reboots shortly for T441631 [07:59:34] +1 [08:06:41] godog: should we add a manual downtime on icinga in the procedure to reduce the noise/surprise effect (see -feed) [08:07:02] volans: totally, will do now [08:07:38] icinga UI is so snappy these days [08:07:41] <3 [08:30:55] are puppet failures expected? within the last ~15 min I received a failure email for each instance in my project [08:33:55] JJMC89: not afaik, checking [08:34:44] yes are expected [08:35:10] nevermind [08:52:55] case in point ^ stashbot took 15m to notice the ping timeout [08:53:38] I'll file separate tasks for those [08:53:52] wow [08:57:10] yeah wm-bot did the same didn't it [09:03:18] morning [09:56:45] morning [10:17:14] got down to /dev/sdb ext4 10T 7.9T 1.6T 84% /srv/tools with the uwsgi soft truncation (left 10k lines on each) [10:33:34] https://phabricator.wikimedia.org/T443531#12514710 oops :D [10:34:42] lol [10:38:24] for today's riddle we got T443577 [10:38:24] T443577: Occasional high ping from irb-1120.cloudsw1-c8-eqiad.eqiad1.wikimediacloud.org - https://phabricator.wikimedia.org/T443577 [10:39:40] "yay" [10:39:53] does that happen (on the same hop?) for traffic in the other direction? [10:40:21] good question, I have not tested that yet [10:40:35] also haven't tested tcp/udp yet [10:40:49] gotta run an errand then lunch [10:41:05] my immediate first thought is there's some interface that's getting bursts maxing it out every minute [10:44:09] dcaro: do you know what's up with the (acked) toolsbeta-harbor alerts? [10:46:25] not really, there's a task to follow up on it [10:46:35] or was there? I'd swear I created one :/ [10:47:40] server returned HTTP status 401 Unauthorized from prometheus [11:09:22] hmm... not sure why it requires auth :/ [11:12:34] restarted harbor containers in toolsbeta, now it works [11:54:12] do we have a supposedly communicated limit for NFS usage in toolforge? [11:54:39] as in: can I create a task saying you're using too much space for the top tools in nfs usage? [12:25:04] I think we do not [12:27:14] given we run at 85% used space maybe we should :D [12:27:18] *set one [12:35:44] yep, there was a task also to check again if it's possible setting quotas [12:35:56] (iirc) [12:36:32] yes discussed extensively in the past [13:13:27] there is no formally defined quota but i have opened a bunch of tasks saying that [disk space used by tool] is not reasonable [13:26:18] if we discuss a formal proposal for a quota, what would be the way to enforce it? [13:26:54] i.e., is there already a way we could enforce this by e.g. locking a tool from writing to NFS until they clean up?) [13:31:12] probably just a process checking every now and then [13:31:17] found this task, but frankly the reasoning for not doing it seems a bit contradictory to the idea... setting quotas would help in "using NFS less" T284967 [13:31:18] (until we have the quotas setup) [13:31:18] T284967: Implement filesystem quotas on NFS volumes that are multi-tenant - https://phabricator.wikimedia.org/T284967 [13:31:48] a more recent one: T439494 [13:31:48] T439494: Improving toolforge NFS patrol - https://phabricator.wikimedia.org/T439494 [13:31:51] another relevant task is T439494 [13:32:41] :D [13:32:58] I think reporting per-tool used space and get alerts based on that would be an excellent first step, hard quotas is harder IMHO for non-obvious benefits [13:33:10] I think that the main use case today that would not be sorted yet when using PVCs/k8s provided storage would be access to other tool homes [13:34:34] +1 to the reporting, even if we setup quotas we would still want reporting/alerts [13:35:12] indeed [13:35:22] did we have something like that? (it kinda rings a bell that we stopped because it was too stressing for NFS to have the scanning done) [13:42:05] aputhin: lol great timing [13:54:31] xd, yep, I was not sure if stashbot just malfunctioned or what [13:58:19] in T439494 I has suggested some alternatives, I think that having duc run once a week could be a decent starting point, will not catch a sudden rise in writes but would be good for natural growth [13:58:20] T439494: Improving toolforge NFS patrol - https://phabricator.wikimedia.org/T439494 [14:11:55] is https://wikitech.wikimedia.org/wiki/Incidents/2026-10-21_PHP_security_update/Community#Availability_of_Wikimedia_services up to date for toolforge? (I got pinged by eric asking some questions and I pointed to that page) [14:15:17] the 'Updates and guidance' section is missing some later blog posts at least [14:15:38] but the general status seems accurate, the infra is up and all things that remain down are pending for the individual maintainers to update things or reply to our emails [14:15:50] ack, exactly what I said [14:24:08] yep, we did and we did. But I also don't think we tried very hard to make it work properly when it misbehaved. [14:24:27] oh, ^ was regarding nfs usage monitoring [16:42:33] volans: just updated the toolforge section [16:47:49] <3 [20:11:07] volans|off: I have attempted to silence the heat-api alerts. [22:04:43] any roots around who can help me with a wmcs root key change? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363482