[07:23:23] greetings [08:04:20] today's riddle T441765 [08:04:20] T441765: Investigate periodic bots disconnections - https://phabricator.wikimedia.org/T441765 [08:10:59] do those times correspond to cloudgw or cloudnet failovers? [08:11:00] Morning! [08:11:21] wm-bot has a public ip per https://openstack-browser.toolforge.org/project/wm-bot, I wonder if we have bots running in cloud-vps (not toolforge) but without floating ips to compare against [08:19:36] taavi: yeah I don't know re: natted cloudvps bots, I'm fairly sure those are not cloudgw/cloudnet failover due to the ~5d "phase" [08:23:18] morning [08:27:54] I bet there are bots for every combination :D [09:35:17] taavi: for auth on eventgate that's the most I manage to get: https://phabricator.wikimedia.org/T441140#12451039 [10:35:50] volans: well, at least we have a path forward :D [10:36:02] I assume it's on us to write that auth support to eventgate? [10:36:27] I see andrew open to our other option too, so if we feel more to manage that in our own setup and just push to kafka once I think they will be ok [10:36:48] if we prefer to follow their option then yes, most likely the authn/z would be on us [10:39:25] if we can otherwise get that to eventgate then i think i slightly prefer that (as 'we' are already somewhat familiar with it) over coming up with a yet an another thing, i'd say [10:41:35] true, at teh cost of adding another "black box" step in the middle. But that's probably ok [10:41:50] I think the authn/z should not be a blocker to start having the data though [11:32:37] objections to me roll-restarting cloudvirtlocal ? [11:36:09] none here [11:38:10] ack, proceedign [11:39:20] hah wmcs.openstack.cloudvirt.safe_reboot is busted when not passing --exclude-kernel [11:43:21] which can't be used anyways on cloudvirtlocal heh [12:09:00] ok now https://grafana.wmcloud.org/d/8GiwHDL4k/infra-kubernetes-cluster-overview reports basic etcd health metrics [14:00:25] Good morning! [15:19:31] Hi godog or andrewbogott (it looks like you both relatively recently looked at T390228)! I’m looking at whether the advice at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Container_memory_limits is still current (for T441425 which is requesting 16GiB total). With the new worker nodes you added in that task, does 16GiB sound feasible now? [15:19:32] T390228: [toolforge] Rebuild Toolforge workers for bigger node sizes and in new VXLAN network - https://phabricator.wikimedia.org/T390228 [15:19:32] T441425: Request increased quota for oka-bot Toolforge tool - https://phabricator.wikimedia.org/T441425 [15:20:09] And do you know where I could check? T439512 suggests the grafana dashboard isn’t exact [15:20:10] T439512: [o11y,k8s,infra] Current stats used for capacity are not reliable enough - https://phabricator.wikimedia.org/T439512 [15:20:28] thilp: no, 16G will never schedule in practice [15:22:05] I don't recall if it was the WMCS sync or a tools-infra internal one where I said that someone(TM) needs to make a product decision on which container sizes we support (and then we need to (re)build workers to support those sizes). until that happens, I would be willing to say that we are not granting quota requests to bump the per-container limits [15:22:28] taavi: we have 32G nodes now so it's /possible/ that one would schedule although I agree that it would have to get very lucky. But, tangentially, we should decide if we're following through on replacing all worker nodes with the new 24/30 size [15:22:42] Let's talk about this on Thursday. I bet we can just do it. [15:22:55] we can just do what? [15:23:57] replace the remaining smaller workers with the larger size [15:24:08] probably yeah [15:24:52] https://grafana.wmcloud.org/d/8GiwHDL4k/infra-kubernetes-cluster-overview?from=now-30d&to=now&timezone=utc&var-cluster=P8433460076D33992&viewPanel=panel-38 is the graph on what one could currently schedule memory-wise. it seems pretty clear that 16G is not currently what we can support [15:27:23] andrewbogott: did we ever look into optimising the CPU/RAM ratio of workers based on actual usage? [15:28:16] thilp: T441425 not seem to be asking for a bump for the per-container limit, rather it's asking for a bump to the total memory usage across containers? which is fine to bump [15:28:16] T441425: Request increased quota for oka-bot Toolforge tool - https://phabricator.wikimedia.org/T441425 [15:29:59] I believe you’re right! Thanks for pointing it out! [15:30:01] taavi: The new 24/30 size is based on current usage but we would probably benefit from a closer look before we commit. [16:54:22] * dhinus off