[04:13:27] Just checking to know if all is well on my side regarding toolforge membership requests. Is it normal for us to go days (6 days now) without getting a single toolforge membership request? [05:50:19] hey, I was pages by harbor 1h ago. It is resolved now, but writing this message here in case it went unnoticed [08:37:11] Raymond_Ndibe: I think it can be normal yes, for example I see a previous gap of 4 days 2024-07-30 and 2024-08-04. [08:44:42] a.rturo: thanks for reporting and sorry about the page! I have added an override in VictorOps until the end of the month. [11:43:49] cloudinfra-cloudvps-puppetserver-1 is down, looks like an OOM issue like for the prometheus hosts [11:44:37] I'm rebooting it from Horizon [11:50:43] "soft reboot" was enough, the server is back up [11:52:12] there's no trace of "oom" errors in the logs, but Grafana shows the available memory jumped from 5.5GB to 0.5GB, then to 0.1GB, then stopped reporting [11:56:34] I created T373092 for traceability [11:56:35] T373092: cloudinfra-cloudvps-puppetserver-1 became unresponsive - https://phabricator.wikimedia.org/T373092 [13:22:02] Is cumin1001.eqiad.wmnet going by a different name of late? [13:24:05] The documentation suggests that it is still the same name. Can anyone else not get into it, or is that just me? [13:24:28] it's now cumin1002 [13:24:43] Neat, I'll update wikitech [13:24:53] thanks, I've updated a few but many are still using the old name [13:36:22] Rook: I'm trying "deploy.sh" for superset, tofu wants to create local_file.kube_config (fine), but also openstack_containerinfra_cluster_v1.k8s_126_2 and openstack_containerinfra_cluster_v1.k8s_127 [13:36:25] is that expected? [13:36:39] "Plan: 5 to add, 0 to change, 0 to destroy." [13:37:36] no that is not...I thought I got rid of that, is your branch stale? [13:37:47] oh yeah possibly [13:37:54] I need to rebase [13:39:46] now I'm down to "Plan: 3 to add, 0 to change, 0 to destroy." [13:40:01] it still wants to create a cluster, but no longer two [13:40:28] hmm...Oh I don't know if superset has an object store [13:40:56] Things start in PAWS and migrate to other things. So it might not have got one yet... [13:41:02] maybe I should run it from your directory where you have the tfstate? [13:41:51] Yep nothing there. Yes please use my account, checkout whatever in the superset-deploy directory and run it there. Could you open a ticket to get the tofu state for superset in an object store? [13:42:06] ok I will create a phab :) [13:42:11] Thanks! [13:44:58] T373111 [13:44:58] T373111: Store tofu state in Object Storage - https://phabricator.wikimedia.org/T373111 [13:47:52] I can't seem to access bastion.tf-infra-dev.codfw1dev.wikimedia.cloud, I appear to be a reader and member in the tf-infra-dev project, and can get into another host in codfw, though I can't get into that one as myself or as root. Any thoughts on what I'm doing wrong? [13:49:35] I also cannot log in, andrewbogott recently upgraded openstack in codfw, so that might be related? [13:50:14] That could be. The node was just deployed today so maybe it isn't running puppet or otherwise finding ldap? [13:50:37] I guess puppet as root would work if it was only ldap not working? [13:54:11] I never remember the exact plumbing for auth [13:55:26] I think the keys for root are placed on the system in /etc/ by puppet, so they might not be getting there [13:57:28] I think it's dns [13:58:35] That makes sense too. Oh, I just named it the same as an old node, is that causing a problem? [13:58:57] yeah, dns says it's 172.16.128.44 but nova says it's 172.16.128.148 [13:59:05] It shouldn't be a problem to re-use hostnames but it's racey [13:59:12] I'll give it a new name and try again [13:59:16] Thanks! [15:46:38] Wikibugs posts tickets with the security tag? [15:49:41] Yeah, security tag alone doesn't mean anything [15:50:09] Just a vague "this task may have some non specific security related concerns" [16:11:01] Good to know. What tag keeps it from posting? [16:12:11] There isn't one specifically. I think it depends on the visibility policies, and whether it comes into the feed it uses