[01:28:11] <wm-bb>	 <catib8122> https://www.facebook.com/160982553987911/posts/4786081928144594/?mibextid=rS40aB7S9Ucbxw6v
[01:33:51] <wm-bb>	 <catib8122> https://www.facebook.com/WWW.TECHYBLAZE.C0M?mibextid=rS40aB7S9Ucbxw6v
[01:54:32] <wm-bb>	 <catib8122> Elon Musk reacts to 1952 German manuscript connecting his name to an uncanny prediction about Mars
[01:54:34] <wm-bb>	 <catib8122> https://www.uniladtech.com/science/space/elon-musk-reacts-to-1952-german-manuscript-connecting-him-to-mars-517196-20250127
[13:16:46] <arturo>	 !log admin manual failover of cloudgw1004 to cloudgw1003 T382356
[13:16:56] <andrewbogott>	 !log admin test log
[13:26:40] <arturo>	 !log admin manual failover of cloudgw1004 to cloudgw1003 T382356
[13:26:48] <arturo>	 !log admin manual failover of cloudgw1004 to cloudgw1003 T382356
[13:26:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[13:26:55] <stashbot>	 T382356: replace cloudgw100[12] with spare 'second region' dev servers cloudnet100[78]-dev - https://phabricator.wikimedia.org/T382356
[15:54:59] <bd808>	 !log catalyst-dev Promoted jnuche from reader to member
[15:55:01] <stashbot>	 bd808: Unknown project "catalyst-dev"
[15:56:00] <bd808>	 ugh. I bet that is the UUID bug that I haven't merged the patch for yet...
[15:56:29] <bd808>	 yup
[15:56:50] <bd808>	 !log 7209100e0e744a4fbdf447534d4eb825 Promoted jnuche from reader to member
[15:56:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:7209100e0e744a4fbdf447534d4eb825/SAL
[15:57:09] <bd808>	 !log 7209100e0e744a4fbdf447534d4eb825 Promoted thcipriani from reader to member
[15:57:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:7209100e0e744a4fbdf447534d4eb825/SAL
[15:57:19] <arturo>	 :-S
[17:18:05] <dancy>	 Hello cloud folks! I have a fuzzy question.  I see that there's a DNS name  `puppet` accessible from any project which maps to 172.16.7.124.  What service is listening on that address and what does it do?   Does it have magic that redirects to the project-local puppetmaster?  If so, what happens if there are two VMs with the role::puppetserver::cloud_vps_project role (as is the case in `devtools` at the moment).
[17:20:06] <andrewbogott>	 dancy: we run a cloud-wide puppet server for most projects that don't need a bespoke server. That dns entry will always refer to that server.
[17:20:48] <andrewbogott>	 So, nothing magic or complicated.
[17:27:40] <dancy>	 Ah ok.. so for a project w/ a custom puppetmaster, using "puppet" as the name of the puppet server is incorrect.
[17:28:34] <bd808>	 something like {project}-puppetserver01 would be rcommended
[17:29:11] <bd808>	 If you put the FQDN of your puppetserver in the `puppetmaster` hiera setting that should work too though
[17:29:33] <bd808>	 https://wikitech.wikimedia.org/wiki/Help:Project_puppetserver#Step_2:_Setup_a_puppet_client
[17:29:38] <mutante>	 the "puppet" thing used to work (in prod) but then things changed with puppetmaster-puppetserver, afaict
[17:29:45] <mutante>	 this might have affected cloud too
[17:29:59] <andrewbogott>	 dancy: hm if you call it 'puppet' I'm not sure what happens -- I suspect that it'll pick your local one by preference but I wouldn't recommend counting on it.
[17:30:16] <mutante>	 dancy: i think this is "used to be a thing but not anymore" roughly
[17:31:05] <andrewbogott>	 even if you did have a local server named 'puppet' the first run of the VM will still use the central puppetserver since resolv.conf is likely not set up until after that first run
[17:31:20] <dancy>	 Gotcha.  Thanks everyone.  I was debugging failed puppet run emails from devtools and  I found that puppetmaster-1003 had `puppetmaster: puppet` in its puppet config 
[17:31:30] <mutante>	 its probably there because production had that 
[17:31:47] <dancy>	 puppetmaster-1004 also has the bad setting.
[17:32:03] <bd808>	 dancy: that would be typical of a project puppetserver that itself gets config from the central puppetservers
[17:32:14] <andrewbogott>	 dancy: that's on purpose though. You don't want a puppetserver to manage itself in most cases.
[17:32:14] <dancy>	 I fixed puppetmaster-1003 and refreshed the CA and re-signed all client certs.
[17:32:35] <andrewbogott>	 So typically there's a cascade of control. Central puppetserver manages project-local puppetserver, which manages other VMs in the project.
[17:32:51] <dancy>	 ah, interesting.  I may need to make some tweaks then.
[17:33:01] <andrewbogott>	 If you have a project-wide override of puppetmaster: then you need to override /that/ for the puppetserver itself in VM-specific config
[17:33:33] <mutante>	 dancy: oh, you are trying to fix the puppet runs in devtools.. yea.. that is a a deep rabbit hole
[17:33:39] <mutante>	 I have tried that before
[17:33:50] <mutante>	 and it also had to do with rebuilding the puppetmaster
[17:33:54] <mutante>	 and expired certs
[17:34:12] <mutante>	 once I gave up on that I created a new local puppetmaster and ran into another issue
[17:34:41] <mutante>	 dancy: if you REALLy want to.. first let's see which instance is even using the local puppetmaster and which is not
[17:35:11] <mutante>	 then lets question if there is a reason to use the local master at all for this specific one
[17:35:26] <dancy>	 https://www.irccloud.com/pastebin/18yfXCbR/
[17:35:26] <mutante>	 since the easiest fix is to not use a local one
[17:35:40] <dancy>	 I have no idea why a local puppetmaster is used in this project.  
[17:35:46] <mutante>	 yea, every single instance is a bit of a different story
[17:36:03] <dancy>	 rm -r /devtools?!
[17:36:12] <mutante>	 presumably to have local secrets
[17:36:20] <dancy>	 ah, that makes sense.
[17:36:26] <mutante>	 are you interested in a specific instance more than another ?
[17:36:39] <mutante>	 gitlab-runners ?
[17:37:01] <mutante>	 how about you just try to fix gitlab-runner test instance and ignore the rest and leave those for me, heh
[17:37:05] <dancy>	 My goal was to get rid of the emails that I was getting every day about failed puppet runs.  I think I've now eliminated all of them except for puppetmaster-1004.
[17:37:26] <mutante>	 you actively fixed some puppet runs?
[17:37:32] <mutante>	 or just deleted the mails, heh
[17:37:40] <mutante>	 I also get those and yea, thanks for looking !
[17:37:54] <mutante>	 I have an open ticket for that and thought nobody else cares
[17:38:11] <dancy>	 I fixed the CA state and resigned the client certs.
[17:38:15] <mutante>	 and that puppetmaster setup was a bit frustrating
[17:38:22] <mutante>	 oooh, hell yea:)
[17:38:36] <mutante>	 is it still working now? because at some point I also had it fixed
[17:38:39] <mutante>	 and later it was broken again
[17:38:44] <mutante>	 checks:) yay
[17:38:52] <dancy>	 I just got it cleaned up today so we'll see tomorrow I think.
[17:38:59] <mutante>	 thanks a lot!
[17:39:08] <dancy>	 run-puppet-agent works twice in a row on the previously offending nodes
[17:39:17] <mutante>	 ok, I will keep an eye on those mails. 
[17:39:20] <mutante>	 I appreciate this
[17:39:43] <dancy>	 Glad someone other than me will feel benefit from it!
[17:40:07] <mutante>	 to clarify, the fix was all about the CA on the local master/server, right
[17:40:12] <andrewbogott>	 thx dancy, sorry for the confusing setup
[17:40:22] <mutante>	 no "change which master an instance or the project uses"
[17:40:37] <andrewbogott>	 feel free to update the docs if you feel adequately enlightened :)
[17:40:51] <mutante>	 because then I will still discuss with others if we can stop using the local master 
[17:41:06] <mutante>	 we already reduced the number of instances using it at some point
[17:41:33] <dancy>	 mutante: Correct.  All on the CA/puppetserver side 
[17:41:33] <mutante>	 one thing is secrets for gitlab-runner but we could do that in a different way. already chatted with J.elto
[17:41:49] <dancy>	 Awesome
[17:41:52] <mutante>	 so if we all agree we dont really need the local master .. we will just remove iut
[17:42:16] <mutante>	 ok, sounds all like progress. tbc :)
[17:42:38] <dancy>	 Thanks again everyone for the info. That was helpful
[17:43:04] <mutante>	 dancy: you basically resolved https://phabricator.wikimedia.org/T382960
[17:44:30] <mutante>	 (not just rm -rf devtools, btw. having test instances still good and some use cases, even if not that many)
[17:44:49] <dancy>	 Nod.  I just was being silly.
[17:45:00] <mutante>	 ;) thanks again, and cu later
[19:57:36] <inflatador>	 is there an easy way to track down the horizon project for https://apt-browser.toolforge.org/ ? It's been down a couple of days. I pinged legoktm but he hasn't responded yet. Thought I might peek and and see if there was something obvious to fix
[19:59:31] <wm-bb>	 <lucaswerkmeister> if it’s toolforge (rather than cloud vps) then https://toolsadmin.wikimedia.org/tools/id/apt-browser should be correct
[20:02:20] <wm-bb>	 <lucaswerkmeister> I don’t know what’s wrong with it, the webservice seems to be running
[20:02:55] <wm-bb>	 <lucaswerkmeister> the kubectl logs show some /healthz responses, but no new ones when I attach with --follow, so who knows how old those /healthz entries in the logs are
[20:03:36] <wm-bb>	 <lucaswerkmeister> aha, `kubectl logs --timestamps` is what I want. the /healthz responses are from two days ago
[20:03:46] <wm-bb>	 <lucaswerkmeister> ah, the pod is in “killing” status
[20:03:55] <wm-bb>	 <lucaswerkmeister> but I guess something in the k8s side is stuck
[20:04:05] <mutante>	 inflatador: if it's a toolforge tool, it should appear in https://hay.toolforge.org/directory/  | if it's a cloud vps project, it should appear in https://openstack-browser.toolforge.org/project/ 
[20:04:21] <mutante>	 the toolforge.org URL strongly implies it's the first
[20:04:36] <wm-bb>	 <lucaswerkmeister> /me peeks at tools-k8s-worker-nfs-37
[20:04:37] <mutante>	 though I guess technically you could click a proxy for that in cloud vps 
[20:05:21] <mutante>	 also: the regular wikitech wiki search. if it's a project then there are always some auto-generated pages about it
[20:05:45] <wm-bb>	 <lucaswerkmeister> I see a handful of processes in D state, might be the usual NFS stuff?
[20:05:49] <wm-bb>	 <lucaswerkmeister> though not *that* many processes tbh
[20:06:19] <wm-bb>	 <lucaswerkmeister> but one of the processes is from apt-browser (`tee /data/project/apt-browser/logs/rocket.log`)
[20:07:25] <inflatador>	 mutante ah, thanks for that directory link! And lucaswerkmeister thanks for checking k8s
[20:07:41] <wm-bb>	 <lucaswerkmeister> I tried sudo kill -9’ing it but it doesn’t seem to have done anything (which is not surprising if it is indeed stuck in NFS)
[20:07:53] <wm-bb>	 <lucaswerkmeister> let’s see if I can find the docs for what to do with a worker in this state
[20:08:00] <wm-bb>	 <lucaswerkmeister> though IIRC I don’t have enough permissions to do the needful
[20:09:01] <wm-bb>	 <lucaswerkmeister> yeah, I can’t run cookbooks https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses
[20:09:25] <wm-bb>	 <lucaswerkmeister> (though an `ls` still worked, so the server isn’t totally dead… not sure what the right thing to do then is)
[20:09:45] <inflatador>	 How'd you figure out which worker the service 'lives' on? Or even that it was hosted in toolforge k8s?
[20:09:56] <inflatador>	 (sorry, new to this whole thing)
[20:10:07] <wm-bb>	 <lucaswerkmeister> I ran `sudo become apt-browser` with my toolforge root powers ;)
[20:10:15] <wm-bb>	 <lucaswerkmeister> and then kubectl get pods, kubectl get deployments, etc.
[20:10:28] <wm-bb>	 <lucaswerkmeister> the worker is in `kubectl get pod apt-browser-85d46f5946-c6ptd -o yaml | less` buried in the output ksomewhere
[20:10:57] <wm-bb>	 <lucaswerkmeister> though you can also see that particular detail without privileges at https://k8s-status.toolforge.org/namespaces/tool-apt-browser/pods/apt-browser-85d46f5946-c6ptd/
[20:11:21] <wm-bb>	 <lucaswerkmeister> and I think any toolforge member can SSH into the toolforge nodes in theory (they just can’t do very much in them, except stare at the “no user-serviceable parts inside” banner that gets printed on login ^^)
[20:11:54] <wm-bb>	 <lucaswerkmeister> `tools-k8s-worker-nfs-37.tools.eqiad1.wikimedia.cloud` is the full hostname for SSH purposes
[20:11:58] <arturo>	 can I help?
[20:12:08] <arturo>	 I'm briefly online
[20:12:28] <wm-bb>	 <lucaswerkmeister> you probably can
[20:12:34] <wm-bb>	 <lucaswerkmeister> *some* processes in tools-k8s-worker-nfs-37 are stuck in D state
[20:12:41] <wm-bb>	 <lucaswerkmeister> though others still seem to be running happily
[20:13:08] <wm-bb>	 <lucaswerkmeister> `apt-browser-85d46f5946-c6ptd` pod in the apt-browser tool if you want to look at it from the k8s side
[20:13:28] <wm-bb>	 <lucaswerkmeister> or PID 3029 on that worker
[20:13:47] <arturo>	 ok, thanks for the pointer, let me see
[20:14:19] <wm-bb>	 <lucaswerkmeister> thanks
[20:14:47] <arturo>	 it wont let me ssh
[20:15:20] <inflatador>	 did y'all poke anything? It appears to be back up
[20:15:37] <wm-bb>	 <lucaswerkmeister> huh, I’m inside… load average 7/9/10 which is a bit high but shouldn’t prevent ssh
[20:15:47] <wm-bb>	 <lucaswerkmeister> but `ps aux | awk '$8=="D"'` output suddenly became empty
[20:16:25] <wm-bb>	 <lucaswerkmeister> I didn’t do anything that should have an effect since that kill -9 I mentioned earlier
[20:16:51] <wm-bb>	 <lucaswerkmeister> I guess it went through after all, the process seems to be gone now
[20:17:00] <wm-bb>	 <lucaswerkmeister> at which point k8s restarted the pod
[20:17:04] <wm-bb>	 <lucaswerkmeister> so… yay?
[20:17:32] <inflatador>	 {◕ ◡ ◕}
[20:17:43] <inflatador>	 You get a "yay" from me anyway ;)
[20:17:46] <wm-bb>	 <lucaswerkmeister> wondering if I should !log any of this ^^
[20:18:38] <arturo>	 I did nothing
[20:18:45] <arturo>	 I think the VM had high memory usage
[20:18:55] <arturo>	 and the OOMkiller triggered
[20:19:03] <wm-bb>	 <lucaswerkmeister> ah, okay
[20:19:05] <arturo>	 at the same time, the VM network is flapping
[20:19:28] <arturo>	 https://www.irccloud.com/pastebin/XaXcn253/
[20:19:52] <inflatador>	 Flapping network + NFS? Sounds "fun" ;P
[20:20:24] <arturo>	 yeah, I'll reboot the VM anyway, just to clear to ugly kernel errors
[20:20:38] <wm-bb>	 <lucaswerkmeister> load avg is back to 1/4/7 now btw
[20:20:45] <wm-bb>	 <lucaswerkmeister> but yeah ok ^^
[20:21:00] <wm-bb>	 <lucaswerkmeister> (and 11Gi available according to free -h)
[20:21:04] <arturo>	 @lucaswerkmeister load avg is expected to go high if IO is somehow blocked
[20:27:44] * arturo offline again
[20:27:50] <wm-bb>	 <lucaswerkmeister> thanks!
[21:26:29] <bd808>	 inflatador, mutante: Hay's Directory is a partial listing of some tools that have very explicitly opted-in to being listed there. https://toolhub.wikimedia.org/ has everything that Hay's has plus much more. 
[21:27:05] <bd808>	 https://toolsadmin.wikimedia.org/tools/ is the canonical Toolforge resource because it includes all Toolforge tools even if they have never had a toolinfo.json record created.
[21:34:35] * inflatador adds to notes