[13:08:36] <wm-bb> <lucaswerkmeister> anyone familiar with wikibugs online? it’s stopped posting to -dev and -operations AFAICT [13:08:49] <wm-bb> <lucaswerkmeister> but I’m not sure which of its jobs I should restart, none of them seem obviously broken [13:09:03] <wm-bb> <lucaswerkmeister> (irc, gerrit and znc all have recent messages in `kubectl logs`) [13:11:41] <wm-bb> <lucaswerkmeister> if nobody’s around I’ll just try restarting some of them soonish and hope it helps [13:13:46] <wm-bb> <lucaswerkmeister> (I think I would first try restarting znc, there’s some suspicious “cannot send to nick/channel†in its output) [13:15:51] <wm-bot> !log lucaswerkmeister@tools-bastion-13 tools.wikibugs toolforge jobs restart znc [13:15:53] <stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [13:17:19] <wm-bot> !log lucaswerkmeister@tools-bastion-13 tools.wikibugs toolforge jobs restart irc [13:17:20] <stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [13:19:25] <wm-bb> <lucaswerkmeister> okay looks like the irc restart fixed wikibugs [13:19:33] <wm-bb> <lucaswerkmeister> (the znc restart had no effect that I could see) [13:45:24] <andrewbogott> Nemo_bis, can you respond to (or act on) https://phabricator.wikimedia.org/T367528#9945179 ? [17:18:53] <bd808> @lucaswerkmeister: thanks for tending to wikibugs. When you restarted ZNC the irc bot died and the new k8s spawned replacement was still initializing when you killed it too. Unfortunately I don't have any durable logging for the ZNC right now so I can't see what was up with it before the restart. [17:20:23] <wm-bb> <lucaswerkmeister> ah, damn :( [17:28:40] <bd808> @lucaswerkmeister: no big deal. you were doing the needful I'm sure. [17:28:48] <wm-bb> <lucaswerkmeister> I tried to ^^ [21:11:40] <jhathaway> I would like to run the update facts script on puppetmaster.cloudinfra.wmflabs.org, but I hit a timeout sshing to puppetmaster.cloudinfra.wmflabs.org [21:11:55] <jhathaway> I can't recall if I have been able to access the host before [21:23:34] <mutante> jhathaway: I found that the IP that name points to is bound to what is now called cloudinfra-cloudvps-puppetserver-1 [21:24:17] <mutante> jhathaway: try this name: [21:24:18] <mutante> cloudinfra-cloudvps-puppetserver-1.cloudinfra.eqiad1.wikimedia.cloud [21:24:54] <mutante> had to add it to config so it goes via the bastion but I was able to get on it as root [21:25:31] <mutante> I can confirm that the documented name just times out [21:28:49] <jhathaway> ah!, thanks mutante [21:32:50] <jhathaway> now my connection get refused, who controls access to the cloudinfra project? [21:34:15] <mutante> jhathaway: I feel like "syncing puppet facts" did not need ever need access to "cloudinfra" [21:34:37] <mutante> wasn't this a local puppetmaster in the actual "puppet-diffs" project? I could swear that was where we synced to [21:34:53] <mutante> or is this not about the puppet compilers? [21:35:09] <jhathaway> correct, this is about the puppet compilers [21:35:24] <mutante> but cloudinfra is a different project [21:35:38] <mutante> that's the central puppetmaster [21:35:52] <mutante> now I wonder if the local puppetmaster was removed [21:37:14] <jhathaway> good question, at present the hosts in the puppet-diffs projects are pointed to puppetmaster.cloudinfra.wmflabs.org [21:37:56] <jhathaway> whereas the hosts in the devtools project are pointed to puppetmaster-1003.devtools.eqiad1.wikimedia.cloud [21:38:16] <mutante> for devtools I can say that it depends on which instance it is [21:38:28] <mutante> but puppet-compiler is puppet-diffs [21:39:18] <jhathaway> stat says the puppet.conf file for pcc-db1001 was changed on 2024-04-09 [21:40:30] <mutante> reads https://wikitech.wikimedia.org/wiki/Server_Admin_Log/Archive_78#2024-04-09 [21:41:39] <mutante> well,, wrong SAL [21:42:04] <jhathaway> the genesis of problem was pcc failing here, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1054661?tab=checks [21:42:19] <jhathaway> complaining that facts did not exist for those hosts [21:42:39] <mutante> nothing in SAL of the project [21:42:40] <jhathaway> though they puppet clean, so puppetmaster.cloudinfra.wmflabs.org has their facts [21:43:22] <jhathaway> most definitely [21:52:08] <bd808> cloudinfra is the shared project for all Cloud VPS things. SAL for it is probably under "admin" typically -- https://sal.toolforge.org/admin