[11:10:31] https://versions.toolforge.org/ seems to be unresponsive (reported by hashar in -operations) [11:10:55] other tools are fine AFAICT [11:11:22] I can’t see anything special in the tool, nothing recent in error.log and nothing in kubectl logs [11:16:56] I wonder if the recent phabricator improvements could’ve broken something that the versions tool relied on [11:17:09] but OTOH I’m pretty sure the tool was still working yesterday [11:27:13] there are two alerts about puppet failures on tools-k8s-worker-nfs-16, but the versions pod is running on tools-k8s-worker-nfs-77 AFAICT [11:28:57] !log lucaswerkmeister@tools-bastion-15 tools.versions webservice restart # trying to fix unresponsive tool; seems to have had no effect at the k8s level AFAICT [11:28:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.versions/SAL [11:29:29] the deployment is apparently 5y152d old btw… probably could use a hard restart (webservice stop + webservice start) [11:29:42] but I’ll leave it up for a bit longer in case someone else wants to take a look [11:40:05] I don't see it stuck on nfs [11:40:06] https://grafana-rw.wmcloud.org/d/RFhIBshHz/global-tools-stats?orgId=1&from=now-6h&to=now&timezone=browser [11:42:56] it uses a deprecated webservice image [11:43:26] I think that's probably the issue (so it won't recreate the webservice deployment) [11:44:07] probably should move to php7.4, I forced the recreation of the pod just deleting it [11:44:29] and that made it work :) [11:45:52] let me try real quick to run with 7.4 [11:46:07] okok, so far so good [11:46:46] it seems to be working with 7.4, a restart should work as usual now [21:09:10] looks like webservices had another wobble, 20:09 - 20:14 UTC [21:11:01] don't see a huge amount of sessions on the haproxy graphs, but a gap in requests (though backends say they where up)...