[18:34:13] <Skynet>	 tools-static.wmflabs.org is 504ing
[18:40:35] <bd808>	 andrewbogott: ^ I think something may be hammering tools NFS
[18:41:50] <bd808>	 hmmm.. load on tools-nfs-2.tools.eqiad1.wikimedia.cloud is nominal
[18:45:42] <andrewbogott>	 iotop shows intermittent spikes although not more than it should be able to handle...
[18:50:46] <bd808>	 I thought I had an "ah ha!" with the gob of "(101: Network is unreachable)" errors in the nginx log on tools-static-15, but then I remembered that is the Nginx IPv6 address resolution bug (the host is IPv4 only, but nginx still does an AAAA lookup for the upstream proxy)
[18:52:52] <andrewbogott>	 Looks like tools-static-15 can see NFS just fine...
[18:52:56] <andrewbogott>	 so is it just that the service died?
[18:53:13] * andrewbogott tempted to reboot
[18:53:44] <bd808>	 load is only 2... maybe jsut restart nginx?
[18:54:20] <andrewbogott>	 done
[18:54:46] <andrewbogott>	 access.log looks busy and happy
[18:55:19] <andrewbogott>	 error.log is full of complaints about not reaching proxy-03.project-proxy.eqiad1.wikimedia.cloud
[18:55:55] <bd808>	 "[2607:f8b0:4004:c09::5f]:443" is the unreachable thing in those log lines
[18:56:09] <andrewbogott>	 Oh right, that was the red herring you mentioned
[18:56:10] <bd808>	 let me find the phab task (its upstream mess)
[18:56:18] <andrewbogott>	 OK, so... is the service actually back?
[18:56:23] <andrewbogott>	 and I'm chasing shadows?
[18:56:56] <bd808>	 yeah, I think it is working as expected now. https://tools-static.wmflabs.org/jenkins-build-stats/2025/01/2025-01-29/30-day/
[18:57:10] <andrewbogott>	 cool
[18:57:51] <bd808>	 T212871 is the IPv6 bug
[18:57:52] <stashbot>	 T212871: tools-static tries to use ipv6 when proxying to downstream hosts - https://phabricator.wikimedia.org/T212871
[19:18:03] <Skynet>	 Can confirm
[19:18:09] <Skynet>	 It's working now
[20:55:36] <wm-bot>	 !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed 223cf553fb (l10n updates: ar)
[20:55:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL
[20:55:55] <wm-bb>	 <lucaswerkmeister> o_O first time I see “completed” as a pod status in kubectl get pods
[20:56:46] <wm-bb>	 <lucaswerkmeister> hm, in the describe output it says State: Terminated and below that (indented) Reason: Completed
[20:56:53] <wm-bb>	 <lucaswerkmeister> did we do a k8s upgrade recently 🤔
[23:51:47] <andrewbogott>	 lucaswerkmeister: not that recently, we're due for another upgrade