[14:11:57] !log tools.nyandata deleting old generated SMW tarballs from TarballDownloader/build to free up disk space. tool appears to be abandoned at least since grid shutdown. # T396220 [14:12:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.nyandata/SAL [14:12:00] T396220: 2025-06-06 Toolforge NFS cleanup - https://phabricator.wikimedia.org/T396220 [15:08:53] same! [17:58:12] hey, could you remind me which host was the cloud gateway host where I could look into the NAT table [17:58:30] arturo told me a couple weeks ago .. just lost the host name i think [18:36:00] is someone here who just used http://performance-testing-graphite.wmftest.org by any chance [18:54:52] mutante: what exactly are you trying to do? [18:56:51] those were unrelated things. one is I want to know who uses that wmftest.org URL above because it overloaded squid proxy on an install server [18:57:15] and the other is I want to know if 172.16.4.235 and 172.16.5.127 are toolforge IPs [18:57:26] that wmftest.org URL seems to be hosted at Hetzner, not at WMCS [18:57:30] because they made a bunch of requests to phabricator [18:57:46] and I am surprised I can see private IPs in my apache log [18:58:01] Hetzner, interesting. ok [18:58:28] I did find the cloudgw host, so you can ignore that part [18:58:32] 235.4.16.172.in-addr.arpa domain name pointer metricsinfra-alertmanager-3.metricsinfra.eqiad1.wikimedia.cloud. [18:58:32] 127.5.16.172.in-addr.arpa domain name pointer metricsinfra-alertmanager-2.metricsinfra.eqiad1.wikimedia.cloud. [18:58:47] so that seems related to T394446 [18:58:48] T394446: Consider setting up an https://github.com/knyar/phalerts instance in metricsinfra - https://phabricator.wikimedia.org/T394446 [18:59:03] aha, where did you look that up please? they did not resolve for me while on cloudgfw [18:59:06] cloudgw [18:59:15] thanks [18:59:34] any cloud vps hosted VM will work, I used a toolforge bastion by default [19:00:15] alright, thanks. that "phalerts" ticket is very useful [19:00:30] because it caused a "ph alert" :p [19:02:01] in general none of what you asked for has has anything to do with the cloudgws. so please please ask us for help, much safer than poking at things yourself and you'll usually get an answer to the actual problem much faster :-) [19:03:00] in particular our web proxy has proper access logs, if you're trying to figure out if someone's visiting some web service hosted by us that's much easier than looking at NAT mappings :-) [19:05:27] it does, because on cloudgw I can look into the NAT table to find out if those IPs appear there [19:05:50] looking at the NAT table is not poking at things [19:06:40] and I was told to do exactly that last time I asked the team [19:09:01] your answer was helpful and asking here is exactly what I did.. so no need for this type of discussion [19:20:31] I reject the notion that I was "poking at things" that seems to imply I did something inappropriate. Facts are I made zero changes to anything in your environment. I also disagree with the statement that "none of that has anything to do with the cloudgws" when in fact it answered the question if these IPs are in cloud. [20:59:23] !log admin restarting all designate services on all cloudcontrols in eqiad1 [20:59:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:21:59] andrewbogott: that magic trick worked yet again [21:29:32] it's the best trick [21:31:29] it's my best devops trick too, but I feel like "real SREs" look down on my for being quick to reboot and pray the problem disappears :) [21:36:46] the secret is that there are no real sres [21:40:55] !log tools restarting tools-prometheus-9 and tools-prometheus-8, lots of tools metrics just went dark [21:40:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL