[08:55:49] * arturo online
[09:10:49] <dcaro>	 morning!
[09:11:50] <arturo>	 o/
[09:13:22] <dcaro>	 welcome back :)
[09:24:58] <arturo>	 thanks!
[09:28:35] <dcaro>	 hmm... a curl to tools.wmflabs.org takes ~8s from my machine, but 0.05 from itself, something in between is slowing it down
[09:37:58] <dcaro>	 hmm, same from within cloudvps (tools-proxy-8), so it's not my network
[09:38:22] <arturo>	 the VM or the redirect setup is somehow overloaded
[09:38:40] <dcaro>	 the vm seems idle
[09:38:44] <dcaro>	 (idleish)
[09:39:11] <arturo>	 oh, ok
[09:39:37] <dcaro>	 <20%cpu, <500M ram (out of 4G)
[09:57:34] <dcaro>	 what's the path for the traffic between two vms? (tools-proxy-8 -> tools-legacy-redirector-2) 
[09:58:30] <arturo>	 I don't think tools-proxy-8 is in the path for tools.wmflabs.org but I could be wrong
[09:59:03] <dcaro>	 I mean because I did a curl from there
[09:59:12] <dcaro>	 and it was still slow
[09:59:40] <dcaro>	 (from tools-proxy-8 to tools-legacy-redirector-2)
[10:00:15] <arturo>	 tools.wmflabs.org goes to 185.15.56.60 which is a floating IP attached to tools-legacy-redirector-2
[10:00:32] <dcaro>	 wait, it's ssl
[10:00:46] <dcaro>	 from localhost, trying to do https is as slow
[10:01:01] <dcaro>	 hmm, is entropy still a thing?
[10:01:59] <arturo>	 one potential improvement would be to get rid of the HTTP->HTTPS upgrade, and resolve to toolforge.org directly, thus removing one hit in the box
[10:02:04] <arturo>	 so, from
[10:02:40] <arturo>	 current http://tools.wmflabs.org/sometool -> https://tools.wmflabs.org/sometool -> https://sometool.toolforge.org
[10:02:59] <arturo>	 to http://tools.wmflabs.org/sometool -> https://sometool.toolforge.org
[10:03:20] <dcaro>	 that should help some I guess, not sure the % of traffic that comes from http
[10:03:45] <arturo>	 last time I checked, it was significant http traffic
[10:04:45] <dcaro>	 it's actually most of it yep
[10:05:23] <arturo>	 that means each hit is actually 2 hits in the box
[10:16:44] <dcaro>	 I think not all get redirected though
[11:52:21] * dcaro lunch
[16:24:06] <arturo>	 andrewbogott: there was a legit alert about this today https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Users_not_in_bastion_project the fix was well documented, and the root cause too (my human mistake) so we are all good
[16:45:04] * arturo offline
[16:45:28] <dcaro>	 might be something to automate (iirc there's a cookbook to add a user to a project already)
[16:46:26] <arturo>	 yeah, but I did it manually, which is known to be a cause of failure :-(
[20:44:18] <andrewbogott>	 It ought to happen automatically, keystone should prevent there from ever being a non-bastion user in another project. So there's probably something in the keystone logs.