[03:44:04] <halfak>	 ORES is acting up?
[03:44:18] <halfak>	 Maybe.  Checking on it.  Looks like there are a lot of timeouts happening. 
[03:44:21] <halfak>	 o/ yuvipanda 
[03:44:33] <halfak>	 Know if anything weird is going on with labs?
[03:44:54] <yuvipanda>	 uh
[03:44:54] <halfak>	 Usually when I see 2 minute downtime alerts, it's because of labs having a hiccup. 
[03:44:56] <yuvipanda>	 no
[03:45:01] <halfak>	 But, this time, it's too often. 
[03:45:02] <yuvipanda>	 I've mostly managed to be unaware of anything
[03:45:06] <halfak>	 kk
[03:45:08] <yuvipanda>	 in the last few weeks
[03:45:10] * halfak digs into workers
[03:45:18] <yuvipanda>	 am going to head off again now too, unfortunately :(
[03:45:19] <yuvipanda>	 sorry!
[03:45:35] * yuvipanda is back working from tuesday - friday again
[03:45:35] <halfak>	 No worries. 
[03:45:43] <halfak>	 I got this. 
[03:45:46] <halfak>	 :) 
[03:45:59] <halfak>	 Thanks for responding.  Enjoy vacation :D
[03:47:18] <halfak>	 Hmm... Looks like we are getting ~twice as many requests per minute since last night. 
[03:48:24] <halfak>	 It doesn't look like we're overloaded. 
[03:50:06] <halfak>	 Yeah.  Queue is good. 
[03:50:14] <halfak>	 Even cached requests are responding slowly. 
[03:51:30] <halfak>	 Load balancer isn't overloaded.
[03:51:46] <halfak>	 Web-01 looks OK
[03:52:40] <halfak>	 We'got all of our uwsgi processes running.  Neither of the webnodes are overloaded
[03:52:44] <halfak>	 Weird!
[03:56:48] <halfak>	 OK!  Now I'm getting somewhere. 
[03:57:01] <halfak>	 Looks like web-01 responds really slow, but web-02 responds fast
[03:57:25] * halfak checks the log
[03:57:40] <halfak>	 Nothing fancy
[04:00:01] <halfak>	 Ok.  Restarting service. 
[04:00:10] <halfak>	 Already told morebots
[04:03:42] <halfak>	 Woah.  Now -02 is slow
[06:17:44] <halfak>	 Hmm... Still not back to normal. 
[06:18:06] <halfak>	 Checking out worker-04, it seems that only one worker process was running (at 100%)
[06:22:31] <halfak>	 Restarted workers on -04 and -02 
[06:25:12] * halfak watches precached for a bit
[10:27:24] <Amir1>	 yuvipanda: hey, around?
[16:21:16] <halfak>	 Arg.  Looks like we're intermittently slow again
[16:22:02] <halfak>	 Weird.  Responses are *fast(
[16:23:15] <halfak>	 Looks like the celery workers are down on worker-02
[16:23:25] <halfak>	 We get into this state where there is one worker using 100% CPU
[16:28:27] <halfak>	 Looks like ores-worker-03 flatlined at 24% CPU 24 hours ago
[16:28:34] <halfak>	 I must have missed it in my restarts last night. 
[16:29:51] <halfak>	 Yeah.  worker-04 looks to have been the same way. 
[16:30:02] <halfak>	 Weird since the top of -04 looked normal. 
[16:31:09] <halfak>	 Oh wait.   I misread.  -04 looks like it went back to normal with last night's reboots. 
[16:31:30] <halfak>	 precached suggests that we've solved something. 
[16:32:04] <halfak>	 OK.  Back day-off. 
[16:32:05] <halfak>	 o/