[17:14:15] <Matthew_>	 !ping
[17:14:15] <wm-bot>	 Pong.
[17:14:19] <Matthew_>	 Whoop
[17:15:12] <Matthew_>	 Related question: Is there a specific reason why I don't have admin on the bots project?  I remember you had one I just don't remember what it is.
[17:15:43] <petan>	 no idea I think you just never asked for it?
[17:15:49] <Matthew_>	 Okay.  May I have it? :)
[17:16:11] <petan>	 done
[17:16:54] <Matthew_>	 Thank you.
[17:17:14] <Matthew_>	 Looks good.
[17:18:39] <petan>	 LOG [7/27/2016 1:10:39 AM]    [INFO]: I was kicked from #Baj0 by Cepita because of: bot
[17:18:40] <petan>	 LOG [7/27/2016 7:29:12 AM] [WARNING]: SIGTERM - Shutting down
[17:19:00] <Matthew_>	 Oh yeah that one...
[17:19:06] <petan>	 1:10 AM is last action logged from network
[17:19:14] <petan>	 then reboot
[17:19:18] <petan>	 6 hours later
[17:19:20] <Matthew_>	 We had some spanish speakers that spent quite a bit abusing the bot.  That was one of their channels.  
[17:19:29] <wm-bot>	 Successfully parted channel: #baj0
[17:19:29] <Matthew_>	 @part #Baj0
[17:19:36] <petan>	 well that kick probably didn't cause problems
[17:19:48] <Matthew_>	 Yeah.  I just recognize the channel.
[17:19:55] <petan>	 it's just that's the last logged moment when the bot waqs still online
[17:20:13] <petan>	 the labs outage probably happened some time between
[17:20:23] <petan>	 I can check channel logs there should be a hole during that time
[17:20:39] <Matthew_>	 Okay.
[17:20:42] <petan>	 I was hoping for some errors that would explain what happened
[17:22:09] <Matthew_>	 Yeah...
[17:22:14] <Matthew_>	 Check for RSS feed errors as well.  
[17:22:41] <wikibugs>	 10WM-Bot: Bot is non-responsive - https://phabricator.wikimedia.org/T148420#2723295 (10Matthewrbowker) 05Open>03Resolved a:05Matthewrbowker>03Petrb Instance has been restarted (thanks @Krenair ).  @Petrb has restarted the bot.  It appears to be working.
[17:22:41] <wm-bot>	 Hey wikibugs, you are welcome!
[17:22:51] <petan>	 lol
[17:23:02] <Matthew_>	 lol
[17:23:34] <petan>	 wm-bot: :o?
[17:23:34] <wm-bot>	 Hi petan, there is an error, I'm a stupid bot and not intelligent enough to hold a conversation with you :-)
[17:23:59] <petan>	 that could change actually heheh I found some AI library that I wanted to incorporate into you
[17:24:37] <petan>	 anyway
[17:24:42] <petan>	 postgres logs have even bigger hole
[17:24:56] <Matthew_>	 Could the database have gone down?  
[17:25:17] <petan>	 so if there was a problem on labs db server was hit even before wm-bot's instance
[17:25:17] <Matthew_>	 (I can't access the logs on the huggle machine so I don't know)
[17:25:47] <petan>	 here we go
[17:25:55] <petan>	 that machine survived so I can see the logs :P
[17:26:00] <petan>	 NFS server failed on labs
[17:26:16] <Matthew_>	 That would explain it...
[17:26:21] <Matthew_>	 Ohhhhhh... what machine is it?
[17:26:30] <petan>	 labstore1003.eqiad.wmnet
[17:26:38] <petan>	 labstore.svc.eqiad.wmnet
[17:26:41] <petan>	 there is more of them
[17:27:27] <Matthew_>	 Never mind.  I saw https://phabricator.wikimedia.org/T148255 which is labvirt1005