[14:56:41] <halfak>	 Hey average, you around?
[14:57:38] <average>	 yes Mr. Halfak
[14:57:48] <average>	 how may I help you ?
[14:57:55] <halfak>	 Do you have access to s5-analytics and would you be willing to try out a query for me?
[14:58:11] <average>	 can you please write the full name of that machine ?
[14:58:32] <mr_halfak>	 :)
[14:58:49] <mr_halfak>	 The alias is s5-analytics-slave.eqiad.wmnet
[14:58:55] <average>	 thanks, trying to log in now
[14:59:34] <average>	 I get permission denied on that host
[14:59:52] <halfak>	 OK.  No worries.
[15:00:03] <halfak>	 I'll try to get one of the DBAs.  Thanks for trying. :)
[15:00:09] <average>	 cool
[15:16:24] <average>	 halfak: I might be able to help you with that query
[15:16:44] <halfak>	 Oh yeah?
[15:16:52] <average>	 halfak: do you still need it ?
[15:17:58] <halfak>	 Sure.  I just want *any* query to return results on the machine.
[15:18:04] <qchris>	 average, halfak: Note that the s5 slave is currently lagging ~1/2 an hour. Be easy on it :-)
[15:18:28] <halfak>	 OK that might be relevant to the problems I'm having.
[15:18:35] <halfak>	 qchris, how do you check?
[15:19:23] <qchris>	 Very dumb... as I lack real access... I check the recent changes for wikis that are expected to see lots of updates.
[15:20:02] <halfak>	 so like "SELECT max(rc_timestamp) FROM recentchanges"?
[15:20:26] <qchris>	 halfak: Yes.
[15:20:35] <qchris>	 That worked reliably up to now.
[15:20:53] <qchris>	 If that is too distanc from now, I go check ganglia for the real lag.
[15:21:40] <average>	 I am connected to the s5-analytics-slave mysql
[15:21:42] <average>	 uhm
[15:21:53] <halfak>	 Gotcha.  That's my strategy too.  It's clunky though.  We should really have notifications sent to a mailing or maybe to this channel when replag gets high.
[15:21:58] <average>	 qchris: I can run queries with limit to avoid hitting it
[15:22:33] <halfak>	 I'd love to get a ping the next time that I accidentally lock up replication.
[15:25:40] <qchris>	 halfak: ping from whom? and ping once which threshold of replication lag is passed?
[15:30:31] <halfak>	 Good question.  Preferably a ping from a script throttled at once/hour or more.  I'd like to start with something like a 10 minute replag and see how often we get pinged.
[16:22:22] <ottomata>	 oook hi average, i'm making some coffee but can help
[16:40:28] <ottomata>	 great, Snaps
[16:40:31] <ottomata>	 null will work just fine
[17:44:45] <Snaps>	 ottomata: OKAY, Ill push it
[18:50:48] <qchris>	 halfak: Sorry. I missed your response. I have a system that runs twice a day and checks for 10 minute replag.
[18:51:16] <qchris>	 That was like 1 fail every two weeks. (Estimated)
[18:52:02] <qchris>	 If you want something reliable ... Icinga comes into mind.
[18:52:28] <qchris>	 Icinga does lag checks IIRC. Did you ask ops to include you in the Icinga alerts?
[19:56:41] <halfak>	 qchris: I haven't heard of Icinga.  Is there anything you could link me to?
[19:57:09] <qchris>	 https://icinga.wikimedia.org/icinga
[19:57:22] <qchris>	 But that requires authorization? ... mhmm
[19:57:43] <qchris>	 Last time I checked ... it worked without IIRC
[19:58:15] <qchris>	 Either ask in the ops channel, or ping ottomata when he comes online again.
[19:58:18] <halfak>	 No worries.  I'll go bug people in #-labs when I get a minute.  Thanks!
[19:58:37] <qchris>	 Ok.