[00:28:17] <yuvipanda>	 bd808: how's the prod logstash stuff going along?
[00:28:35] <yuvipanda>	 (just curious)
[00:29:10] <bd808>	 good! I think I have one puppet patch that needs merging
[00:29:13] * bd808 looks
[00:29:46] <bd808>	 yuvipanda: Just this one left -- https://gerrit.wikimedia.org/r/#/c/205971/
[00:30:09] <bd808>	 I want to turn up group1 logging tomorrow
[00:30:30] <bd808>	 and then group2 on thursday if everything looks ok
[00:55:26] <yuvipanda>	 bd808: sweet
[00:56:12] <bd808>	 yuvipanda: if you wanted to merge that puppet patch that would be even sweeter ;)
[00:56:55] <yuvipanda>	 bd808: is it basically 'try not to use logstash on the old nodes'?
[00:57:28] <bd808>	 It turns off data storage for the elasticsearch on the old nodes
[00:57:54] <yuvipanda>	 right right
[00:57:55] <bd808>	 I have temp rules on the cluster to do that now but they won't survive a full cluster restart
[00:58:32] <yuvipanda>	 bd808: alright, you around to verify right now if I merge?
[00:58:57] <bd808>	 yeah
[00:59:07] <yuvipanda>	 bd808: doing
[01:02:33] <yuvipanda>	 bd808: you have root on the logstash hosts don't you?
[01:02:48] <bd808>	 Yeah. I can force puppet runs
[01:02:55] <bd808>	 if we are ready for that
[01:03:01] <yuvipanda>	 bd808: yeah, puppet merged
[01:03:14] <bd808>	 k. I'll start on 1001
[01:04:01] <bd808>	 config cahnges look right
[01:06:20] <yuvipanda>	 bd808: sweet :D
[01:06:29] <yuvipanda>	 bd808: let me know when I can leave laptop :)
[01:06:38] <bd808>	 ugh problem
[01:06:47] <bd808>	 Invalid initial heap size: -Xms1Gm
[01:07:17] <bd808>	 that's from logstash itself.
[01:07:21] <bd808>	 looking
[01:07:58] <yuvipanda>	 bd808: ok
[01:08:16] <bd808>	 I'll have a patch in 2 minutes. easy fix in hiera
[01:09:18] <yuvipanda>	 bd808: cool
[19:54:28] <legoktm>	 AaronSchulz: https://gerrit.wikimedia.org/r/209316
[19:55:56] <anomie>	 legoktm: Do we worry about maintaining compatibility with 1.24 and earlier in CentralAuth?
[19:56:12] <legoktm>	 anomie: nope, just master!
[22:09:53] <yuvipanda>	 bd808: how’s logstash going?
[22:10:11] <bd808>	 not broken yet
[22:11:08] <yuvipanda>	 bd808: :D
[22:11:22] <yuvipanda>	 bd808: how crazy would it be to setup a logstash / ES instance just on labs instances?
[22:11:30] <yuvipanda>	 like, for graphite it’s pretty crazy and a no-no
[22:11:35] <yuvipanda>	 would ES survive?
[22:15:21] <bd808>	 yuvipanda: It probably would. the beta instance works ok
[22:15:32] <bd808>	 disk space is the big thing
[22:32:37] <yuvipanda>	 bd808: right.
[22:32:46] <yuvipanda>	 bd808: so I guess it’ll need what, 3 nodes or something like that?
[22:32:57] <yuvipanda>	 bd808: also do you think you’ll have time to do the NFS logging on beta thing?
[22:37:34] <bd808>	 yuvipanda: off my conference call now. Got some time to work this out?
[22:37:43] <ori>	 *conpherence
[22:38:45] <bd808>	 yuvipanda: first question is how much log data do you want to keep I guess
[22:38:56] <yuvipanda>	 bd808: just went out for food :( I'll brb in about 20mins?
[22:39:15] <bd808>	 bah. I'll be back in another meeting then
[22:39:16] <yuvipanda>	 bd808: yup, and then why not keep it on the instance 
[22:40:31] <yuvipanda>	 bd808: I can chat now tho
[22:40:33] <yuvipanda>	 On IRC 
[22:40:48] <bd808>	 yuvipanda: I think the place to start for a tools logstash thing is just to build a 2-3 node cluster and then see how it scales
[22:41:11] <bd808>	 the nice thing about elasticsearch is that it is elastic
[22:41:20] <bd808>	 you can add more nodes pretty easily
[22:41:35] <yuvipanda>	 bd808: yeah. How hard to you think building a 3 node cluster will be?
[22:41:42] <yuvipanda>	 I'll restrict it to admins I guess 
[22:41:46] <bd808>	 a couple of hours
[22:41:53] <bd808>	 it's all in puppet
[22:42:21] <bd808>	 I have a project for logstash already actually that we could use
[22:42:45] <bd808>	 It probably all needs to be rebuilt from scratch
[22:43:28] <bd808>	 once we have a cluster up we can start adding log sources to it and see how things go
[22:44:19] <bd808>	 I've done some multi-tenant tricks with elasticsearch in a past life too that we can try out
[22:46:24] <bd808>	 basically you can create aliases that act as "views" by concating a constant where clause so that when you request from that alias you can't get records that don't match the alias
[22:46:32] <yuvipanda>	 bd808: ah sweet.
[22:46:54] <yuvipanda>	 bd808: right. and then have some way of authentication
[22:46:57] <bd808>	 scaling that to the number of tools + projects * daily indices may get tricky
[22:47:07] <bd808>	 but It should be possible
[22:47:12] <yuvipanda>	 bd808: wait, so stepping back
[22:47:17] <yuvipanda>	 are you suggesting one logstash for all of labs?
[22:47:22] <yuvipanda>	 with multi tenancy?
[22:47:24] <yuvipanda>	 I’m back in the office
[22:47:35] <yuvipanda>	 bd808: I can hop onto a hangout if you prefer that, or we can continue on IRC
[22:47:59] <bd808>	 I've got 15 minutes if you want to talk
[22:48:03] <bd808>	 might be a bit faster
[22:48:14] <yuvipanda>	 bd808: yep, moment
[22:53:56] <SMalyshev>	 anybody knows why it could be that test.wikidata.org does not have $.jStore defined (while wikidata.org does have it)?
[22:54:20] <SMalyshev>	 $.jStorage
[23:03:32] <yuvipanda>	 bd808: sorry, have a meeting with spage to start redoing docs for toollabs :|
[23:03:58] <bd808>	 yuvipanda: I've got a Reading planning meeting of some sort
[23:04:07] <bd808>	 we can talk tomorrow maybe
[23:04:09] <yuvipanda>	 bd808: :D Let me update that bug with list of things
[23:51:23] <wikibugs>	 10MediaWiki-Core-Team, 10Security-Reviews: kafkatee security review - https://phabricator.wikimedia.org/T75950#1267540 (10csteipp)