[07:25:10] <average>	 damn keyboard
[16:00:07] <ottomata>	 milimetric: this is totally weird
[16:00:20] <milimetric>	 labs?
[16:00:20] <ottomata>	 the namenode crashes when I try to do hive things
[16:00:20] <ottomata>	 yeha
[16:00:33] <milimetric>	 hm
[16:00:53] <milimetric>	 improper java version?
[16:01:29] <ottomata>	 don't thikn so
[16:01:34] <ottomata>	 dunno its just dying with no output
[16:01:49] <average>	 can you restart the namenode ?
[16:02:05] <ottomata>	 yeah
[16:02:07] <ottomata>	 i can start it and use it fine
[16:02:15] <ottomata>	 and then I can start hive-server2 and hive-metastore
[16:02:24] <ottomata>	 but then when I try to do something with hive, the namenode crashes
[16:02:48] <average>	 stacktrace ?
[16:05:23] <ottomata>	 oof and these nodes are so slow
[16:05:52] <ottomata>	 hm, i should make a good vagrant instance for this, eh?
[16:05:54] <ottomata>	 i need a new one anyway
[16:05:57] <ottomata>	 and have been meaning to :p
[16:06:06] <ottomata>	 haven't done it because others hadn't needed it
[16:06:09] <ottomata>	 now you guys do!
[16:06:41] <average>	 well I can manage over here in my local setup
[16:06:51] <ottomata>	 aye
[16:07:04] <ottomata>	 i want one for me anyway, that uses the cdh4 module and the same setups
[16:07:10] <ottomata>	 not going to do it now
[16:07:14] <ottomata>	 but maybe in a few weeks or something
[16:07:30] <ottomata>	 i had a messy instance that i was using for it, but it is too messy now
[16:07:33] <ottomata>	 need somethign cleaner
[16:10:03] <ottomata>	 hmm you know, it isn't hive that is causing it to die
[16:10:06] <ottomata>	 it just dies after a while
[16:10:19] <ottomata>	 with no error messages
[16:10:25] <ottomata>	 i'm running in a foreground process
[16:10:26] <ottomata>	 and it just died
[16:11:08] <average>	 is there a way to increase log verbosity ?
[16:13:14] <average>	 ottomata: try to do a namenode recovery
[16:13:20] <average>	 ottomata: haddop namenode -recover
[16:14:34] <average>	 *hadoop
[16:14:44] <average>	 or hdfs namenode -recover
[16:14:54] <average>	 ottomata: read about this here http://blog.cloudera.com/blog/2012/05/namenode-recovery-tools-for-the-hadoop-distributed-file-system/
[16:15:23] <average>	 but maybe backup stuff first if it's possible
[16:15:40] <average>	 uhm, not sure if it's an fs corruption problem though
[16:20:35] <ottomata>	 yeah we don't care about any of the data here
[16:20:39] <ottomata>	 we just want the things to work
[16:20:46] <ottomata>	 but i don't think this is a hdfs problem
[16:46:10] <ottomata>	 milimetric:
[16:46:14] <ottomata>	 i think your node just isn't powerful enough
[16:46:21] <milimetric>	 heh
[16:46:21] <ottomata>	 1 processor, 2G mem
[16:46:29] <ottomata>	 mysql can't even start with default values as is
[16:46:30] <milimetric>	 ok, cool, i'll delete it
[16:46:48] <ottomata>	 yeah i'd do at least 2 processors on these
[16:46:59] <milimetric>	 ok
[16:47:07] <milimetric>	 and maybe like 8G?
[16:53:47] <grrrit-wm>	 (PS1) Milimetric: Config files can be specified in environment [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/97753
[16:53:59] <grrrit-wm>	 (CR) Milimetric: [C: 2 V: 2] Config files can be specified in environment [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/97753 (owner: Milimetric)
[16:55:31] <milimetric>	 ottomata: how do I permanently set environment variables properly?
[16:55:41] <milimetric>	 in /etc/profile.d ?
[16:56:25] <milimetric>	 hm, no... i want these to be available to the whole system
[16:56:44] <milimetric>	 i made wikimetrics read from env variables
[16:56:53] <milimetric>	 but i don't know how to set them properly in labs
[16:56:59] <milimetric>	 average: any idea?
[16:59:00] <average>	 milimetric: on ubuntu /etc/environment https://help.ubuntu.com/community/EnvironmentVariables#System-wide_environment_variables
[16:59:22] <milimetric>	 huh, ok
[16:59:30] <milimetric>	 i saw that but everyone says something different
[16:59:33] <milimetric>	 i'll try it, thanks!
[17:01:33] <average>	 milimetric: I gave you a link to official Ubuntu documentation. The server you deploy on is most likely Ubuntu. There is a note in there about /etc/profile.d/   "Note: Some systems now use an envvar.sh placed in the /etc/profile.d/ directory to set system wide environment strings."
[17:02:55] <average>	 my machine doesn't have /etc/profile.d/envvar.sh . I'm running Ubuntu Raring 13.04 on my laptop
[17:03:46] <average>	 I have some other machines with Ubunt precise 12.04 LTS on them, and those don't have that envvar.sh either
[17:04:30] <average>	 but, the good part is you can try it out and check with printenv or just env , to see what variables are set and what values they have. Apparently you can put your vars in many different files. One of them must be right
[17:04:39] <ottomata>	 milimetric: why do you want to set global env variables?
[17:04:40] <CommanderData>	 I told you yesterday it would be done by the end of today
[17:05:19] <average>	 ottomata has a point, you might want to write your settings in a configuration file instead of system-wide environment variables
[17:05:36] <milimetric>	 it's hard to explain
[17:05:54] <milimetric>	 but environment variables are much easier because I have no way of specifying
[17:06:01] <milimetric>	 through wsgi
[17:06:06] <milimetric>	 what config file to use
[17:07:12] <average>	 you can read your configuration from /etc/wikimetrics/config.json for example
[17:07:19] <average>	 that's where configuration files normally sit
[17:07:25] <milimetric>	 ugh
[17:07:29] <milimetric>	 wikimetrics is dead :(
[17:07:39] <milimetric>	 so it doesn't find those variables in /etc/environment
[17:09:39] <ottomata>	 milimetric: still not sure I understand
[17:10:07] <milimetric>	 so in wikimetrics.wsgi, I can only do something like "from wikimetrics.configurables import app as application"
[17:10:13] <milimetric>	 and then mod_wsgi takes it from there
[17:10:50] <milimetric>	 so I have command line parameters for specifying each config file
[17:10:54] <milimetric>	 like you can do this:
[17:11:04] <milimetric>	 wikimetrics --mode web --db-config blah.yaml
[17:11:15] <milimetric>	 but you can't do that from wikimetrics.wsgi
[17:11:51] <milimetric>	 so, in the argument parser, I'm giving --db-config the default os.environ['WIKIMETRICS_DB_CONFIG'] or 'wikimetrics/config/db_config.yaml'
[17:12:20] <milimetric>	 this lets it work in all the 3 ways that people want to run it
[17:12:40] <milimetric>	 SO, the problem is, if I add things to /etc/environment, echo $WIKIMETRICS_DB_CONFIG seems to work
[17:12:49] <milimetric>	 BUT, print os.environ does not show that variable
[17:13:06] <average>	 yes, because mod_wsgi might mess with your env variables...
[17:13:40] <milimetric>	 oh weird... now it does
[17:13:49] <milimetric>	 but yea, apparently mod_wsgi messes with the variable
[17:14:07] <milimetric>	 so os.environ has my stuff if I just run python
[17:14:12] <milimetric>	 but not from the wsgi process
[17:14:14] <milimetric>	 wtf
[17:14:20] <average>	 SetEnv <ENVVAR_NAME> <VALUE>
[17:14:27] <average>	 inside your VirtualHost blocks in Apache
[17:14:28] <milimetric>	 in apache?
[17:14:30] <milimetric>	 k
[17:18:42] * average  gazes at the window while snow starts to fall

[17:19:21] * yuvipanda  notes that we have a uwsgi module in puppet now, and mod_wsgi is not really preferred in the python world anyway

[17:19:21] <ottomata>	 yeah that's apache, and will probably work
[17:20:14] <halfak>	 yuvipanda: I didn't know that mod_wsgi is not preferred.  Do you have a good source for me to read up?
[17:20:20] <yuvipanda>	 moment
[17:21:01] <milimetric>	 damn... ottomata, average, i SetEnv and it doesn't seem to be picking it up
[17:21:03] <milimetric>	 wth
[17:21:12] <milimetric>	 anyone wanna jump in the hangout?
[17:21:28] <average>	 milimetric: have a read through this first http://drumcoder.co.uk/blog/2010/nov/12/apache-environment-variables-and-mod_wsgi/
[17:21:41] <milimetric>	 k
[17:21:52] <milimetric>	 i did read that, it's exactly what i did...
[17:22:11] <average>	 milimetric: try to isolate the problem with a bare-bone python app that sends out the env variables in the body of the page so you can see them in your browser
[17:22:23] <milimetric>	   SetEnv WIKIMETRICS_DB_CONFIG "/var/lib/wikimetrics-config/db_config.yaml"
[17:22:23] <milimetric>	   SetEnv WIKIMETRICS_WEB_CONFIG "/var/lib/wikimetrics-config/web_config.yaml"
[17:22:23] <milimetric>	   SetEnv WIKIMETRICS_QUEUE_CONFIG "/var/lib/wikimetrics-config/queue_config.yaml"
[17:22:23] <milimetric>	   WSGIDaemonProcess api user=www-data group=wikidev threads=10 python-path=/usr/lib/wikimetrics/wikimetrics
[17:22:23] <milimetric>	   WSGIScriptAlias / /usr/lib/wikimetrics/wikimetrics/api.wsgi
[17:22:53] <milimetric>	 maybe that WSGIDaemonProcess is messing it up
[17:23:22] <average>	 milimetric: put WSGIDaemonProcess before your SetEnvs
[17:26:01] <milimetric>	 http://stackoverflow.com/questions/9016504/apache-setenv-not-working-as-expected-with-mod-wsgi
[17:26:03] <milimetric>	 ugh
[17:26:09] <milimetric>	 i ... hate this
[17:27:12] <yuvipanda>	 halfak: hmm, so I was looking for concrete writing, and found http://blog.pythonanywhere.com/36/ and http://nichol.as/benchmark-of-python-web-servers and others
[17:27:30] <halfak>	 Thanks yuvipanda.  :)
[17:27:32] <yuvipanda>	 halfak: uwsgi has also been definitely faster / easier to setup / use in my personal experience
[17:29:20] <halfak>	 Good to know.  I was using CherryPy for a while, but made the switch to Apache2 when I started supporting SSL.  I'll have to check out uwsgi.
[17:29:37] <halfak>	 I do a lot of awkward things to live in Apache land.
[17:30:25] <yuvipanda>	 halfak: heh, yeah. I don't use apache for anything - even the mw I used to run was on nginx+php-fpm
[17:30:53] <yuvipanda>	 halfak: I'll also be adding direct uwsgi protocol support to the labs proxy soon, so that'll increase performance
[17:32:32] <yuvipanda>	 halfak: oh, btw - I'm planning on building a tool that lets anonymous users run SQL queries against the labsdb via a simple web interface. Think it'll be useful to the research community?
[17:32:45] <halfak>	 Oh god yes.  How do I help.
[17:32:49] <halfak>	 ^?
[17:33:02] <yuvipanda>	 :D
[17:33:19] <yuvipanda>	 I've the arch mapped out, should start building once some of my toollabs stuff is done.
[17:33:52] <yuvipanda>	 halfak: big question in my mind is if it should be something like jsfiddle - it also stores results as a permalink you can share
[17:34:03] <halfak>	 I'd really like to auto-log queries for transparency and meta-analysis.
[17:34:19] <halfak>	 Get out of my head :P
[17:34:20] <yuvipanda>	 indeed, jsfiddle does that :D
[17:34:51] <yuvipanda>	 so a jsfiddle like thing, or a simpler 'REPL', where you type queries and hit enter and see a result and keep going, copy paste stuff out and you're done
[17:34:51] <halfak>	 How would you like to handle the problem of running expensive queries?
[17:35:10] <halfak>	 Yuvipanda -- Any chance you see ipython notebook fitting into this?
[17:35:10] <yuvipanda>	 halfak: right now? hard limit them to x minutes, where x is something I can find out experimentally
[17:35:24] <yuvipanda>	 halfak: so, I've a working ipython notebook implementation already...
[17:35:25] <yuvipanda>	 :P
[17:35:29] <yuvipanda>	 halfak: just haven't packaged
[17:35:30] <yuvipanda>	  it
[17:35:48] <yuvipanda>	  halfak also it's impossible to do ipython notebooks *anonymously*. you will need a toollabs account
[17:35:58] <yuvipanda>	 while this tool can be made secure enough to be used just over the web
[17:36:00] <halfak>	 Good
[17:36:51] <halfak>	 I've got to run in a minute.  Do you have a project page or other docs/description I could check out?
[17:36:51] <yuvipanda>	 halfak: what would be more useful to the community at large? easier ipython notebooks or the SQL thing?
[17:36:59] <halfak>	 SQL thing.
[17:37:03] <yuvipanda>	 deal.
[17:37:11] <yuvipanda>	 halfak: i've them but they're on physical paper
[17:37:21] <yuvipanda>	 halfak: i'll mail them to you when I write it up
[17:37:21] <halfak>	 Lame
[17:37:21] <halfak>	 lol
[17:37:24] <halfak>	 email I hope
[17:37:26] <yuvipanda>	 haha
[17:37:26] <yuvipanda>	 :D
[17:37:34] <yuvipanda>	 postage to the US is expensive and slow :P
[17:37:38] <halfak>	 Hard to crtl-F paper.
[17:37:59] <yuvipanda>	 yup, but I was doing it when I take my  hourly break from keyboard use
[17:38:03] <yuvipanda>	 YAY Carpel Tunnel!
[17:38:14] <Ironholds>	 yay!
[17:38:26] <Ironholds>	 my doctors sent me my vaccination records. I don't need to get stabbed.
[17:38:38] <halfak>	 yuvipanda: OK.  I'm outta here, but I'll be back to bug you about this query interface later.
[17:38:45] <yuvipanda>	 halfak: sweet
[17:38:45] <halfak>	 :)
[17:38:59] <yuvipanda>	 halfak: happy to find someone interested :P was hard to drum up interest for the ipython thingy :D
[17:57:00] <grrrit-wm>	 (PS1) Milimetric: wsgi needs environment set up [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/97760
[17:57:10] <grrrit-wm>	 (CR) Milimetric: [C: 2 V: 2] wsgi needs environment set up [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/97760 (owner: Milimetric)
[17:58:08] <average>	 yuvipanda: did you get that kinesis ?
[17:58:17] <yuvipanda>	 average: yup!
[17:58:19] <yuvipanda>	 helps a fair bit
[17:58:26] <yuvipanda>	 i still take enforced timeouts
[18:01:02] <average>	 cool/q ottomata
[18:01:04] <average>	 damn
[18:01:46] <yuvipanda>	 :D
[18:02:36] <ottomata>	 heya
[18:02:56] <yuvipanda>	 ottomata: did you find time to migrate your haproxy solution to the labs dynamic proxy?
[18:05:49] <ottomata>	 no yuvipanda, i don't use it much
[18:05:49] <ottomata>	 but
[18:05:52] <ottomata>	 i did have a use case today
[18:06:01] <ottomata>	 where I wanted to use instance proxy (for dan's cluster)
[18:06:02] <ottomata>	 just for testing
[18:06:12] <ottomata>	 i didn't want to configure anything because it isn't regularly used
[18:06:20] <ottomata>	 so instance-proxy is kinda nice
[18:06:25] <ottomata>	 because it works with ports for all instances by default
[18:06:28] <ottomata>	 without having to configure anythign else
[18:06:28] <yuvipanda>	 yeah, true
[18:06:38] <yuvipanda>	 this is a more 'permanent' thing, plus ssl.
[18:06:44] <yuvipanda>	 instance-proxy won't go away anytime soon :D
[18:07:06] <ottomata>	 ok good :)
[18:07:07] <ottomata>	 cool
[18:07:07] <ottomata>	 yeah
[18:07:08] <ottomata>	 awesome
[18:07:36] <yuvipanda>	 I'll probably upgrade it at some point to be a bit faster
[18:07:42] <yuvipanda>	 and support websockets and such
[18:22:34] <average>	 ottomata: how do you deal with hadoop nodes running out of RAM ?
[18:22:39] <average>	 like in a real full-scale setup
[18:23:04] <average>	 I left desktop machine with the nodes all up for like 48h , now I have to vagrant halt; vagrant up;
[18:23:13] <average>	 cause JVM is greedy
[18:23:23] <ottomata>	 never run into problem in prod, because have lots of ram
[18:23:36] <average>	 aaaaah, I see
[18:26:43] <drdee>	 average: you have to tune the max number of reducers and mappers that can run and how much memory each of them can consume and make sure that the total memory consumption of the reducers and mappers is not larger than the total memory that you have available
[18:28:17] <average>	 drdee: hi
[18:28:34] <average>	 drdee: oh, I see, I'm gonna have to look in these xmls to see where I can set that
[18:28:44] <average>	 drdee: I have some small VMs, they have like 380mb ram each
[18:28:53] <drdee>	 yeah that's not a lot
[18:29:13] <drdee>	 the number of mappers is harder to tune as that depends on the block size of your hdfs partition
[19:06:31] <tnegrin>	 milimetric: you around?
[19:06:38] <milimetric>	 bah!
[19:06:40] <milimetric>	 yes
[19:06:40] <milimetric>	 sorry
[19:06:51] <tnegrin>	 np
[20:35:45] <ottomata>	 heya milimetric, can I get you to review some python code for me?
[20:35:50] <milimetric>	 sure
[20:35:53] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/97830/
[20:36:00] <ottomata>	 sorry I wasn't so responsive about the env vars before
[20:36:03] <ottomata>	 did you get that settled?
[20:37:25] <ottomata>	 probably the thing that would do with the best review is the flatten_object method in JsonLogster
[20:38:49] <milimetric>	 ok, sorry, was interrupted
[20:38:57] <milimetric>	 looking now
[20:38:59] <ottomata>	 k
[20:44:47] <milimetric>	 I like it ottomata
[20:44:58] <milimetric>	 you could probably make it shorter with reduce but it wouldn't be as clear
[20:45:09] <milimetric>	 so I just looked at flatten_object
[20:46:03] <average>	 reduce lol. guido v. rossum said reduce was mostly redudant, since it can only be used effectively in sum(..) or product(..)
[20:47:52] <ottomata>	 milimetric: there were a couple of much simpler versions I found
[20:47:56] <ottomata>	 but they didn't work with lists in the object
[20:48:26] <milimetric>	 yeah, I like the hasattr __iter__ check
[20:48:28] <milimetric>	 that's nice
[20:48:37] <milimetric>	 because it should work with everything
[20:54:34] <ottomata>	 thanks milimetric
[20:54:39] <ottomata>	 so ja, did your env thing get worked out?
[20:54:52] <milimetric>	 yes
[20:55:06] <milimetric>	 so environment variables are evil
[20:55:22] <milimetric>	 but necessary to get *something* passed to wikimetrics via wsgi
[20:55:46] <milimetric>	 so in wsgi itself, I just made it read from /etc/config/wikimetrics/
[20:56:07] <milimetric>	 and store it in os.environ['WIKIMETRICS_*_CONFIG']
[20:56:22] <ottomata>	 ok, that's cool
[20:56:28] <ottomata>	  /etc/config is weird
[20:56:34] <milimetric>	 then I just used my command line parameters to pass in config overrides for the queue (which starts via upstart)
[20:56:46] <milimetric>	 yeah, we can put it wherever when you puppetize it
[20:56:51] <milimetric>	 i don't know these things :)
[20:57:01] <milimetric>	 i had it in /var/lib/wikimetrics-config before
[20:57:14] <milimetric>	 i just randomly guess paths
[20:59:04] <ottomata>	 haha
[20:59:09] <ottomata>	  /etc/wikimetrics?
[20:59:14] <milimetric>	 sure
[20:59:20] <ottomata>	 :)
[20:59:22] <ottomata>	 or well
[20:59:24] <ottomata>	 even so
[20:59:28] <milimetric>	 when do you wanna work on the puppet for that?
[20:59:33] <ottomata>	 if you are deploying this via git deploy, we might need to talk about that more
[20:59:36] <milimetric>	 yep
[20:59:41] <ottomata>	 ummmmm, i have some time right now!  i just kinda finished a bunch of things for the day
[21:00:09] <milimetric>	 I've gotta talk to Teresa for a minute
[21:00:11] <yuvipanda>	 halfak: re: SQL anonymous access, I just had this other idea that's probably awesomer and also much easier / faster to do. Let me know when you have time to talk for a few minutes :D
[21:00:11] <milimetric>	 but sure
[21:00:47] <ottomata>	 k
[21:00:52] <halfak>	 yuvipanda: Sure.  What's up?
[21:01:15] <yuvipanda>	 halfak: so I was thinking - I could just use mysqlproxy and expose that directly.
[21:01:30] <yuvipanda>	 halfak: so anyone who wants can just directly connect from their local mysql client of choice
[21:01:33] <yuvipanda>	  no registration needed
[21:01:43] <yuvipanda>	 we can still limit it agressively - mysq1proxy lets you do that
[21:01:49] <yuvipanda>	 and have logs to figure out what people are using
[21:02:08] <yuvipanda>	 and I think it's fine securitywise too.
[21:02:27] <halfak>	 So mysqlproxy would make sure we don't get a DoS?
[21:02:33] <yuvipanda>	 halfak: pretty much
[21:02:40] <halfak>	 Looks solid.
[21:02:48] * halfak  is reading up on it for the first time

[21:03:04] <yuvipanda>	 halfak: :D
[21:03:21] <yuvipanda>	 halfak: I hadn't thought of it either. I'm also reading up properly only now
[21:05:41] <yuvipanda>	 halfak: there'll be folks who would find this useful, right? My only worry is that I'll pour effort into it and then it'll just sit there :)
[21:06:46] <halfak>	 I have little doubt that the public querying interface will be useful.  One of the things that we do a lot when discussing data is cite claims with queries to the DB.  Linking those queries to a public UI would be amazing.
[21:07:13] <halfak>	 Personally, the world changed substantially for me as a researcher when I first got access to a live copy of enwiki that it could run long queries on.
[21:07:29] <yuvipanda>	 alrighty then!
[21:07:41] <yuvipanda>	 halfak: I'll setup a proxy that supports just enwiki first, and then let's see where that goes?
[21:07:49] <halfak>	 Sounds good.
[21:07:54] <yuvipanda>	 halfak: note that this will have *no* UI.
[21:07:59] <yuvipanda>	 halfak: you connect via a normal mysql client
[21:08:07] <yuvipanda>	 we can bolt on a web UI later if we really want.
[21:08:35] <halfak>	 Hmm..  Less useful without the web UI.
[21:08:50] <average>	 I'm brewing some cheap tea !
[21:09:01] <halfak>	 But I'm happy to explore the implications with you.
[21:09:20] <halfak>	 One great example of uses for this proxy was the L2 hackathon that we had a couple of weeks ago.
[21:09:40] <halfak>	 I can guarantee some UI free use of such a proxy during the next L2 Research hackathon.
[21:09:52] <yuvipanda>	 when is that?
[21:11:06] <halfak>	 Not scheduled yet, but probably Feb.
[21:11:09] <halfak>	 Maybe March
[21:11:51] <average>	 what's a L2 Research hackathon ?
[21:12:05] <average>	 it sounds like a rocket "L2"
[21:12:56] <halfak>	 https://meta.wikimedia.org/wiki/Research:L2
[21:13:15] <yuvipanda>	 halfak: ah. I expect to have a proxy up way before end of the week :D
[21:13:15] <halfak>	 L2 is short for Labs^2 ("Labs squared")
[21:13:40] <average>	 oh, that's a clever and sneaky name
[21:13:41] <halfak>	 The community research lab I'm trying to build on top of/beside labs.
[21:14:25] <yuvipanda>	 milimetric: is the access to labsdb from wikimetrics puppetized?
[21:14:38] <milimetric>	 no
[21:14:50] <milimetric>	 we're puppetizing wikimetrics as we speak actually
[21:14:54] <yuvipanda>	 milimetric: ah.
[21:15:09] <milimetric>	 but wait, why would that part specifically need to be puppetized?
[21:15:10] <CommanderData>	 I heard there was a solar flare today
[21:15:19] <yuvipanda>	 milimetric: well, I want to use that specific part :P
[21:15:23] <milimetric>	 isn't it just having some things in your /etc/hosts
[21:15:23] <yuvipanda>	 in a different project
[21:15:25] <milimetric>	 oh!
[21:15:33] <yuvipanda>	 can you point me to what needs to be done?
[21:15:37] <milimetric>	 sure
[21:15:41] <milimetric>	 so ottomata did it
[21:15:50] <halfak>	 yuvipanda: You should join up to #wikimedia-labs2
[21:15:53] <milimetric>	 but as far as I know, it's just adding all the labs config to /etc/hosts
[21:16:05] <yuvipanda>	 halfak:set on autojoin :)
[21:16:15] <yuvipanda>	 milimetric: can you pastebin it?
[21:19:14] <ottomata>	 eh?  i dunno how to do that :p
[21:22:17] <milimetric>	 ottomata: definitely, you set up the wikimetrics instance in labs to be able to hit the labsdb hosts
[21:22:30] <milimetric>	 there was one other thing  that needed to happen except /etc/hosts
[21:23:17] <ottomata>	 oh hm
[21:23:38] <ottomata>	 i think in prod there willb e somethign else, not sure
[21:25:02] <milimetric>	 yuvipanda: I emailed it to you
[21:25:37] <yuvipanda>	 milimetric: ty!
[22:06:53] <ottomata>	 alriighty i'm outty laatas
[22:46:25] <grrrit-wm>	 (PS1) Milimetric: Teresa Tso found some bugs [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/97850
[22:46:26] <grrrit-wm>	 (CR) Milimetric: [C: 2 V: 2] Teresa Tso found some bugs [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/97850 (owner: Milimetric)