[00:02:20] <ottomata>	 uhhhhHH
[00:02:21] <ottomata>	 yes?
[00:02:22] <ottomata>	 dunno
[00:02:24] <ottomata>	 yes?
[00:02:25] <qchris>	 Ok.
[00:05:02] <grrrit-wm>	 (PS1) Ottomata: Modify pom.xml to build with WMF archiva and CDH 5.0.2 [analytics/wikihadoop] (wmf) - https://gerrit.wikimedia.org/r/165940
[00:05:40] <ottomata>	 qchris: gmime a quick review ^ :)
[00:05:51] * qchris looks
[00:07:09] <qchris>	 repo looks good. But why the version changes around log4j and junit?
[00:07:54] <ottomata>	 to match what we had in archiva :)
[00:08:28] <ottomata>	 oh i didn't mean to leave those comments in there
[00:08:40] <ottomata>	 or, qchris, do you thikn I should jsut add those to archiva and leave the original versions?
[00:09:18] <qchris>	 Not sure ... how will we use wikihadoop.
[00:09:26] <qchris>	 From inside the cluster or outside?
[00:09:37] <qchris>	 (No clue what wikihadoop really is)
[00:10:19] <ottomata>	 inside, i will probably add it to artifacts in refinery
[00:10:35] <qchris>	 Oh.
[00:10:35] <ottomata>	 qchris, it is a Custom InputFormat for mediawiki xml dumps
[00:10:43] <qchris>	 Yikes!
[00:10:44] <qchris>	 :-D
[00:11:01] <qchris>	 It compiles ... so I just CR+2 to not block you.
[00:11:03] <ottomata>	 drdee and aaron were involved with this 2 years ago
[00:11:07] <ottomata>	 ha, ok :)
[00:11:17] <ottomata>	 danke
[00:11:31] <grrrit-wm>	 (CR) QChris: [C: 2 V: 2] "Compiles for me and passes tests." [analytics/wikihadoop] (wmf) - https://gerrit.wikimedia.org/r/165940 (owner: Ottomata)
[00:17:54] <grrrit-wm>	 (PS1) Ottomata: Add wikihadoop-0.2-wmf1.jar and symlink (via git-fat and archiva) [analytics/refinery] - https://gerrit.wikimedia.org/r/165942
[00:18:55] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] Add wikihadoop-0.2-wmf1.jar and symlink (via git-fat and archiva) [analytics/refinery] - https://gerrit.wikimedia.org/r/165942 (owner: Ottomata)
[00:32:24] <DarTar>	 halfak, ottomata: https://github.com/diegoceccarelli/json-wikipedia
[00:32:31] <DarTar>	 he’s actually a friend of mine
[00:35:04] <ottomata>	 cool!
[11:08:59] <wikibugs>	 Analytics / Refinery: Make webrequest partition validation handle races between time and sequence numbers - https://bugzilla.wikimedia.org/69615#c9 (christian) Happened again for:    2014-10-09T17:xx:xx/2014-10-09T18:xx:xx (on upload)
[16:47:59] <grrrit-wm>	 (PS3) Ottomata: Use tsv format when outputting webrequest faulty hosts files [analytics/refinery] - https://gerrit.wikimedia.org/r/150963
[17:20:12] <grrrit-wm>	 (CR) Ottomata: [C: 2 V: 2] "Gage and I are about to restart Oozie jobs, I want this to go in." [analytics/refinery] - https://gerrit.wikimedia.org/r/150963 (owner: Ottomata)
[17:31:25] <ragesoss>	 hey all. are there problems with wikimetrics right now, or is it working normally?
[17:32:47] <ragesoss>	 LiAnna says she's getting 504 gateway timeout errors when trying to download a report.
[17:36:38] <nuria>	 raegesos: let me look
[17:36:49] <nuria>	 ragesoss: let me look
[17:36:56] <ragesoss>	 thanks!
[17:43:16] <nuria>	 ragesoss: you know that reports are not stored "for ever" right?
[17:43:31] <ragesoss>	 nuria: yeah. this as something just run.
[17:44:44] <nuria>	 ragesoss: a gateway timeout suggests availability on labs having issues rather than a problem with wikimetrics, let me try to repro
[17:45:05] <nuria>	 ragesoss: i sm in sf currently so i might not run into the same issue
[17:45:29] <ragesoss>	 nuria: this was from someone else in SF having the problem.
[17:45:39] <nuria>	 ragessos: cannot repro, just run a report: https://metrics.wmflabs.org/static/public/1504481.json
[17:45:43] <ragesoss>	 (I'm trying to figure out exactly what they were running, as I didn't have the problem with a test)
[17:45:58] <nuria>	 ragesoss: ok, let us know when you know
[17:46:16] <tnegrin>	 just an FYI -- Robla and Mark B asked me to forward the [Uu]defined bug to the ops list
[17:46:28] <tnegrin>	 more eyes/shallow bugs and all
[17:50:28] <ragesoss>	 nuria: it sounds like it's not related to the query, as they ran almost the same one as me, and wikimetrics reported the successful run, it was just the csv download that was and is not working for them.
[17:50:39] <ragesoss>	 but working fine for me with basically the same query.
[17:50:55] <nuria>	 ragesoss: same cohort?
[17:51:13] <nuria>	 ragesoss: or perhaps theirs is a bigger one, we have some problems with size
[17:51:28] <ragesoss>	 nuria: we uploaded the same csv file (2095 names, 2090 validated for me)
[17:52:22] <ragesoss>	 nuria: actually, nevermind, they were running per-user and I just ran aggregate.
[17:52:26] <ragesoss>	 I'll test again.
[17:52:31] <ragesoss>	 could be the size issue.
[17:54:51] <nuria>	 ragesoss: we have an open bug for very large cohorts and results, so an workarround can be to split the cohort. If size is bigger than 40.000 you might
[17:54:55] <nuria>	 run into issues
[17:55:05] <ragesoss>	 nuria: size is only ~2000
[17:55:28] <nuria>	 ragesoss: then i doubt size  is the issue
[17:55:52] <ragesoss>	 nuria: looks like I'm going to hit the same issue, now that I ran the query with individual results.
[17:56:19] <ragesoss>	 waiting for the csv, but it's taking many tens of seconds.
[18:02:35] <nuria>	 ragesoss: can you run the report again making it public so we can get resukts as json?
[18:02:40] <nuria>	 *results
[18:03:05] <ragesoss>	 nuria: yes: https://metrics.wmflabs.org/static/public/1504496.json
[18:03:28] <nuria>	 ragesoss: ok, no problems with json but cvs not so great right?
[18:03:36] <ragesoss>	 nuria: right.
[18:03:56] <nuria>	 ok, would you please file a bug with deatails so we can repro?
[18:04:14] <ragesoss>	 nuria: I also just ran into this when I first tried to access json for a small run: {
[18:04:14] <ragesoss>	     "message": "no task exists with id: null",
[18:04:14] <ragesoss>	     "isError": true
[18:04:15] <ragesoss>	 }
[18:04:22] <ragesoss>	 I tried again a minute later and it worked.
[18:04:38] <nuria>	 ragesoss: and suggest the json workarround to the user running the other report?
[18:05:10] <nuria>	 ragesoss: will check logs now to see but if you can file a bug with details it will be great, are you familiar with bugzilla?
[18:05:14] <ragesoss>	 nuria: the problem with the json workaround is that the json results lack usernames.
[18:05:50] <ragesoss>	 the reason we're running this report is to check one student editors in our program who have been very active, so we can see if they are doing well and help them out if not.
[18:06:20] <nuria>	 ragesoss: right, and the json only has user-ids right?
[18:06:22] <ragesoss>	 if we have to do db queries to find out the usernames for each id we are interested in, it will be super tedious.
[18:06:27] <ragesoss>	 nuria: right.
[18:06:34] <ragesoss>	 and yes, I'll file a bug today.
[18:06:47] <nuria>	 ragesoss: right so not such a good work arround
[18:15:28] <wikibugs>	 Analytics / Wikimetrics: Story: WikimetricsUser downloads large CSV quickly - https://bugzilla.wikimedia.org/71255#c2 (Sage Ross) Created attachment 16742   --> https://bugzilla.wikimedia.org/attachment.cgi?id=16742&action=edit Cohort of student editors from fall 2014 on enwiki  A cohort of 2095 userna...
[18:15:58] <wikibugs>	 Analytics / Wikimetrics: Story: WikimetricsUser downloads large CSV - https://bugzilla.wikimedia.org/71255#c3 (Sage Ross) In many cases it will not download ever. After many minutes, the user gets a 504 gateway error.  For example: https://metrics.wmflabs.org/reports/result/c04cf328-f198-4a12-81ec-9cb4...
[18:16:13] <wikibugs>	 Analytics / Wikimetrics: Story: WikimetricsUser downloads large CSV - https://bugzilla.wikimedia.org/71255 (Sage Ross) s:enhanc>major
[18:16:28] <ragesoss>	 nuria: I added the cohort as an attachment to that bug ^
[18:16:49] <nuria>	 ragesoss: is Ok to make the dat apublic?
[18:16:54] <nuria>	 *data public
[18:16:57] <ragesoss>	 nuria: yes.
[18:17:05] <nuria>	 ragesoss: ok, excellent
[18:17:09] <tnegrin>	 thanks for the check nuria!
[18:21:14] <wikibugs>	 Analytics / Wikimetrics: Story: WikimetricsUser downloads large CSV - https://bugzilla.wikimedia.org/71255 (nuria) p:Normal>High
[18:25:15] <wikibugs>	 Analytics / Wikimetrics: Story: WikimetricsUser downloads large CSV - https://bugzilla.wikimedia.org/71255 (nuria) a:nuria
[18:27:05] <ottomata>	 milimetric: also this
[18:27:05] <ottomata>	 http://thinkaurelius.github.io/titan/
[18:27:09] <ottomata>	 re. neo4j
[18:27:13] <ottomata>	 https://groups.google.com/forum/#!msg/aureliusgraphs/vkQkzjN8fo0/9YYgqI4TA0QJ
[19:23:56] <andrewbogott>	 I'm trying to create an RT account for Marcel, can someone provide his email address?
[19:32:41] <andrewbogott>	 47 members, all lurkers?
[19:43:49] <Ironholds>	 andrewbogott, mforns@wikimedia.org?
[19:43:54] <Ironholds>	 it should be in your email account's autocomplete ;p
[19:45:41] <ottomata>	 Ironholds: you have a very long running hive job, just curious, what is it?
[19:45:46] <ottomata>	 and, do you know it is still running?
[19:46:03] <Ironholds>	 ottomata, it shouldn't be? what's the query?
[19:46:07] <andrewbogott>	 Ironholds: thanks
[19:46:15] <Ironholds>	 I mean, does the weird dashboard...thing. say so?
[19:46:21] <Ironholds>	 (more usefully; can you tell when it was launched?)
[19:49:34] <ottomata>	 ironholds, it was showing up in the hadoop gui an hour or so ago, but via the cli
[19:49:40] <ottomata>	 yarn application -status application_1409078537822_24347
[19:49:42] <ottomata>	 on stat1002
[19:49:48] <ottomata>	 SELECT COUNT(DISTINCT(uri_path))
[19:49:48] <ottomata>	 FROM ...200(Stage-1)
[19:49:54] <Ironholds>	 huh!
[19:49:55] * Ironholds thinks
[19:49:56] <Ironholds>	 OH
[19:49:59] <Ironholds>	 wait. no. hrm.
[19:50:05] <ottomata>	 it has been running since sept. something
[19:50:09] <ottomata>	 all mappers have finished
[19:50:13] <ottomata>	 single reducer is trucking along :
[19:50:14] <ottomata>	 :)
[19:50:16] <Ironholds>	 ....okay then no that's not what one care about ;p
[19:50:18] <Ironholds>	 *I
[19:50:30] <Ironholds>	 if it's been running for more than, mmmn, call it 2 days, from me, you can safely kill it.
[19:50:43] <ottomata>	 ok
[19:50:46] <Ironholds>	 it's something I tried and failed to kill, or something I got a broken pipe with and couldn't retrieve the application ID, or...etc.
[19:50:50] <ottomata>	 hmm, ok
[19:51:11] <ottomata>	 done
[19:51:11] <Ironholds>	 you'd think "kill the task if you stop being able to send things to the place you're sending the results to" would be a feature of hive
[19:51:19] <Ironholds>	 but that probably doesn't make sense for reasons I'm unaware of
[19:51:39] <ottomata>	 well, i think usually peole send results to hdfs rather than attached terminal
[19:51:42] <Ironholds>	 if it's a SELECT statement that's writing out to the cli or file? and you can't talk to [place]? yeah, why keep it?
[19:51:44] <Ironholds>	 gotcha
[19:51:46] <ottomata>	 hadoop has no idea if your terminal is still attached
[19:52:21] <ottomata>	 and, i dunno how hive does it, i betcha hive abstracts it so that if you are expecting the results on stdout, it will write to some HDFS tmp dir and then cat it for you
[19:57:04] <ottomata>	 andrewbogott: i think we were all eating lunch!
[19:57:19] <ottomata>	 mforns@wikimedia.org is correct
[19:57:28] <Ironholds>	 gootcha
[20:21:46] <bmansurov>	 milimetric: I'm having trouble installing wikimetrics. When I run `vagrant provision` I get the follow error message: "Error: Invalid parameter archive_tablename at /tmp/vagrant-puppet-5/modules-0/role/manifests/wikimetrics.pp:63 on node mediawiki-vagrant.dev" How can I fix it?
[20:22:47] <milimetric>	 yes, bmansurov, that's a problem with the puppet definition
[20:22:55] <milimetric>	 we updated prod but not vagrant
[20:23:05] <ottomata>	 its updated now, no?
[20:23:06] <ottomata>	 git pull?
[20:23:06] <milimetric>	 mforns can help you, he just went through the same thing
[20:23:09] <ottomata>	 git submodule udpate --init?
[20:23:13] <milimetric>	 no, that's the role in vagrant
[20:23:17] <milimetric>	 i don't think that's updated yet
[20:23:38] <ottomata>	 i think mforns submitted a patch
[20:23:41] <bmansurov>	 ottomata: I did git submodule update --init, but didn't help
[20:23:48] <bmansurov>	 mforns: help?
[20:24:04] <ottomata>	 (we are in a little meeings for the next few mins )
[20:24:26] <bmansurov>	 ok, when you get a chance ;)
[21:05:30] <wikibugs>	 Analytics / Wikimetrics: Valid username is ruled invalid by Wikimetrics - https://bugzilla.wikimedia.org/71923 (Sage Ross) NEW p:Unprio s:normal a:None There's a username in an enwiki cohort I pasted in to Wikimetrics that seems to be a normal one, but Wikimetrics checks it and indicates it...
[21:09:58] <wikibugs>	 Analytics / Wikimetrics: Valid username is ruled invalid by Wikimetrics - https://bugzilla.wikimedia.org/71923#c1 (Sage Ross) Other usernames that I can't validate and don't know why:  Dd-556 Finding Zeno
[21:14:59] <wikibugs>	 Analytics / Wikimetrics: Valid username is ruled invalid by Wikimetrics - https://bugzilla.wikimedia.org/71923#c2 (Sage Ross) Also:  Hansenr11
[21:15:49] <grrrit-wm>	 (PS1) Ottomata: Add UAParserUDF from kraken [analytics/refinery/source] - https://gerrit.wikimedia.org/r/166142
[21:16:28] <wikibugs>	 Analytics / Wikimetrics: Valid username is ruled invalid by Wikimetrics - https://bugzilla.wikimedia.org/71923#c3 (Sage Ross) These usernames are invalid whether pasted in or uploaded via CSV.
[21:16:34] <grrrit-wm>	 (CR) Ottomata: "This will conflict with https://gerrit.wikimedia.org/r/#/c/164264/, but I will fix up and rebase that patch on top of this one." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/166142 (owner: Ottomata)
[22:01:13] <wikibugs>	 Analytics / Wikimetrics: Valid username is ruled invalid by Wikimetrics - https://bugzilla.wikimedia.org/71923#c4 (nuria) The labs db data has issues around September 20th, part of the data for this user is missing. This is by no means a common event as we have not seen this happening before. We will w...
[22:18:33] <bmansurov>	 milimetric: how can I make wikimetrics-web service pick up code changes?
[22:18:51] <milimetric>	 bmansurov: you're in vagrant right?
[22:18:57] <bmansurov>	 yes
[22:19:08] <milimetric>	 I don't think wikimetrics-web is what's serving the site there
[22:19:12] <milimetric>	 I think it's coming out of apache
[22:19:16] <milimetric>	 so try restarting apache2
[22:19:52] <bmansurov>	 milimetric: is it also using mod_wsgi? I'd like see my changes immediately without restarting apache.
[22:20:07] <milimetric>	 bmansurov: it's using mod_wsgi, yes
[22:20:24] <milimetric>	 bmansurov: I develop wikimetrics locally, without vagrant
[22:20:25] <bmansurov>	 milimetric: do you use flask's built in server for development?
[22:20:30] <bmansurov>	 milimetric: ok cool
[22:20:41] <milimetric>	 for me, it makes things much easier
[22:20:56] <milimetric>	 but others find it frustrating to install everything
[22:20:56] <bmansurov>	 milimetric: you've also set up the database locally?
[22:20:59] <milimetric>	 yes
[22:21:02] <milimetric>	 there's scripts/install
[22:21:02] <bmansurov>	 milimetric: i see
[22:21:11] <milimetric>	 and scripts/[a bunch of database creation files]
[22:21:22] <bmansurov>	 cool
[22:23:32] <milimetric>	 but bmansurov, if you want to work locally and avoid vagrant headaches, I can stop by in a few minutes and help you get set up that way
[22:24:14] <bmansurov>	 that'd be awesome, milimetric
[22:52:50] <ragesoss>	 I'm a little unclear from the documentation I saw... data be gotten from wikimetrics via API? I tried some of the examples from UserMetrics/Guide but they didn't work for me.
[22:52:54] <ragesoss>	 eg, https://metrics.wmflabs.org/cohorts/Kmenger/bytes_added?project=mediawikiwiki&start=20130101000000&end=20130401000000&is_user=True
[22:53:27] <ragesoss>	 And what about creating cohorts via API?
[23:14:38] <grrrit-wm>	 (PS1) Mforns: Render invalid cohort members as html [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/166150
[23:14:57] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Render invalid cohort members as html [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/166150 (owner: Mforns)
[23:36:05] <grrrit-wm>	 (PS2) Mforns: Render invalid cohort members as html [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/166150
[23:58:04] <grrrit-wm>	 (PS1) Bmansurov: Add a timerange validator [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/166157 (https://bugzilla.wikimedia.org/70714)
[23:58:10] <grrrit-wm>	 (CR) jenkins-bot: [V: -1] Add a timerange validator [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/166157 (https://bugzilla.wikimedia.org/70714) (owner: Bmansurov)