[06:50:14] UA classification.. [06:50:20] into tablets and mobile [14:26:29] morning guys! ottomata, milimetric, average_drifter [14:27:26] morning! [14:27:53] check this out stats.wikimedia.org/kraken/findbugs.html [14:28:12] haha [14:28:15] this is the first part that is working, now i need to get the javadocs generated as well [14:28:18] bad practice! [14:28:23] love it [14:28:36] why bad practice? [14:32:29] haha that's what it says [14:32:32] BAD_PRACTICE [14:32:35] The class name org.wikimedia.analytics.kraken.pig.isValidIPv6Address doesn't start with an upper case letter [14:33:22] we will fix it or turn off that rule :D [14:33:34] i am very happy with this personally [14:33:47] and i will add the javadocs for our UDF's as well [14:34:14] so that people who want to use a Pig UDF know what input parameter it requires and what output it generates [14:35:51] cool! [14:47:08] http://stats.wikimedia.org/kraken/apidocs/ [14:47:14] it needs work :) [14:56:52] drdee, how's the babe? how's the sleep?! [14:57:17] she is awesome! and she sleeps really well [14:57:44] you totally would like her, :) [14:58:48] my father called me a couple of days ago, he told me that at my age, he already 2 kids, me and my sister, and that I should explain to him wth am I doing :) [14:58:51] :)) [15:00:34] that's a subtle hint :) [15:08:19] ottomata, if you are going to do the log changes, can we do all in one big swoop and then *also* change the filenames of the logs, like appending'_new' or something so it's very clear that there have been changes in the format [15:08:38] or maybe start versioning the log files [15:08:57] like '_v2', '_v3' if we don't do it in one big swoop [15:09:12] else we need a lot of detection logic in the scripts to see what log file we are consuming [15:24:18] ottomata,on what machine is tomcat running, i need the log files [15:24:34] (or check http://hue.analytics.wikimedia.org/oozie/submit_workflow/30) [15:26:16] re log files [15:26:17] sure [15:26:21] we can change the names pretty easily [15:26:35] re tomcat, uhhhhh [15:26:38] an27 I guess [15:26:43] yeah, oozie is an27 [15:27:38] ok [15:28:39] do you know the location of the tomcat log files/ [15:35:08] it looks like upgrading to cdh 4.1.2. caused an issue [15:35:25] when submitting a oozie workflow in hue i get an internal server error [15:38:22] stacktrace: https://gist.github.com/03faf6cd8ea274dd2a01 [15:38:29] no clue :( [15:46:19] morning milimetric! [15:46:28] morning drdee [15:46:40] late morning for me, haven't been feeling too awesome [15:47:37] take some rest [15:53:40] ottomata, could the oozie bug be a permission issue on tomcat? [15:58:18] early patchset https://gerrit.wikimedia.org/r/39587 [15:59:38] where is hashar [15:59:39] :) [15:59:59] heya [16:00:00] ummmm [16:00:01] jenkins doesn't run tests now because of zuul. it runs them only upon merge [16:00:33] I really don't mind zuul, but I'd rather have tests run on each patchset I send [16:01:23] I can't see the logic in only running tests after the merge [16:01:48] when tests are supposed to prevent you from merging broken code [16:02:30] drdee, where is the job file you're using? [16:05:07] all files are in /user/diederik/blog/ (hdfs) [16:05:24] go to http://hue.analytics.wikimedia.org/oozie/list_workflows/ [16:05:32] and then select 'Pig job blog.wikimedia.org' [16:07:43] I just sent an e-mail to Antoine [16:07:54] drdee: Cc-ed you to the e-mail [16:08:50] t [16:08:52] ty [16:10:04] i know the solution to your problem, hold on [16:12:36] oh, drdee, i think i know [16:12:41] your dir permissions were all fukny again [16:12:56] how come i don't touch them [16:12:58] i dunno [16:13:11] you got a weird umask somewhere? [16:13:21] but, blog/ wasn't group readable [16:13:29] or world readable [16:13:40] so that's probably why it couldn't open the .jar files and was giving zip errors [16:13:46] i chmoded them as I tried to read them [16:14:02] so try again I guess [16:19:17] trying now [16:19:24] average_drifter: https://gerrit.wikimedia.org/r/39590 [16:19:31] get hashar to approve this : [16:19:33] :) [16:20:02] ottomata, nope :( [16:20:05] that didn't solve the problem [16:20:17] hm rats [16:24:51] drdee: thanks ! [16:42:45] heh this is kind of funny [16:43:01] brr [16:43:02] I pushed limn to wikimedia (was working on mine and David's fork) [16:43:03] god it's cold. [16:43:06] hey dschoon [16:43:08] yeah here too [16:43:12] 20 degrees [16:43:15] drdee: reading the e-mail [16:43:20] so I pushed limn [16:43:28] we used to be at 431 commits [16:43:33] we're now at 860 [16:43:37] almost exactly twice [16:43:41] maybe finally the graphs will update with you on them [16:43:42] ha [16:43:46] AWESOME! [16:43:51] yay! [16:43:53] https://github.com/wikimedia/limn/graphs/contributors [16:44:14] now oloho is going to say we've written 32 years worth of code in the last 3 months ;) [16:44:15] brb [16:44:59] haha [16:45:36] yeah, i've gotta patch ohloh's line counter and do a pull request to make it see coco [16:48:11] average_drifter: so regarding your tests, first you need to get a +2 on https://gerrit.wikimedia.org/r/#/c/39587/ and that will trigger the unit-tests until we got you merged as a whitelisted developer [16:51:54] ottomata, any other ideas on how to fix the tomcat problem? [16:51:57] these guys https://github.com/TheWeatherChannel/dClass ... that's a wow [16:52:36] is that from the email i sent you? [16:52:50] yes [16:52:58] sub-microsecond time .. [16:52:59] decision tree definitely sounds the way to do device detection [16:53:19] yeah, especially considering how many different types of UA patterns there are [16:53:35] are there wrappers for the C-library? [16:53:55] drdee, sorry, will look into that shortly [16:53:59] ok [16:54:02] focusing on stupid packet loss atm [16:54:02] drdee: we can make some [16:54:07] trying to get som emore metrics in ganglia [16:54:15] me like it! [16:54:40] it would completely solve our custom-regex-based approach to UA parsing that we currently have [16:55:15] ua-parser would solve it too but they have a regex-based approach too, and I guess that's slow(haven't benchmarked that one) [16:56:27] drdee: so do you think we could benefit if would have some wrappers for dClass ( https://github.com/TheWeatherChannel/dClass ) ? [16:56:48] drdee: maybe a long-term objective [16:56:51] yes but please hold your horse :) [16:56:57] ok [16:58:28] ottomata, btw, we have had the same problem before and fixed it obviously , i just don't remember what it was (the tomcat/zipfile issue) [17:00:25] drdee: theoretically there's a lot of inference one can do when you parse a UA. I mean the tree approach makes sense to me. For example, if you were looking for Android and you didn't find the Android string, no point in looking forward for other stuff. Another thing is for example if you didn't find Windows in the UA , there's no point for checking for MSIE browser type [17:00:41] drdee: and that could eliminate lots and lots of branches when searching [17:00:47] totally [17:05:13] let's not start adding features to wikistats though [17:05:22] remember, our goal is to fix bugs [17:05:41] and then replicate all important features on kraken so we can kill it [17:05:55] if you really want to be a badass with dClass, write JNI bindings so we can use it in Pig :) [17:06:03] http://users.cs.cf.ac.uk/Yan.Huang/research/JACAW/JACAW.htm [17:06:09] is how to do that [17:06:32] dschoon, btw: http://stats.wikimedia.org/kraken/apidocs/ [17:06:45] it needs a bit more love but it's a first start [17:06:55] hot [17:07:04] yeah, i saw the findbugs report also [17:07:10] you have that all running out of jenkins? [17:07:10] deployment is all mavenized [17:07:23] because it's trivial to do via maven [17:07:25] great! [17:07:31] no find bugs & apidocs through maven [17:07:44] mvn site-deploy :D [17:07:59] well and writing the pom.xml of cours [17:09:39] ja, with the right plugins [17:11:31] exactly :D [17:11:35] that's the real pain [17:11:43] like debugging wagon-ssh [17:11:44] it is [17:11:46] with dan yesterday [17:11:50] but we fixed it [17:13:24] reminder: no scrum today [17:13:29] platform meeting at 10a PST [17:16:34] milimetric: http://stats.wikimedia.org/kraken/apidocs/ [17:16:42] and thank you for your help yesterday! [17:16:46] yeah, was just reading up [17:16:47] cool [17:16:59] i was gonna ask you about travis ci [17:17:03] ask [17:17:18] after pushing to wikimedia it failed another build so I figured it's worth a shot updating [17:17:26] totally [17:17:27] how can I edit the job? [17:17:34] there's a .travis.yaml file [17:18:03] i have just merged it in [17:18:04] heh [17:18:16] bc diederik's pull req was still outstanding [17:18:18] https://github.com/wikimedia/limn/pull/37 [17:18:36] go nuts. read their docs. [17:18:54] yeah, found it, thanks [17:19:08] http://about.travis-ci.org/docs/ [17:19:48] heading out [17:20:30] HHAHAHAHAHA now i am also an official Limn contributor! [17:21:05] i am gonna beat dschoon's 300k lines of code [17:21:15] bring it, bitch [17:21:36] my will is like an ocean in storm. [17:21:36] but you actually removed more than you wrote [17:21:42] i shall crush you with waves of code. [17:21:43] so that's negative development [17:21:46] of course :) [17:21:51] most code sucks, even mine [17:21:52] bbl [17:22:50] milimetric: are you guys thinkin of using travis instead of the internal jenkins :D [17:23:05] no [17:23:21] i think both is the right answer [17:23:26] for now [17:23:34] that way we can see what works better [17:24:02] but travis won't be able to deploy so I'm guessing Jenkins will be the answer [17:37:19] hashar: ! [17:37:28] hashar: hi ! :) [17:37:40] hashar: when you have some time please check your e-mail [17:38:00] average_drifter: yup I connected just to reply to you :D [17:38:09] :D [17:38:57] travis only for limn as we try to build a community around limn [17:39:17] hasher, actually i might have added the wrong email address of stefan [17:39:23] i added his gmail address [17:39:38] but i think he uses another one :( [17:39:39] sorry [17:39:53] hashar ^^ [17:39:58] average_drifter ^^ [17:40:13] stefan@garage-coding.com [17:40:16] oops [17:40:20] just when I merged it hehe [17:40:21] my fault [17:40:26] will add that one too [17:40:31] thank you :) [17:41:24] though that only whitelist you on mediawiki/core.git for now [17:41:30] ohhhhhhh [17:41:39] but we need to have him whitelisted for analytics/* [17:42:06] also patches to zuul-config.git once merged get deployed automatically? [17:42:14] not yet :/ [17:42:19] I am still deploying that manually. [17:42:21] ok [17:42:24] good to know [17:42:32] whenever I find out a way to verify the conf is valid, I will make jenkins deploy and reload zuul [17:42:41] awesome [17:42:48] so we don't really care about mediawiki/core.git [17:42:51] now I need to whitelist you on your repos [17:43:00] aight [17:44:10] https://gerrit.wikimedia.org/r/39601 [17:44:23] the 'test' pipeline is on patchset submission and let whitelisted people through [17:44:31] AWESOME! [17:45:14] should be deployed on all analytics projects now [17:45:17] sorry for the delay :-D [17:45:28] I completely forgot about your jobs huhu [17:45:59] can you please send a patchset on one of them to test it out ? [17:46:24] yes [17:48:18] drdee: we might have also solved 4) from Erik's list on this patchset [17:48:33] drdee: there was a file MobileDeviceTypes.csv which wasn't read from the right path [17:48:52] drdee: now testing again on a 25M chunk of data to see the outcome [17:49:24] drdee: (the csv file above wasn't read from the right path because Erik moved some files to csv/meta at some point) [17:49:29] I am out, be back later tonight maybe [17:49:30] cya [17:54:01] I get 0.4% Android in non-mobile [17:54:13] on the 25M dataset [17:56:21] I should have all of them in either mobile or tablet [17:56:39] drdee: what should "other mobile" contain ? [17:56:52] drdee: non-tablet and unknown mobiles ? [18:03:00] yes that's sounds about right, and unknown mobile apps [18:03:17] platform url? [18:15:41] ottomata, the oozie error also happens directly using the CLI [18:15:44] so Hue is not invoved [18:24:15] heya guys -- if you're feeling the need for something to do while in the meeting, may i gently suggest we update our project statuses? [18:25:24] the "[add new]" link for each project on https://www.mediawiki.org/wiki/Analytics [18:25:37] ^^ ottomata milimetric drdee average_drifter [18:26:20] the status date should be "2012-12-monthly" [18:26:29] hoookay [18:26:29] man, yet another place for statuses? [18:26:34] i know right [18:26:36] heh [18:26:50] https://www.mediawiki.org/wiki/Wikimedia_Platform_Engineering#Analytics [18:26:58] oh, nm. they got rid of the "monthly" thing [18:27:02] i know. i hate this thing [18:27:04] i think it's awful. [18:27:04] can we pick one? [18:27:06] but we have to do it. [18:27:09] we can't. [18:27:15] this one is stupid and required and stupid. [18:27:21] k [18:27:24] * milimetric confused [18:28:03] I know. [18:28:18] It's because we have this awful habit of trying to integrate everything into mediawiki. [18:28:31] oh no, those are the same things [18:28:31] so we have extremely poor project management tools embedded in MW as gadgets and templates. [18:29:47] Robla showed me how to add the project status widget to my page: https://www.mediawiki.org/wiki/User:DAndreescu [18:30:16] I believe updating that updates the status in both dschoon and my link above [18:30:38] yeah, because it's a template [18:30:43] so it gets transcluded everywhere. [18:30:54] anyway, the analytics homepage has all the links, iirc [18:31:04] i'm adding ones for WikiStats and some other crap [18:33:58] dschoon, i think i have better insight into wikistats [18:34:05] non o [18:34:08] i'm not updating [18:34:16] i'm adding the *link* to the status on our homepage [18:34:18] because it doesn't show up atm [18:34:19] :) [18:34:23] sorry [18:35:23] I can't grok the Totals percentages in SquidReportClients.htm [18:35:38] it "don't" make sense :| [18:36:15] is it a relative total to the "big" total ? [18:36:29] or is it total as in, it should be 100% for that particular category always ? [18:37:49] only god knows. [18:38:09] what url? [18:39:05] drdee: moment I want to publish them so I can give a link [18:39:17] ok [18:39:23] ottomata: you know when we talked about exposing stuff from /home//public on stat1.wikimedia.org ? [18:39:34] ottomata: was it ~/public or somethin else. I forgot :( [18:40:19] www [18:40:24] ok [18:42:25] drdee: http://stat1.wikimedia.org/spetrea/regression-tablets-discrepancy_for_config_editors/reports/2012-10/SquidReportClients.htm [18:42:43] the good news is that I fed it just tablet data and it has 0.0% in "non-mobile" [18:43:01] the bad news is I don't understand every number in the table. In particular the totals [18:43:04] :D [18:43:11] what row/column are you talking about? [18:43:33] OHHHHHH I [18:43:42] those numbers don't add up at ALLL! [18:44:04] so I guess every total row [18:44:11] just summarizes it's own sub table [18:44:14] hope that makes sense [18:44:22] yes it does [18:44:27] so all the totals should be 100% [18:44:27] so 'Browsers, other mobile' does not add up [18:44:45] yes [18:45:04] welll......... [18:45:24] check with erik [18:45:26] ok [18:45:27] i don't know either [18:46:09] milimetric -- you should get in touch with chris about what he's saying now [18:46:19] was definitely thinking that :) [18:46:28] maybe we can cross-pollinate our efforts on clientside testing [18:46:50] yes, I heard rumors about krinkle setting up TestSwarm too [18:47:01] which makes sense :) [18:47:54] yeah. [18:49:13] he's Chris McMahon -- cmcmahon@wikimedia.org [18:54:58] 162.5% ... that must definitely be wrong [18:55:31] aaand.. there's another one with 1275.3% [18:55:34] contacted Erik [18:55:46] he said he'll be available in ~1h to talk with me [18:58:23] ok [18:58:52] ottomata, I FIXED IT [18:58:57] just a simple reboot [18:59:12] yeah!? awesome, yeah acutally, i didn't restart oozie or hive when I upgraded, only the hadoop services [18:59:50] i got my stuff in ganglia too, but I can't create the aggregate graph there that I want to until the js stuff is fixed [19:00:40] can you add those services to the khadoop scripts? [19:00:46] so they are restarted by default [19:01:12] hmm, yeah, but an27 is not a hadoop node atm, but yeah somehting like that shoul dhappen [19:01:14] http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Analytics%20cluster%20eqiad&h=analytics1005.eqiad.wmnet&v=0.03333150772188008&m=kafka_producer_KafkaProducerStats-webrequest-blog.ProduceRequestsPerSecond&r=hour&z=default&jr=&js=&st=1356029343&ti=kafka_producer_KafkaProducerStats-webrequest-blog.ProduceRequestsPerSecond&z=large [19:01:33] also good [19:01:33] uster%20eqiad&h=analytics1005.eqiad.wmnet&v=0&m=kafka_producer_AsyncProducerStats-webrequest-blog.AsyncProducerDroppedEvents&r=hour&z=default&jr=&js=&st=1356029343&ti=kafka_producer_AsyncProducerStats-webrequest-blog.AsyncProducerDroppedEvents&z=large [19:01:42] .. [19:01:58] what do i do with that second bit? [19:02:10] i had the exact same question!!! [19:02:20] and discussed it with ottomata and he won! [19:02:44] everything after uster%20 [19:02:51] psh [19:02:56] http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Analytics%20cluster%20eqiad&h=analytics1005.eqiad.wmnet&v=0&m=kafka_producer_AsyncProducerStats-webrequest-blog.AsyncProducerDroppedEvents&r=hour&z=default&jr=&js=&st=1356029343&ti=kafka_producer_AsyncProducerStats-webrequest-blog.AsyncProducerDroppedEvents&z=large [19:03:26] foooooodtime [19:03:41] wiieeeeee [19:05:39] arrghh another issue [19:05:41] No lease on /tmp/temp1724652392/tmp1948408304/_temporary/1/_temporary/attempt_1355947335321_0118_r_000000_0/part-r-00000 File does not exist. Holder DFSClient_attempt_1355947335321_0118_r_000000_0_1085940269_1 does not have any open files. [19:06:44] brb, grabbing lunch [19:12:27] I'm going out before the nearest store closes, bb in 1h [19:28:56] i lied I'm going to lunch now [19:51:29] I'm back [20:21:40] back [20:31:10] drdee, do you know if I can embed images in mediawiki with a remote url? [20:31:27] lunch, brb [20:31:33] ottomata: you cannot [20:32:01] booo [21:23:36] hey cool! [21:23:37] http://analytics1001.wikimedia.org:81/monitor/ [21:23:47] ah you don't need that 81 [21:23:48] http://analytics1001.wikimedia.org/monitor/ [21:42:37] drdee: I talked with Erik [21:43:04] drdee: it seems that back in 2010 the only separation made was mobile/non-mobile [21:44:44] but since then we have many new categories like "other mobile" "tablets" "Mobile applications" , "Browser version, tablets" "WAP" etc [21:44:51] these categories, or rather sets... [21:45:09] these sets have various relations between them, some are mutually exclusive, some are included in others [21:45:57] * average_drifter goes back to some bug-punching [21:49:18] drdee: a bit messy but Travis CI is now building Limn and running unit tests: https://travis-ci.org/wikimedia/limn [21:50:22] average_drifter: dschoon was saying that I can install a local version of npm on Jenkins [21:50:52] milimetric: did he mention how ? [21:51:09] no [21:51:10] :) [21:51:21] but that'd be great because I just got Travis CI working [21:51:27] dschoon: how do you install npm locally ? [21:51:33] so Jenkins would be very similar with the exception that we don't have npm [21:51:38] milimetric: does Travis also run qunit ? [21:51:41] yes [21:51:42] https://travis-ci.org/wikimedia/limn [21:51:45] wow [21:51:51] that's the last build with all the qunit tests passing [21:52:02] it doesn't provide a nice report of what tests failed, etc. [21:52:13] milimetric: I searched for "qunit" on that page [21:52:16] but that's a "nice to have" since people can just run the tests locally [21:52:17] I didn't get a match [21:52:36] search "138 tests" [21:52:43] it doesn't say qunit [21:52:56] oh [21:53:04] so it actually ran the qunit , nice :) [21:53:07] and I'm not sure what that nasty error at the end is, but it's not affecting it [21:53:08] yea [21:55:18] so .. how do you run the tests without qunit ? [21:55:21] phantomjs or somethin ? [21:56:41] milimetric: ^ [21:57:18] yes [21:57:53] https://github.com/wikimedia/limn/blob/develop/test/phantom.runner.js [21:58:17] average_drifter: that file lets you load the url and exit depending on success/failure of the whole suite [21:58:46] so in Travis I configured it to basically do: [21:58:46] npm start & sleep 3 && npm test ; killall node [21:59:24] which starts the server, gives it a sec to fire up, runs that file I linked with phantomjs, then cleans up the server process [22:00:37] cool [22:00:47] so it doesn't need a browser anymore [22:01:03] you can run tests now with just phantom. I think phantom had some browser engine behind it ? [22:01:28] phantom is v8 [22:01:31] it's headless webkit [22:01:53] or maybe I'm mistaken. I got to see phantom in 2011 .. almost 2013 now [22:02:09] ps http://casperjs.org/ [22:02:13]