[00:00:23] the first graph creation UI for limn dschoon: http://dev-reportcard.wmflabs.org/graphs/create [00:00:47] yeah [00:00:57] travis is pissy about config.json [00:01:04] i was gonna fix it, but... deliverables [00:01:33] milimetric: this is awesome [00:01:53] awesome work, milimetric! [00:02:08] it looks different from your demo. [00:02:24] datasources are below the graph controls... you get that too? [00:02:28] eh, I removed all the stupid pull right pull left crap [00:02:43] I couldn't get it to look right because as you're like adding metrics to the graph, everything flows funny [00:02:48] you're welcome to make it pretty, as always [00:02:50] *nod* [00:02:54] maybe i will! [00:03:01] after i finish the mobile session stuff, tho [00:03:07] basically, I've given up on making stuff pretty. I try to not make it horrid :) [00:03:13] yeah, def [00:03:26] if you do look at it, pull the datasources [00:03:40] I've updated all of them to the new format and added some metricDefs where they were needed [00:03:56] ori-l: yeah, i know. but i am pretty sure there are cases where: s = "foo"; s is not "foo" # True [00:04:12] i've run into it a few times. maybe it's only in multiprocessing land [00:04:26] erosen: the preview has some quirks like coloring and stuff [00:04:31] i think you're right that the important semantics involve interning [00:04:37] or maybe i'm just high. [00:04:37] I'm aware of that but it seems the graph itself is usually ok [00:04:55] do be brutal, I'm gonna sign off for tonight but I want a nice fat stack of complaints tomorrow morning erosen :) [00:05:16] i'll work on it [00:05:18] would you accept a nice, fat stack of patches instead? [00:05:30] i can totally add a few layers of abstraction [00:06:00] maybe support for defining new graphs using an OCaml DSL? [00:10:17] dschoon: you can apparently trick interning if you set your mind to it [00:10:25] >>> ''.join(list('foo')) is 'foo' [00:10:25] False [00:10:39] huh. [00:10:53] interesting. [00:11:01] it makes sense [00:11:33] there's no hint to the interpreter that the two may share content [00:12:35] *nod* [00:12:44] and it has optimizations for literals [00:12:55] >>> 'f'+'oo' is 'foo' [00:12:56] --> True [00:13:48] >>> s1 = 'foo' [00:13:48] >>> s2 = ''.join(s1) [00:13:49] >>> s1 is s2 [00:13:51] False [00:13:53] >>> id(s1) [00:13:55] 4299569832 [00:13:57] >>> id(s2) [00:13:59] 4299569912 [00:15:10] this is interesting because strings are immutable [00:15:25] you'd think that sequence of characters would have object identity irrespective [00:15:52] >>> hash(s1) [00:15:52] -4177197833195190597 [00:15:52] >>> hash(s2) [00:15:53] -4177197833195190597 [00:16:01] as epxected. [00:17:21] you'd think that sequence of characters would have object identity irrespective [00:17:22] dunno why [00:17:26] anyways, [00:17:29] >>> 'f'+'oo' is 'foo' [00:17:49] exactly. it's it checking the intern cache based on the sequence? [00:18:01] that gets optimized away by the parser [00:18:30] mm. [00:18:35] >>> def f(): [00:18:35] ... return 'f' + 'oo' [00:18:35] ... [00:18:35] makes sense. [00:18:37] >>> import dis [00:18:39] >>> print dis.dis(f) [00:18:41] 2 0 LOAD_CONST 3 ('foo') [00:18:43] 3 RETURN_VALUE [00:18:45] None [00:18:53] what is dis dis you speak of. [00:19:03] huh! [00:19:07] dis..cover for yourself:P [00:19:20] just was looking at it [00:19:22] pretty cool [00:19:28] i am not at all displeased! [00:20:02] * ori-l disperses [00:20:53] i've played around a bit with byteplay [00:21:12] i somehow did not know about dis, despite it, uh, shipping with python [00:21:42] i had a lot of fun with inspect when first learning python [00:24:10] this, for instance, is a TERRIBLE idea: [00:24:14] def find_calling_instance(Type): [00:24:14] frame = inspect.currentframe().f_back [00:24:14] try: [00:24:15] while frame: [00:24:17] for v in frame.f_locals.values(): [00:24:20] if isinstance(v, Type): return v [00:24:21] frame = frame.f_back [00:24:23] finally: [00:24:25] del frame [00:40:32] oh, neat [00:40:35] HaeB: ping [00:40:47] hi [00:41:10] mutante synced all the blog changes, but there was an hour during which both implementations were up [00:41:28] so basically you'll have two hours missing and one hour doubled [00:41:31] from today [00:42:00] if the traffic patterns were right maybe it all balances out :P [00:48:17] see, when that happens to me, i end up with 25G of doubled data, and sorting things out requires implementing a while-loop in xml [00:49:12] i feel like if i could understand why people thought xml was a good idea, i'd be like inches from the level of empathy required for sainthood [01:02:28] drdee, milimetric: ...did either of you just kill my mobile-backfill coordinator? [01:02:35] no [01:02:36] please say yes. [01:02:42] i don't like ghosts or gremlins. [01:02:49] i am a delicate flower! [01:03:31] created many cards, I have [01:10:50] thx average [01:27:53] drdee: other than tomasz, who else would care about #61 being done? [01:28:15] i'm cc'ing analytics-l [01:28:22] as well as you and kp [01:28:24] primarily tomasz, maybe some other mobile team members [01:39:43] suggestions, drdee? [01:50:09] drdee: how is udp-filter used in production ? [01:50:12] with what params I mean [01:50:14] just curious [01:50:17] I'm trying something out [01:51:41] average: ssh oxygen.wikimedia.org -- cat /etc/udp2log/oxygen [01:52:58] permission denied [01:53:02] dschoon: can you gist it pls ? [01:54:28] you can find the files in puppet [01:54:40] check the project out of gerrit [01:55:05] but for example, here's the filter for TeaHouse: [01:55:08] pipe 10 /usr/bin/udp-filter -F '\t' -d en.wikipedia.org -p /wiki/Wikipedia:Teahouse >> /a/log/webrequest/teahouse.tsv.log [01:55:22] GLAM: [01:55:22] pipe 10 /usr/bin/udp-filter -F '\t' -p _NARA_ -g -m -b country >> /a/log/webrequest/glam_nara.tsv.log [01:55:30] the API log: [01:55:31] pipe 100 /usr/bin/udp-filter -F '\t' -p /w/api.php >> /a/log/webrequest/api-usage.tsv.log [01:55:54] the ones on oxygen are all zero-partner IP ranges [01:55:56] not exciting. [01:56:03] thank you very much [01:56:06] (those were all from emery) [01:56:16] you've just opened a door to some things :) [01:57:29] drdee: https://mingle.corp.wikimedia.org/projects/analytics/cards/591 [02:01:16] dschoon: just read 591 [02:01:32] dschoon: mind you, there is a new version of dclass from Rezan Naghibi [02:01:43] we should test it out! [02:01:48] and maybe OpenDDR databases also have new updates [02:01:58] dschoon: make a card please [02:02:17] maybe OpenDDR has paid their hosting bill :P [02:02:32] NOPE [02:02:38] http://openddr.org/ [02:04:16] dschoon: maybe just download the new openddr , convert to .dtree , replace with what you're currently using and try it out [02:04:26] as for dclass, latest is 2.2.0 from 9-April-2013 ! https://github.com/TheWeatherChannel/dClass/commits/master [02:04:30] some day. [02:04:53] I needed to open that ticket before sending out #61. [02:40:26] average: udp-filter has a man entry [02:40:38] but indeed as dschoon pointed out check puppet as well [02:40:53] dschoon, thx for card 591 [02:41:07] giant. sticky. mess! [02:41:17] oh, wait. that's the dclass one, isn't it [02:43:23] average: dclass has now native java support [02:48:00] oh ? :) [02:49:50] didn't know [13:50:38] moooorning guys [13:58:42] drdee: morning [14:00:26] hey [14:01:13] oh didn't see you there :) [14:01:14] morning [14:01:30] drdee what do you mean in your email by "not sure if this is actually possible"? [14:01:37] Whether or not the create a graph card is done? [14:01:42] if you are ready to demo or not [14:01:49] yeah, we can demo [14:01:52] cool! [14:01:55] Evan gave me a big list of feedback [14:02:01] MVP :) [14:02:02] hangout? [14:02:04] sure [14:02:07] k, one sec [14:02:59] what?!! Why would you call me a minimally viable player? [14:07:17] most valuable player! [14:43:13] :P [14:43:21] I forgot to do that like 40 minutes ago :) [15:15:26] milimetric; we can close https://github.com/wikimedia/limn/issues/84 right? [15:15:49] I haven't personally verified it [15:16:03] I'm in the middle of a change but I will check and close right away [15:17:06] maybe we can add the link to the commit that fixed it [15:22:12] yep, it was this: https://github.com/dsc/limn/commit/7a6cc61956b1eddc6946ee60c4062d431bec6513 [15:45:58] it's time to push to wikimedia/limn again so it looks active :) [15:59:12] mornin [15:59:16] ottomata, you about? [15:59:35] yuppershi [15:59:43] i'm about to head into the office, but [15:59:44] morning dschoon [15:59:49] (maybe drdee as well) [15:59:54] have either of you seen this before? http://analytics1010.eqiad.wmnet:8088/proxy/application_1366081130423_0783/mapreduce/attempts/job_1366081130423_0783/r/FAILED [15:59:57] i have no idea what to make of it [16:00:07] i don't like the big FAILED [16:00:08] (change that to localhost:8888) [16:00:38] buh, sorry, it just moved the logs [16:00:39] here: http://analytics1010.eqiad.wmnet:19888/jobhistory/task/task_1366081130423_0783_r_000000 [16:00:49] http://analytics1010.eqiad.wmnet:19888/jobhistory/logs/analytics1015:8041/container_1366081130423_0783_01_000025/attempt_1366081130423_0783_r_000000_0/stats/syslog/?start=0 [16:01:30] Caused by: java.lang.ClassCastException: org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl$CompressAwarePath cannot be cast to java.lang.Comparable [16:01:31] ? [16:01:50] i know! [16:01:53] what the hell! [16:02:05] outfile compression isn't even on! [16:02:21] and further, how can it NOT be comparable?! [16:02:31] that sounds like a bug [16:02:33] dunno what MergeManagerImpl$CompressAwarePath [16:02:33] is [16:02:46] me either. [16:02:57] but http://analytics1010.eqiad.wmnet:19888/jobhistory/conf/job_1366081130423_0783 [16:03:04] isn't;t that the CDH 4.2 bug? [16:03:05] filter key on compress [16:03:12] what do you mean, drdee? [16:03:17] 1 sec [16:03:18] me googling [16:03:24] the output is not big, btw [16:03:29] 800M or so [16:03:51] yeah i think it's this one: [16:03:51] https://issues.cloudera.org/browse/DISTRO-461 [16:03:52] i told you about this beore [16:04:04] we are waiting for 4.2.1 [16:04:12] there is a custom jar which we could put in [16:04:15] for the moment [16:04:18] see jira ticket [16:04:27] okay [16:04:38] heading into the office -- we shall continue in 15 [16:05:15] the jira ticket contains an updated hadoop-mapreduce-client-core-2.0.0-cdh4.2.0.jar [16:05:18] which we can download [16:31:23] [travis-ci] develop/a1b99fb (#128 by milimetric): The build is still failing. http://travis-ci.org/wikimedia/limn/builds/6419850 [16:32:14] something fishy is going on with kripke [16:32:20] I don't wanna say it's the gluster problem [16:32:26] but it might be the gluster problem [16:32:32] noooo that would never happne [16:32:37] also - please for the love of all that's good nobody restart it [16:32:38] :) [16:35:12] drdee: yep. [16:35:14] that's totally the bug [16:35:17] nice catch. [16:36:23] ottomata: can we get that jar on the cluster, say, er, now? [16:36:33] it'd be pretty cool if the session stuff worked as a result [16:36:41] mingle card: https://mingle.corp.wikimedia.org/projects/analytics/cards/595 [16:36:52] most mingly [16:37:26] should this be 'expedited'? [16:37:28] ottomata: the jar is at https://issues.cloudera.org/browse/DISTRO-461 [16:37:29] i think it should [16:37:30] drdee: yes. [16:37:34] k [16:39:09] milimetric: why are you worried about kripke right now? [16:39:19] just deployed my latest for the demo [16:39:58] is something wrong? [16:40:31] [travis-ci] develop/fe5c3bb (#129 by milimetric): The build is still failing. http://travis-ci.org/wikimedia/limn/builds/6420118 [16:41:40] ^^ milimetric [16:41:56] eh... nothing yet [16:42:01] but this is what happened last time [16:42:06] basically sudo npm install gives an error [16:42:13] something about not being able to write to your home directory [16:42:23] heh [16:42:23] then you're like - oh weird, lemme restart [16:42:26] BOOOOOOOOOOOOOOOOOOM [16:42:36] i ... do not understand [16:42:37] :) [16:42:40] as i have never had this problem [16:42:56] but if you'd like, let me know when you're done and i can double-check things [16:42:56] well so gluster manages the home directories on labs or something [16:43:03] ahh [16:43:03] right [16:43:13] i think it's an nfs mount, acutally [16:43:22] try to login to kripke and do sudo npm install inside the test-reportcard limn folder or something [16:43:32] if it works for you, then it's probably just me [16:43:35] but it doesn't matter for now [16:43:39] (gluster managers some sort of hypervisor settings or whatever) [16:43:41] brb a sec [16:43:50] the key thing is: don't touch anything because the box might blow up and we might lose everything if we do [16:43:54] KEY! [16:44:02] yes :) [16:44:04] brb [16:44:47] right, I should've said that, sorry. ottomata: it is possible that kripke will cease to function upon the next restart. That rsync we were talking about might be wise to do [16:44:57] just to get everything off so we have at least some backup [16:46:48] and yes, i'm aware of this issue :) [16:46:59] the new wifi is stupid. [17:05:37] ottomata: is hue actually puppetized 4realz? or is the banner at the top of the conf files just an artifact of the original setup? [17:05:51] if it's not in op-puppet, we should probably commit the conf to kraken/etc [17:06:52] average: can you expand on card #583? i don't know what you mean by it [17:14:59] drdee: do you know if hive is working? [17:15:29] dschoon: what's the new wifi? [17:15:34] i saw some email but forgot [17:15:46] the WMF network [17:15:50] just new auth [17:16:26] dschoon: yes hive is working [17:16:44] it just times out for me [17:16:52] but you first need to create tables that mirror the data structure of the log files [17:16:56] well, hue [17:17:00] mmmm; that's new [17:17:01] the beeswax UI times out [17:17:15] http://localhost:8888/beeswax/table/default/webrequest_100_all [17:17:15] is the hive server running? [17:17:18] yes [17:17:20] (i checked) [17:17:22] k [17:17:31] it used to work prior cdh 4.2 [17:17:46] maybe something happened during the upgrade? [17:18:13] but create a mingle card :D [17:19:00] boo [17:19:18] i'll restart it and see what happens. [17:19:23] i guess i could, you know, look at the logs [17:21:14] yes apparently that's one of their raisons d'etre, they like to be looked at :D [17:39:10] fixed hive. [17:39:17] it was actually a beeswax/hue problem [17:59:59] dschoon can you make sure that the hangout is running in SF? [18:00:06] on my way now. [18:06:41] ottomata: are you still in the hangout? [18:07:42] i'm in the new one [18:07:49] https://plus.google.com/hangouts/_/5b70172d0f7418695ff6d98f3cb53bbb7097e020 [18:07:55] hmm [18:59:57] drdee: what's the best view in mingle of where https://mingle.corp.wikimedia.org/projects/analytics/cards/60 is sitting now ? [19:00:18] tfinc: you mean the current status of it ? [19:00:44] average: yup. we talked about it moving back from "done" to another state and i was wondering where i could keep track of that [19:01:10] tfinc: http://analytics.wmflabs.org/reports/r52/out_sp/EN/TablesPageViewsMonthlySquidsMobile.htm [19:01:16] tfinc: this is the latest report [19:01:40] tfinc: you can see it featuring as a link in all the reports on stats.wikimedia.org (for example in http://stats.wikimedia.org/wikimedia/squids/SquidReportCrawlers.htm ) [19:02:13] this is listed as a report by Author:Erik Zachte (Web site) [19:02:20] tfinc: we are counting only wikipedia.org now [19:02:20] which makes me think wikistats [19:02:31] tfinc: no, I generated it [19:02:54] tfinc: if you want I can put my name on it [19:03:05] average: yeah, clean up the footer to remove any confusion [19:03:13] tfinc: ok [19:04:41] tfinc: we only counting wikipedia.org now . The latest documentation is available in http://www.mediawiki.org/wiki/User:Spetrea/New_mobile_pageviews_documentation [19:04:46] average: will the report include current count month to date just like the old one did ? [19:05:06] yup, i saw that on the page [19:05:06] tfinc: I talked with Erik today about adding the prediction for the current month yes [19:05:27] average: what will it take to run it for the other projects ? [19:05:49] tfinc: you mean separated reports for each of the projects right ? [19:06:00] yup, i'm happy to put in a card for it [19:06:18] tfinc: sure, it's not hard to add the logic for that [19:06:32] tfinc: I'd say it could be done in 2-3days at most [19:06:43] cool. i'll put in a story about it [19:06:54] ok, please tell me the card after you write it [19:07:48] sure, i'll use drdee as point though so that we have on central contact [19:07:57] so that he can schedule it [19:08:27] ok [19:08:38] and what will it take to move it to stats. ? [19:09:01] tfinc: Erik is working on having it on stats [19:09:32] tfinc: our workflow is 1) we run the reports on stat1 2) there is a script which rsync's to stat1001 which hosts stats.wikimedia.org [19:15:59] drdee: average : where in mingle can i check know its progress? [19:16:15] finding reports on stats. is pretty hard [19:16:32] tfinc: https://mingle.corp.wikimedia.org/projects/analytics/cards?favorite_id=758&view=%3EWIP+-+Feature+Analysis but i am in sprint planning [19:17:57] tfinc: please refresh the 60 card in your browser, I added an update [19:18:08] at the bottom of the page [19:18:14] drdee: https://mingle.corp.wikimedia.org/projects/analytics/cards?favorite_id=758&view=%3EWIP+-+Feature+Analysis doesn't seem to have it. [19:18:32] average: thanks [19:18:42] np [19:29:21] brb [20:01:51] brb smoothie [20:08:00] drdee: sorry for dropping out of hangout--internet disappeared... [20:08:09] no worries [20:19:06] back [20:39:12] k, i'm heading out, gonna drive back to NYC [20:39:15] laters all! [21:12:53] uh [21:13:01] i think i should head home for the rest of the day [21:13:05] otehrwise i am going to infect everyone [22:04:12] [travis-ci] develop/9e711d6 (#130 by milimetric): The build is still failing. http://travis-ci.org/wikimedia/limn/builds/6429041 [22:09:25] nite everyone, I'm gonna go play video games :) [22:16:26] laterz milimetric [22:40:07] holy shit i feel like ass. [22:40:25] i am *so* sorry to anyone in our meeting today if you get sick [23:44:28] New review: Tim Starling; "(1 comment)" [analytics/log2udp2] (master) - https://gerrit.wikimedia.org/r/58449