[13:05:46] morning guys [13:05:52] girls [13:32:42] morning [13:32:46] thx for email dan [13:32:53] does this mean i can close the bugzilla ticket? [13:33:24] ottomata! good morning [13:33:38] so faidon gave you +2!!!! on limn puppet but he didn't' merge :( [13:33:54] nice job! [13:33:58] moorning! [13:34:05] yayyy [13:34:39] milimetric ^^ [13:34:47] woo! [13:34:59] ottomata: I especially liked Faidon's "I really like this" [13:35:13] just want you to know I was very proud of you when I read that [13:35:13] yeah that's hard won [13:35:20] me too [13:35:35] I'll wait for confirmation from some people before we close that bug drdee [13:35:42] k [13:39:02] haha, aw [13:42:57] so drdee, I saw that we're probably collecting everything we need and that our mobile app count is probably wrong for some other reason? [13:43:35] "probably collecting everything" [13:43:42] uhmmm [13:43:49] we need to be sure :) [13:51:47] milimetric: there is the issue that ssl api calls do not contain the query string in the log files so that's one source of inaccuracy [13:52:15] maybe we should grep those wmf app user agent strings in the mobile sampled files and see how often we find them there [13:52:31] ok [13:52:51] so thats in the sample/../mobile/*.log.gz files? [13:53:18] yes, ottomata, did you recently rsync the mobile sampled log files to stat1? [13:54:53] i started to yesterday, lemme check em [13:55:10] yeah, big gap there though [13:55:18] the last few days are there [13:55:24] /a/squid/archive/mobile [13:58:16] cool ty [13:59:32] if you need more recent stuff, I htink some exists on locke [13:59:40] i probably should copy that over to stat1, eh? [13:59:59] i don't think it matters too much [14:00:07] we're just trying to get a feel for counts [14:00:09] k [14:01:02] so, for example, mobile.log-20121209.gz is all data from December 9th, sampled 1/1000? [14:01:06] or not sampled? [14:02:44] sampled 1/100 [14:03:04] i should make the log name say that, eh? [14:03:09] I'll fix that on the new ones [14:03:32] oh it is fixed on the new ones [14:03:50] new ones will be called mobile-sampled-100.tsv.log [14:04:50] milimetric, do you prefer /usr/local/share/limn or /srv/limn [14:04:52] for limn install? [14:05:27] I have no preference for that, because if you asked me I'd say C:\Program Files\Wikimedia\Limn [14:05:29] :P [14:06:17] IOW, you and Faidon are the LInux conention dudes [14:06:21] *convention [14:07:53] hmm, reading FHS, i think /usr/local/share is better, even though Faidon likes /srv [14:08:00] i think he likes /srv because limn is a wmf thing [14:08:02] hmm, [14:08:09] i'm going to keep it in /usr/local/sahre [14:10:30] milimetric, where is reportcard running right now? on what labs instance? [14:15:52] the instance is called reportcard [14:16:17] ottomata ^ [14:16:37] k danke [14:16:38] got it [14:33:29] ok, so I grepped around a bunch drdee [14:33:40] k [14:33:43] the most I ever got in a day was 2800 [14:33:45] the least was 900 [14:33:52] for WikipediaMobile/.*Android [14:33:59] the average looks around 1400 [14:34:05] *1000 [14:34:05] so that's *100 [14:34:10] otto said 100 [14:34:13] ok [14:34:21] so that's at least 90000 [14:34:43] so we should find way more in kraken [14:34:46] divided by 96 because we're looking at 15 minute increments, around 1000 [14:35:19] which comes sort of maybe close to the total we got when we were ignoring IS_PAGEVIEW [14:35:22] I think we got 500 [14:35:38] i think we are still off [14:35:58] i'll have to look at more than just exact 15 minute span :) [14:36:05] that could definitely be a problem too [14:36:06] true [14:36:29] also, grepping is much faster than pig [14:36:32] but the other platforms also need to be looked at [14:36:35] yes [14:36:39] not for testing [14:36:44] like much faster period [14:36:56] if you are operating on a small enough bit of data [14:37:03] grepping is gonna be faster for sure [14:37:10] so does pig scale better? [14:37:13] yeah [14:37:37] i'm going to make a up a rule of thumb that may totally be false: [14:37:38] k, makes sense [14:37:44] if you've got more data than fits in memory [14:37:49] hadoop will be better [14:38:47] hm, I'd challenge that since grep doesn't keep anything in memory, and it would be totally inadequate to do anything besides counting and parsing [14:39:34] I think the distributed part is where it starts shining. Building a way to coordinate grepping across a cluster does not sound fun :) [14:40:04] ha, yea [14:52:54] " Installing reportcard.wikimedia.org on stat1001." [14:52:56] WOOOOOOOOOOT [14:53:07] hehe [14:53:10] we'll see :) [14:54:28] woot for the attempt :) [14:55:57] yeah this is awesome [14:56:13] either way, we have puppetized Limn man! Nothing else matters [14:56:18] haha [14:56:24] The world is our oyster [14:56:32] milimetric, mind if I try new limn puppet stuff on reportcard instance? i might take it down for a sec [14:56:49] isn't that self-hosted? [14:56:58] yeah, but i'm going to checkout from my latest commit [14:57:01] oh ok [14:57:03] sure, no prob [14:57:15] i don't think people really hit reportcard when it's not around the metrics meeting [15:15:44] milimetric: updated https://github.com/wikimedia/metrics/tree/master/pageviews/kraken [15:16:15] I haven't committed my kraken java pageview changes [15:16:26] i wanted to figure out the problem first [15:16:46] but thanks for the update [15:21:52] k [15:22:00] brb coffeee TIME [15:38:22] gm [15:50:14] yeehaw, reportcard.wmflabs.org is running off of the exact same puppet class as stat1001 will use to run reportcard.wikimedia.org [15:50:15] so fancy! [15:50:29] wanna get a review of that latest puppet class, but it should be easy [16:07:02] self-merge it ottomata :) [16:09:31] ha, yeah i can [16:09:42] i'm just not sure about my misc::limn::instance define [16:09:49] well, its location is the only part i'm not sure about [16:24:22] mornin [16:24:38] milimetric: do you think there's a problem with your regex in java, then? [16:24:54] no, I doubt it [16:25:01] it's a pretty plain straightforward regex [16:25:02] you read the docs for Pattern? [16:25:20] I assume .* means any character 0 or more times :P [16:25:20] http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html [16:25:35] the only tricky regex is the iPhone mobile app one [16:25:37] here is why i ask. [16:25:38] http://docs.oracle.com/javase/7/docs/api/java/util/regex/Matcher.html#matches() [16:25:42] because i'm using negative lookahead [16:25:52] Matcher#matches() *must match the entire input* [16:25:55] we're not using matches, Diederik had coded it with matcher().find() [16:25:57] it cannot be a fragment of a line [16:26:03] and I left it that way [16:26:12] there are unit tests that cover these concerns too [16:26:14] do you reset between calls? [16:26:18] written by Diederik and updated by us [16:26:46] a new matcher is created [16:26:50] lemme double check that [16:27:01] i'm just sayin [16:27:09] try matching when the pattern doesn't start at the beginning of the line [16:27:15] make sure it doesn't fail [16:27:27] the MATCHES keyword in pig *does* [16:27:31] it has to cover the whole line [16:28:01] https://github.com/wikimedia/kraken/blob/master/kraken-generic/src/main/java/org/wikimedia/analytics/kraken/pageview/UserAgent.java#L142 [16:28:42] I'll add tests to cover that even though they don't fail right now [16:28:49] yeah [16:29:57] ottomata: where is the checkout of the cluster config to update for deploy? [16:29:59] i know an10 [16:30:04] but where on disk? [16:30:12] i want to bump memory to 32 [16:30:33] kraigparkinson, dschoon: https://plus.google.com/hangouts/_/97b1a231706418d1b3f6ec3a6175b43caa200e7b [16:30:57] yep, all tests pass [16:31:11] ottomata: go to what drdee said [16:33:20] ottomata: comeeee [16:38:15] ottomata: https://mingle.corp.wikimedia.org/projects/analytics/cards?favorite_id=756&view=%3EWIP+-+Problems [16:40:06] oh shit dschoon, I just remembered your hack last month to make the stacked charts work (aligning the data) [16:40:27] I can't remember if we have to do anything to fix that [16:40:50] we do. [16:49:00] drdee, kraigparkinson: updated https://mingle.corp.wikimedia.org/projects/analytics/cards/472 [16:49:28] milimetric can you also have a look at https://mingle.corp.wikimedia.org/projects/analytics/cards/472 ? [16:50:37] nice. beat me to it [16:50:54] chowing down on some panang [16:50:56] in a bit [16:53:34] nice, well said whoever quoted me :) [16:54:10] :) [16:54:11] <-- [16:56:21] ottomata: "alerts on failed kafka hadoop consumer imports" great idea [16:59:32] drdee, ottomata: https://mingle.corp.wikimedia.org/projects/analytics/cards/476 [17:00:03] yeha, dschoon, i was going to work on that today [17:00:06] 476 [17:00:16] should be an hours worth of time maybe [17:03:06] milimetric, average_drifter: SCRUM! [17:07:28] I won't be able to get to the standup [17:07:37] I won't be able to get to the standup [17:07:50] I'll get home in ~2h [17:21:56] ottomata, drdee https://mingle.corp.wikimedia.org/projects/analytics/cards/477 [17:44:22] drdee, tell me exactly what you want to know, so I can be sure how to ask [17:49:11] does 100% of the wmf mobile apps stuff get proxied to the mobile varnish servers [17:49:45] drdee: what. no querystring?! [17:49:54] for ssl traffic [17:49:58] why! [17:50:01] how! [17:50:04] ohh [17:50:07] wait. [17:50:08] no! [17:50:10] this makes no sense! [17:50:18] look in the logs [17:50:19] it has to get proxied back to a squid or a varnish [17:50:24] and those log the data, right? [17:50:31] then nginx is not setting it [17:51:30] then the request would completely fail 100% of the time [17:51:43] is all SSL API traffic is broken? [17:51:52] i think we'd know that [17:51:57] so this cannot be true. [17:52:13] it sounds like just the logs from *nginx* do not contain the qs [17:52:23] which is fine -- we have the logs from the next hop [17:52:31] we should be ignoring nginx anyway as it creates double-counts [17:56:38] on a related note, i think i need a muffin. [17:56:44] back in a moment. [17:56:47] (avec muffin) [18:02:25] muffins: success [18:06:04] yes! milimetric, reportcard puppetization works! [18:06:06] its ready on stat1001! :) [18:06:08] woo! [18:06:19] just neesd a deploy and a domain : [18:06:32] on that subject, ottomata~ [18:06:39] when you make the DNS changes/request [18:06:51] can we stake out a few subdomains and point them at stat1001? [18:07:16] the fact that metrics.wikimedia.org is an API is misleading [18:07:24] but if it's first-come, first-serve, we better get on it [18:07:46] haha, it usd to be metrics-api, but they wanted metrics [18:07:48] but, yes, I can do that [18:07:54] analytics.wikimedia.org data.wikimedia.org [18:07:57] whatchu want? [18:08:33] i'm cool with analytics [18:08:36] any other idaes? [18:08:37] i might get asked about data [18:08:39] and whats it for [18:08:46] a place to host datasets [18:08:55] *shrug* [18:09:01] just point them all at stat1001 :) [18:09:36] Eeeegggg, i dunno, i'm a-scerrred [18:09:41] i'll do analytics [18:10:25] okay. [18:15:29] (that's in addition to reportcard.wikimedia.org, pls) [18:15:39] ja [18:16:22] ottomata: is your kraken-public cron checked in somewhere? [18:16:32] if i want to make changes/additions to it, where do i do that? [18:16:46] also: do you know about hadoop fs -getmerge ? [18:16:52] no, not checked in [18:17:08] i do not know about -getmerge! [18:17:38] cron is at [18:17:41] an27 [18:17:47] in stats user's crontab [18:18:03] oh wait, is it checked in? [18:18:11] oh it sure is! [18:18:14] in puppet [18:18:21] manifests/misc/analytics.pp [18:18:51] its my limmification cron job that isn't checked in [18:19:03] ahh [18:19:03] okay [18:19:26] gonna try to write a generic workflow for a merge-n-rename job [18:20:54] cool [18:21:24] ! [18:21:27] -getmerge! [18:21:46] how does it sort them? [18:21:50] like, sorts the contents? [18:21:53] i assume name order [18:21:53] or maybe sorts by filename [18:21:57] yeah probably [18:22:05] unfortunately, it only does hdfs -> local [18:22:19] yeah, but [18:22:22] check out my cronjob on an02 [18:22:28] it probably does what you might be doing [18:22:34] no [18:22:39] /opt/kraken/bin/zero_carrier_limnify_hdfs_put.sh [18:22:39] i want to go hdfs->hdfs [18:22:48] yeah [18:22:54] i mean, partly what you are doing, but you could make it generic [18:23:01] https://github.com/wikimedia/kraken/blob/master/bin/zero_carrier_limnify_hdfs_put.sh [18:23:08] hm [18:23:09] though [18:23:14] i guess you're rsyncing [18:23:46] i just really want to be able to define everything in one place [18:23:50] one coordinator [18:23:56] yeah [18:24:12] wait, you mean a single coordinator for all of this type of work? [18:24:27] pig script -> concat sort [18:24:41] no, it would be a workflow [18:25:03] that you can append to your coordinator, and parameterize to do what you want [18:25:12] i want to move toward us writing workflows like you would functions [18:25:23] it takes arguments and performs a task [18:25:24] ah yeah [18:25:25] yeah [18:25:29] sounds good [18:25:40] yeah most of those workflows in kraken oozie dir right now are very non DRY [18:25:45] would be easy to abstract those [18:25:47] yes. [18:25:50] i'm working on that. [18:25:56] and the scripts as well [18:26:01] but, I think we could do the same with the coordinator files too, they are mostly very similar [18:26:04] there was just a lot we didn't know [18:26:05] yes. [18:26:12] yeah awesome! [18:26:26] most of the stuff could be defined as variables in the .properties files [18:26:40] even the dataset stuff could I think [18:26:42] so if you look at https://github.com/wikimedia/kraken/blob/master/pig/mobile_device_props.pig [18:26:42] maybe... [18:26:43] hmm [18:27:00] like [18:27:01] date_bucket_format [18:27:04] i've added parameters for the format as well as the regex [18:27:11] like it [18:27:12] so it doesn't have to apply to a fixed freq [18:27:17] love it [18:27:36] dschoon: i am suggesting to drop https://mingle.corp.wikimedia.org/projects/analytics/cards/352 (wikistats search box) don't think we either have the skills and time to invest in this , are you ok with that? [18:27:42] had meant to do that too, well, at least with abstracting the regex so it wasn't named 'hour' or 'day' since it really doesn't matter [18:27:42] but i like it a lot [18:28:04] no I don't htink we should do that [18:28:15] drdee: that was just something EZ and I worked on while he was here [18:28:25] it was never meant as a big investment of time [18:28:28] what's the status? [18:28:40] we brainstormed. [18:28:47] i wrote some CSS to make things easier to read. [18:28:52] what should i do with the card? [18:28:53] nothing more than that. [18:28:57] leave it in the backlog. [18:29:09] ottomata: https://github.com/wikimedia/kraken/blob/master/oozie/mobile/device/props/workflow.xml [18:29:15] though it's all commented out [18:29:26] the idea was that the workflow took a bunch of params, as you were staying [18:29:35] rollup targets, etc [18:29:54] the coordinator wasn't fun, but it worked https://github.com/wikimedia/kraken/blob/master/oozie/mobile/device/props/coordinator.xml [18:30:10] the OOM forced me to remove it [18:30:46] dschoon: card https://mingle.corp.wikimedia.org/projects/analytics/cards/439 : can't we infer that from the referer field? [18:30:57] (Log SSL Traffic as a Header [18:30:57] ) [18:31:14] no? [18:31:25] 1. I think SSL usually strips the referer [18:32:12] 2. it gets reverse proxied to machines that have no way of knowing otherwise [18:32:25] the squid/varnish referer entry would start with https://wikimedia something [18:32:52] i don't think they have a referer [18:33:02] i think the browser removes the field [18:33:23] but anyway [18:33:32] not all SSL traffic starts SSL [18:33:46] it'd be a bad metric [18:33:50] we should just do it right if we're going to do it [18:33:59] and setting X-Forward-Proto isn't hard or verbose [18:34:24] ottomata: man, now THIS was a manly job: http://localhost:19888/jobhistory/jobcounters/job_1364239892421_0177 [18:35:04] ok :) just trying to reduce our workload ;) [18:35:18] it's low priority [18:35:25] there's nothing wrong with it staying in the backlog forever [18:35:30] that's the point of the backlog [18:35:35] it's not like Inbox 0 [18:35:36] :) [18:35:50] and certainly the backlog is not First Come, First Served [18:37:13] no it's more like b(l)ack(h)log(e) [18:37:20] :D [18:37:23] :P [18:37:26] sometimes [18:37:39] but one of the goals of grooming is for us to go through it regularly [18:37:44] and partially order things [18:38:07] btw, everyone [18:38:20] drdee, ottomata, milimetric -- we should all make sure we're familiar with http://oozie.apache.org/docs/3.3.1/WorkflowFunctionalSpec.html#a3_Workflow_Nodes [18:38:40] k [18:38:48] that section lists the basic building-blocks beyond "pig script" we have to work with in jobs, when building a dataflow [18:38:52] (1on1) [18:38:56] kk [18:39:03] just pushing that on the stack for everybody [18:39:15] ottomata: when you get a chance, [18:39:16] RFaulkner-WMF:~ rfaulkner$ curl http://metrics.wikimedia.org [18:39:16] curl: (7) Failed to connect to 2620::861:2:208:80:154:155: No route to host [18:39:17] RFaulkner-WMF:~ rfaulkner$ curl http://metrics-api.wikimedia.org [18:39:18] curl: (7) Failed to connect to 2620::861:2:208:80:154:155: No route to host [18:39:37] ip6? [18:39:46] rfaulkner_: export http_proxy http://brewster.wikimedia.org:8080 [18:39:48] yeah sorry [18:39:48] no it smy fault [18:39:48] i'm fixing [18:39:48] but in 1on1 [18:40:00] that'll work for http for now [18:40:07] not https or ipv6 (i think) [18:40:09] ok fixed rfaulkner [18:40:15] or ignore me :) [18:40:53] awesome, thank you :) [18:53:59] btw, ottomata [18:54:26] lmk when you're game for changing some cluster params [18:54:35] i think we should re-enable jvm reuse [18:54:42] as that was not the source of our memory issues [18:56:52] i actually have quite a few changes [18:56:55] but i dunno where to put them [19:04:57] if its just for hte hadoop stuff, i'd say go ahead and commit them to kraken/etc [19:05:07] mk [19:05:08] and i'll help you figure out best way to deploy them in a little while [19:05:12] can we do the update together? [19:05:14] yeah [19:05:14] awesome [19:05:17] sounds perfect [19:05:26] you're gonna love this workflow, btw [19:05:38] the fs action totes has all the tools we want [19:05:44] COOOL [19:05:45] hahah [19:08:55] you can also specify formal parameters for your workflow [19:09:01] (you know, like a function!) [19:09:05] and it'll check them as preconditions [19:09:16] you can also provide defaults [19:09:30] so we don't have to fill in the goddamn namenode/jobtracker in every job [19:14:59] guys, I'm attending the meeting in 15 minutes about Explorable Explanations [19:15:00] http://worrydream.com/#!/ExplorableExplanations [19:15:06] it sounds super relevant [19:17:30] milimetric: https://mingle.corp.wikimedia.org/projects/analytics/cards/478 [19:17:48] drdee, I'd highly recommend ^^ [19:17:54] Explorable Explanations [19:17:58] yes i've seen it [19:18:01] very cool [19:18:12]