[06:44:15] New patchset: Rfaulk; "mod. add dependecies for handlers." [analytics/user-metrics] (repair_runtime) - https://gerrit.wikimedia.org/r/72896 [06:44:40] Change merged: Rfaulk; [analytics/user-metrics] (repair_runtime) - https://gerrit.wikimedia.org/r/72896 [07:34:07] New patchset: Rfaulk; "add. create broker context." [analytics/user-metrics] (repair_runtime) - https://gerrit.wikimedia.org/r/72898 [07:34:07] New patchset: Rfaulk; "mod. Write to broker on request rather than multiprocessing queue." [analytics/user-metrics] (repair_runtime) - https://gerrit.wikimedia.org/r/72899 [07:34:46] Change merged: Rfaulk; [analytics/user-metrics] (repair_runtime) - https://gerrit.wikimedia.org/r/72898 [07:35:10] Change merged: Rfaulk; [analytics/user-metrics] (repair_runtime) - https://gerrit.wikimedia.org/r/72899 [13:38:47] morning [13:38:54] Hi milimetric [13:39:08] good afternoon qchris [13:39:11] :) [13:43:45] morrrrning! [15:48:16] drdee, 796 done. [15:48:19] i put it in shipping [15:50:30] you mean 794? [15:50:50] oh no you are right [15:51:02] no 796 [15:51:02] ja [15:51:32] gonna get food before stand up again, back in a bit [15:51:42] no standup [16:00:38] milimetric, can you demo bytes added for wikimetrics? [16:01:02] we can demo tests [16:01:13] so just show the tests we wrote for it and the fact that it passes them [16:01:30] the testing framework is pretty cool and I'm happy to do that [16:01:31] but there's no great UI for it [16:02:01] erosen and I are working on hooking everything up so when we finish the job management task we took on, we can finally show a full use case [16:02:27] drdee ^ [16:02:30] k [16:08:02] okay let's demo the tests [16:47:23] nick erosen [16:47:44] milimetric: want to sync up on job status? [16:47:59] yep, I'm in the hangout still [16:48:07] can you send link? [16:48:35] milimetric: ^^ [16:48:46] https://plus.google.com/hangouts/_/89c71ccf74db0f46cec118f5d13dd83679650568?authuser=1&hl=en [16:56:45] lost ya [17:03:07] AH [17:03:12] standup snuck up on me [17:03:17] OH [17:03:19] today is showcase [17:03:27] no standup! [17:28:11] hi people [17:28:22] anyone know d3.js ? [17:28:53] milimetric: ^ :) [17:28:55] I'm trying to make a calender heat map widget [17:29:10] oh hi OrenBochman [17:29:18] hi milimetric [17:29:21] yes, I know d3 :) [17:29:28] goooooooooooooooooooooooooooooooooooog [17:29:32] :) [17:29:44] we have a demo coming up soon [17:29:47] but I can talk after [17:29:51] luckily I don't have any hair [17:30:04] OrenBochman: http://www.patrick-wied.at/static/heatmapjs/ [17:30:11] OrenBochman: I've used this before and it was working fine [17:30:17] yeah, OrenBochman, average is pretty good with d3 [17:30:29] and Limn uses it heavily (Limn is the project I work with) [17:30:40] I've fiddled with d3 [17:30:41] http://jsfiddle.net/zzmgx/23/ [17:31:06] if you don't sort it out in the next couple of hours, I'll look at it after our demo OrenBochman [17:31:24] I've got a bunch of data in there + the calander hear map sample [17:31:45] the data is dates + edit counts [17:32:30] OrenBochman: what are you using to create the heatmap ? [17:32:44] the problem is putting them together ... [17:34:56] the original sample is from http://jsfiddle.net/zzmgx/23/ [17:35:06] nope [17:35:32] http://bl.ocks.org/mbostock/4063318 [17:37:07] that makes an output similar to github [17:38:40] OrenBochman: so just the colors aren't working [17:38:44] average: right ? [17:39:52] totals [17:40:10] these are totals for jimbo on that day [17:41:07] only some since the full data slows down jsfiddle [17:50:24] hey ottomata [17:50:31] I have this outstanding sonames thing [17:50:45] can you have a look please ? [17:51:06] I tried to change the rule names in Makefile.am but it causes problems [17:51:51] average: are you using faidon's patch? [17:52:07] drdee: did Faidon make a patch ? [17:52:10] yes [17:52:14] looking [17:52:26] you should have received a notification [17:53:16] https://gerrit.wikimedia.org/r/#/c/68711/ [17:53:32] i guess you should rebase from upstream and use dclass native JNI interface [17:54:55] wow, so he doesn't like any of the changes [17:55:05] I thought I would just fix the sonames and we'd be done [17:55:15] anyway, looking at his patch [17:55:25] he is showing you the ropes ;) [17:55:52] ok [17:56:26] New patchset: Milimetric; "work on csv upload and metric forms" [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/72991 [17:56:38] Change merged: Milimetric; [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/72991 [18:00:41] who is working on user metrics ? [18:01:03] I'm off to get some stuff from outside before the place closes [18:11:19] New review: Faidon; "Diederik mentioned on IRC that upstream JNI was based on yours and should pretty similar, so it prob..." [analytics/dclass] (debian) - https://gerrit.wikimedia.org/r/68711 [19:36:40] so, drdee, I am disabling the ip based filters [19:36:45] and starting a single x-cs based filter [19:36:46] right? [19:36:50] basically, grep for x-cs [19:37:15] on what machine do you wanna run the x-cs filter? [19:38:45] oxygen [19:39:46] ummm oh [19:39:49] hm [19:39:49] wait [19:39:50] hm [19:39:54] will this even work? [19:40:09] we kinda need to do unsampled filtering, right? [19:40:12] to catch x-cs? [19:40:24] otherwise we only have a small chance of actually filtering on x-cs [19:40:32] drdee^ [19:40:47] right so i was thinking a cisco box [19:41:00] and rotate the log files daily to stat1002 [19:41:10] bah, hmmmmmmmmmmmMMMMMMM [19:41:22] i don't really like it, i guess we could do analytics1001, since it has been reinstalled [19:41:29] i'd rather not add more stuff to non reinstalled nodes [19:41:48] an01 sounds good [19:41:56] you could puppetize this [19:42:05] yeah i would [19:42:37] i mean, webstatscollector is unsampled, right? [19:42:45] no thats for the reinstall [19:43:00] webstatscollector is unsampled [19:43:09] sooooo maybe this unsampled is ok? [19:43:11] dunno [19:44:04] drdee, i would much rather use oxygen if we can. [19:44:09] is anyone using the ip based logs anymore? [19:44:25] how about gadolinium? [19:44:40] it's a newer more beefy machine [19:45:05] yeah but oxygen is only doing zero stuff right now [19:45:18] if we aren't using ip based filters [19:45:21] let's remove them [19:45:23] and try oxygen [19:45:36] gadolinium is already doing webstatscollector [19:46:21] ok [19:46:42] so, the only thing I need to know [19:46:45] is anyone using ip based filters? [19:46:51] i doubt it, since they are incomplete, right? [19:46:58] erosen: ^^ [19:46:59] the only person that would use that is evan, right? [19:47:02] he's not in this chat [19:47:16] poked him [19:47:32] k [19:50:08] i think i should not drink more coffee, but i think i will anyway [20:09:56] oh drdee! [20:10:05] we can do this unsampled [20:10:07] pretty sure [20:10:11] at least, from a capacity standpoint [20:10:11] how [20:10:15] this isn't the full mobile stream [20:10:18] its just zero requests [20:10:21] true [20:10:25] wait [20:10:26] so its way smaller than what we were doing in kraken [20:10:34] there may be processing limits [20:10:40] but, i *think* we can do it [20:10:48] yurik wants to tag every w0 request with x-cs, also bits etc [20:11:06] how much more will that generate? [20:11:10] a lot [20:11:22] like orders of magnitude a lot? [20:11:27] yes [20:11:29] hm [20:11:32] yurik: ^^ [20:11:40] well, depends on how they are tagged [20:11:44] right now X-Analytics has zero=... [20:11:49] that's what I was going to grep for [20:11:56] does he have to add to zero= field? [20:11:58] or can we use another? [20:11:59] * yurik looks [20:12:50] a bit confused what the question is [20:13:13] you were going to tag every web request for Wikipedia Zero with the x-cs header, right? [20:13:14] ottomata1 & drdee , context pls? [20:13:27] yes [20:13:43] that is an order of magnitude more tags than right now the case is, right? [20:13:55] correct, and the zero ext logic would determine if the language and subdomains are whitelisted [20:14:05] ottomata1: then we cannot do unsampled [20:14:06] drdee, not exactly [20:14:08] hm [20:14:30] yurik; when will this go live? [20:14:43] a few weeks at least [20:14:48] k [20:14:49] well, quick estimate [20:14:49] it looks like grepping for zero= in the stream [20:14:49] generates about 0.5GB / day [20:14:49] not too bad [20:14:52] its not that big of an impact! [20:15:20] here's why: if the carrier is both M & ZERO subdomain ( and currently all our new subscriptions are going both) [20:15:24] we have about 1TB avail on stat1002 [20:17:11] ottomata1: i might be too pessimistic and if we see the packet loss go up we can always introduce sampling [20:17:12] then the only difference is if the language is whitelisted, and since we always whitelist the most popular languages in the country + english, the rest of the languages would not be a big impact [20:17:12] and for the older carriers, where we only whitelisted ZERO for them, they are not that big really to have a big impact overall across all carriers [20:17:12] not sure I follow, i think I don't understand the context, [20:17:12] hm [20:17:51] so yurik, we are about to start capturing some logs (not in hadoop) based on the zero field in x-analytics header [20:17:51] drdee [20:17:51] the problem is [20:17:55] we can't sample to grep [20:18:02] it would actually cause a bit more load [20:18:04] to do sampling after the fact [20:18:15] ottomata1, that's not enough i think - most of our new carriers are M, not ZERO [20:18:16] sampling would help if we can't write to disk fast enough [20:18:20] but i doubt that will be our problem [20:18:34] yurik, no, i mean [20:18:41] The X-Analytics [20:18:57] gets a field added [20:18:59] that looks like [20:19:02] 'zero=' [20:19:08] this is what i'd grep for [20:19:30] if you start adding this field on more requests, i just want to make sure that we are going to be able to handle it [20:19:40] so, i'd say, as long as it won't more than double the number of lines we capture, i think we'll be fine [20:19:45] and it sounds like it is unlikely to do that, right? [20:19:58] ottomata1, understood, i doubt it would even double [20:20:02] great [20:20:16] k [20:20:21] ok, drdee, we will try this :) [20:20:24] ok [20:20:30] evan says no one is uing ip based filters [20:20:36] so i'm going to disable those, and just try this on oxygen [20:20:44] ottomata, the bigger problem is what happens when we add new large carriers :) [20:21:25] ha, ja [20:21:30] but that we'll have to deal with no matter what [20:22:08] correct. But that i think will be the source of the large increases. *I think* [20:22:35] again, all these are guesstimates based on my understanding of the current situation. I don't have enough data :( [20:23:27] ottomata: i think your regex is too lose, i would also add \d{3}-\d{2} instead of .+ [20:24:26] i know, false positives [20:24:29] i was actualy just thikning about removing the regex altogether [20:24:43] but hmm [20:24:46] yeah lets' try that first [20:28:32] hmm, drdee, i mean, i guess we don't have to do it sampled? i'm not sure……….we were doing the IP based ones sampled, which would give the same results as doing this sampled [20:28:47] i guess we decided that there are enough zero requests as to overcome sampling error? [20:29:07] (btw, thanks yurik!) [20:29:12] np [20:32:06] oh good, everyone's talking about zero already. how easy or hard would it be to obtain the number of hits originating from carrier source ip addresses for the global home page at www.wikipedia.org/ (and for that matter, requests for wikipedia.com/, www.wikipedia.com/, wikipedia.net/, www.wikipedia.net/, and wikipedia.org/)? [20:32:38] ottomatta, drdee ^ [20:33:07] ottomatta, drdee, also, there's one carrier whose X-CS contains a * asterisk code in place of the last digit. [20:33:17] wasn't that fixed [20:33:29] ooooo [20:33:51] that's tata india [20:34:10] Guest38052: who ARRhhhhhh you? [20:34:14] but, i think not that hard [20:34:17] drdee, ottomatta - that X-CS is still passed per the varnish file. [20:34:26] that should be fixed [20:34:40] do you actually need to base this on IP addy? or can you rely on X-CS to identify the carrier [20:34:41] ? [20:34:48] ottomata, drdee, i think my client or freenode are in a state of chaos! i'm dr0ptp4kt. will drop off and drop back on….please hold [20:34:57] maybe you, and yurik can come up with an X-CS code for tata india? [20:34:58] ahhh ok [20:34:59] hah [20:36:12] drdee, ottomata would love to fix that one :) [20:36:17] go ahead! [20:36:20] can you retroactivelly fix the logs too? [20:36:35] or you don't log that stuff? [20:36:39] wouldn't want to loose that data [20:37:06] we've got code that handles tata weirdness [20:37:15] you won't lose anything [20:37:20] in fact it will make our lives easeir [20:37:29] uhhh, ok [20:37:32] https://docs.google.com/a/wikimedia.org/spreadsheet/ccc?key=0As-T7jJ1slQGdGVvS2xFcmszdlBVbXNkV25SR2dRdHc#gid=0 [20:37:38] drdee, should I modify the regex to accomodate for that then? [20:37:43] or should I just wait for fix in varnih? [20:37:44] no, we should fix tata [20:37:46] ok [20:37:48] cool [20:37:55] great [20:37:57] yurik: can you push soonish? [20:37:57] then. [20:38:08] zero.tsv.log is going on oxygen! [20:38:10] woot! [20:38:18] i mean i could make the patch if i just would know what tata's xcs code is [20:38:20] drdee, the problem would be to convince ops to do it - they would loose all their cache :) [20:38:29] for that carrier [20:38:45] not strictly necessary, right? [20:38:46] drdee, i could just have the regex check for the first 2 decimals [20:38:51] \d{2]- [20:38:53] or wahtever [20:38:55] 3 [20:38:58] oh 3 [20:39:03] yeah [20:39:06] yeah that could work as well [20:39:06] zero=\d{3}- [20:39:09] and lets pick a number for them :)\ [20:39:17] in that file i posted above [20:39:32] ok you guys tell me what to do, if you are going to change tata soon (like this week), then I will wait [20:39:35] drdee, ottomatta: whoami now? [20:39:36] and keep my regex [20:39:45] you be a dropped packet, my arch nemisis [20:39:59] how about 405-25 [20:40:23] and then have a 'business rule' that says whenever a carrier has multiple x-cs codes [20:40:24] drdee, i was thinking the same thing [20:40:29] pick the lowest one [20:40:30] ottomata, you must learn to love tcp. [20:40:34] dr0ptp4kt: to get you up to speed, we are having some really serious udp2log -> kafka producer problems right now, and rather than worry about solving that with more short term hacks [20:40:44] we are going to collect zero webreuset logs on oxygen using just udp2log [20:40:46] except when those carriers start buying and selling their subdivisions :) [20:40:55] drdee: want to chat wikimetrics db? [20:40:55] they will be copied over to stat1002 (and stat1, for now) [20:40:55] New patchset: Milimetric; "csv is stuck until deployment to labs" [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/73025 [20:40:55] daily [20:41:03] Change merged: Milimetric; [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/73025 [20:41:04] erosen sure [20:41:06] and you can do whatever kind of analysis on them you want (if you have an accoutn on stat1002, which im' sure we can get you) [20:41:11] batcave? [20:41:34] ya [20:41:40] oh, guess what, its disabled anyway :) https://meta.wikimedia.org/wiki/Zero:405-0* [20:41:46] oh great [20:41:58] not really [20:42:04] great for me! [20:42:06] the varnish is still marking that traffic [20:42:13] that's fine, my regex won't match it [20:42:21] ottomata, drdee: to come back to the question on carrier sourced hits for www.wikipedia.org/ and its pre-redirect paths….what were your thoughts on that? they don't go through the mobile varnish cache. wasn't sure if those hits are logged? [20:42:35] ^that is to say, the www.wikipedia.org/ hits [20:42:37] yes, but we do need that data later once we enable them - to see how their traffic changes [20:42:38] oh, they dont' go through cache? [20:42:47] they do :( [20:42:49] then give them a numbah! [20:42:53] and they fragment it [20:43:03] just like every other X-CS [20:43:07] oki [20:43:36] dr0ptp4kt: they don't go through cache? [20:43:54] if they aren't tagged with an x-cs, then you'll probably have to use sampled data [20:44:04] or, do you want this number regularly? or just one off to get an idea? [20:44:13] they ARE tagged [20:44:19] oh that was for that convo, sorry [20:44:20] overlapping [20:44:22] convos [20:44:23] ok cool! [20:44:25] then they will be logged [20:44:29] and it should be easy to figure out [20:44:33] will rename them soon [20:44:36] dr0ptp4kt: do you have an account on stat1002? [20:45:01] i tink you do not [20:45:58] ottomata, not sure. lemme try to connect. [20:49:53] naw, you don't :) [20:50:52] ottomata, concur. just got permission denied. [20:51:17] yo ottomata [20:51:18] quick question: [20:51:24] ottomata, i haven't gotten to requesting a username change…too scary. but i'm happy at the command line [20:51:45] ottomata, abaso ssh access to analytics1002 would be fine [20:51:59] would it cause any inconvenience if the stat1->stat1001 rsync script for /a/public-datasets ran more than once every hour? [20:52:18] even once every 30 mins would be super-helpful [20:52:28] dr0ptp4kt: https://rt.wikimedia.org/Ticket/Display.html?id=5446 [20:52:33] you should get an email too [20:52:36] your manager should approve [20:52:49] DarTar: should be fine [20:52:53] and we're mostly talking of tiny datasets (a couple of Kb) [20:53:27] grand, should I ask drdee to capture this in a mingle card? [20:53:38] nah this is too small [20:53:47] good to hear :) [20:57:16] ottomata, thx, emailed tfinc for approval [21:02:07] DarTar: Cron[rsync public datasets]/minute: minute changed '45' to '*/30' [21:02:11] thar ya go [21:02:15] okey dokey, i'm outty for the day [21:02:20] sweet [21:02:22] its montreal polo time woooo! but first I must fix bike [21:02:23] laataaas [21:02:25] thank you [21:02:52] greap dr0ptp4kt, now we just have to wait 3 days [21:02:57] before we can give you access [21:03:03] that's good, because i just turned on the zero filter [21:03:13] so by then you'll have 2 or 3 days of data to look at :) [21:03:37] ottomata, cool. you saying you truned on the zero filter. for? how does the filter work? sorry, got confused by cross-conversations with yurik :) [21:03:43] yeah [21:03:44] basically [21:03:55] grep all webrequest logs for zero=... [21:04:00] save those to a file [21:04:04] daily, rsync that file to stat1002 [21:04:05] that's it [21:04:29] so you'll just be working with raw gzipped log files on a single node [21:04:37] you can run whatever schmancy analysis you want on it [21:05:00] python, perl, brainfuck, waahhhteveaah [21:05:25] and now, tata! [21:06:22] ottomata, and how does 'zero=' get tagged into the requests for www.wikipedia.org/ (and it's pre-redirects like wikipedia.com/)? sorry, again…too many cross-conversations. [21:10:05] ottomata, or are you saying that the tagging is being turned on at the moment...? [21:12:46] ottomata, that is, based on source ip address. i'm not sure if x-cs is tagged on for www.wikipedia.org/ (webroot) or its pre-redirect requests (wikipedia.com/, www.wikipedia.com/, wikipedia.org/, wikipedia.net/, www.wikipedia.net/). [21:26:38] oh snap, ottomata has left the building. [22:06:32] what's the status of kraken? [22:06:42] happening this month? summer? [22:06:43] in what sense? [22:07:09] in terms of being at a point where anons can submit jobs and eventually they get run on non-public data. (maybe after human review) [22:07:15] we are working hard, we are busy puppetizing it [22:07:28] that part of the roadmap won't happen this summer [22:07:49] so just for NDA people for at least a few months [22:08:36] * jeremyb is writing: [22:08:37] > I think at this point that would either require a change to the format of the domas (anonymized) stats or an NDA. (or kraken! but rumor is that's not yet ready for the general public) [22:09:33] it's not something we worry about right now, we first want to get the platform stable [22:09:41] sure [22:09:48] sent (to the analytics list) [22:10:11] drdee: btw... maybe dclass would be good for this. i met the guy about a month ago [22:10:25] we are using dclass in kraken [22:10:33] also, i accidentally accused you of being montreal a few days ago. but someone corrected me to toronto [22:11:01] ok, i was thinking dclass in varnish and then varnishlog can log dclass output [22:11:04] but whatever :) [22:11:36] oh, wow, didn't realize how late it was [22:11:41] i think i'm gonna run [22:12:16] (or whatever it is that sends varnish to udp now... can't remember offhand) [22:14:38] drdee: Just to make sure I am asking the correct person on #798. erosen has the details on which of the pig udfs he'll need where in what form? [22:17:28] New patchset: Milimetric; "basic alert and info support" [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/73112 [22:17:38] Change merged: Milimetric; [analytics/wikimetrics] (master) - https://gerrit.wikimedia.org/r/73112