[02:35:15] New patchset: Yuvipanda; "Added script to generate all time upload & survival numbers" [analytics/limn-mobile-data] (master) - https://gerrit.wikimedia.org/r/58044 [02:46:21] New patchset: Yuvipanda; "Added script to generate all time upload & survival numbers" [analytics/limn-mobile-data] (master) - https://gerrit.wikimedia.org/r/58044 [03:21:03] Change merged: Yuvipanda; [analytics/limn-mobile-data] (master) - https://gerrit.wikimedia.org/r/58044 [13:46:39] morning all! just got through my weekend email! i'm headed to a cafe. [13:57:53] morning [14:02:43] hey milimetric [14:02:44] good morning :) [14:02:48] New review: Milimetric; "Generating Share Attempts (per day) (/a/limn-mobile-data/mobile/share-attempts.sql)" [analytics/limn-mobile-data] (master) - https://gerrit.wikimedia.org/r/58044 [14:03:09] hey Yuvi [14:03:13] i foudn the bug [14:03:17] fixing and sending a patch [14:03:24] unsure how that went past [14:03:27] k, no prob. Just put the message in there for posterity [14:03:49] sorry, I meant that *I* just put the message in there, you don't have to do anything [14:08:23] oooh, I *think* ori-l is the culprit here [14:08:42] as in, not just me, but him too :) [14:08:51] he split conn_for into make_connection and get_connection [14:08:54] which I didn't notice [14:08:58] but forgot to change it for the scripts [14:12:13] milimetric: patchset coming up [14:12:44] New patchset: Yuvipanda; "Fix bug caused by naming confusion" [analytics/limn-mobile-data] (master) - https://gerrit.wikimedia.org/r/58087 [14:12:47] Cool. If only we were using a compiled language :) [14:12:52] heh :) [14:15:14] milimetric: is there graph node docs somewhere? [14:15:20] milimetric: I need to make a scatter graph, for example [14:15:34] milimetric: and also wondering if we can get, like, just a talbe to show up [14:15:37] or a plain number [14:16:10] the only table support is the button "show as table" [14:16:35] plain numbers aren't supported except on bar charts. I suppose you could have a single bar chart [14:16:46] "single bar"-chart [14:16:52] hmm, yeah. [14:16:54] I could [14:17:10] scatter nodes examples are on the reportcard, one sec [14:17:23] http://reportcard.wmflabs.org/graphs/pageviews_mobile_target_stacked [14:17:32] http://reportcard.wmflabs.org/graphs/pageviews_mobile_target_stacked.json?pretty=1 [14:18:43] as for bar graph, the only example is in the limn-data: https://github.com/wikimedia/limn-data/blob/master/graphs/ordinal_example.json [14:19:37] and sorry, I know you need the documentation for the nodes, we want to write it as bad as you want to read it but have to listen to priorities [14:23:28] milimetric: thanks, will look :) [14:23:30] milimetric: yeah, I understand :) [14:23:36] so little time, so many things to do, etc [14:23:52] milimetric: any way I can subscribe to the cron error emails? [14:24:10] ottomata: can you fulfill YuviPanda's wish? [14:24:24] he desires to get that cron output you sent me this morning [14:26:01] milimetric: can you merge https://gerrit.wikimedia.org/r/#/c/58087/? [14:26:35] oh, i don't have the setup to test those things [14:26:41] you should self-merge [14:29:33] milimetric: hmm, okay [14:29:46] that feels icky, but since there is an error going on now and I tested itp roperly... [14:30:31] New review: Yuvipanda; "Self merging. Makes me feel a bit icky, but I tested it properly + crons are erroring :)" [analytics/limn-mobile-data] (master); V: 2 C: 2; - https://gerrit.wikimedia.org/r/58087 [14:30:32] Change merged: Yuvipanda; [analytics/limn-mobile-data] (master) - https://gerrit.wikimedia.org/r/58087 [14:36:17] moooooorning [14:36:53] mornig! [14:50:57] hi drdee , ottomata , milimetric [14:51:01] :) [14:51:05] morning [14:52:32] milimetric, average_drifter: hangout to look at the latest report? [14:53:32] sure [14:54:36] I'm in the hangout [14:55:59] coming [14:56:17] we're both in drdee [14:56:21] sorry [15:07:11] http://limn.it/issue/02/ [15:37:23] http://stevelosh.com/blog/2013/04/git-koans/ [16:24:33] no kp, it seems [17:00:39] erosen: around? [17:00:46] average_drifter: scrum ? [17:02:33] can someone send the correct hangout link? [17:03:41] https://plus.google.com/hangouts/_/2da993a9acec7936399e9d78d13bf7ec0c0afdbc [17:31:50] dschoon, if you haven't seen this, it's great! [17:31:50] http://limn.it/issue/02/ [17:32:22] i ordered both issues [17:38:41] brb [18:04:22] mil metric, hangout (quick for 5 minutes) [18:04:35] milimetric, hangout (quick for 5 minutes) [18:04:46] howdy [18:04:47] coming [18:23:10] hey ottomata [18:23:31] hihi [18:24:33] soooo, you're right about the 8421 data / eventlogging, trying to think of the best way to clean that up [18:24:48] probably just have this data go to vanadium directly [18:25:52] so i think it's ok to just delete it [18:26:02] i think i just answered my own question [18:26:05] haha [18:26:05] i'll submit a patch :P [18:26:12] but we need to change the frontend first [18:26:16] ok cool [18:26:23] sounds great [18:44:58] erosen [18:45:00] you there? [18:45:01] aue [18:45:01] aye [18:45:10] oxygen is packet lossing again [18:45:14] which sends alerts to all of ops [18:45:19] word [18:45:25] i think we need to disable more filters [18:45:40] agreed [18:46:00] two options that come to mind are just consolidating several ipranges [18:46:08] could help [18:46:09] or just making the single filter on x-cs [18:46:15] yessssssss [18:46:17] hehe [18:46:21] yesssssssss [18:46:36] yeeessssssss [18:46:40] ;) [18:47:09] we are waiting on your permission to do so! [18:47:10] :) [18:47:11] okay, give my 5 min to do another round of checks to make sure this doesn't cause any serious losses in data (as the mobile stream isn't sufficient, so this is our only source of data atm) [18:47:18] ok [18:47:40] mobile stream isn't sufficient because you need API logs too? [18:48:01] we need the log lines from other machines, maybe [18:49:18] depending on whether the fact requests from the specified IP ranges hit non-varnish servers is a mistake (device detection?), or as expected (desktop views that we don't care about) [18:49:26] I should be able to answer that pretty quickly [18:49:46] k [18:57:09] ottomata: drdee: https://gist.github.com/embr/012e97997cd2153b8fb8 [18:57:58] maybe an idea to send this to the @ops list and ask: [18:58:01] those are the counts from a random orange-kenya log which show the relationship between server and site [18:58:10] a) is zero traffic considered to be mobile traffic [18:58:13] k , what's server == x? [18:58:21] main site [18:58:22] sorry [18:58:27] k [18:58:28] and if a) is yes, then why is traffic showing up on the non-mobile varnishses [18:58:43] hmm, its not [18:58:47] at least in this list [18:58:56] i only see 'M' for cp1041-1044 in this list [18:59:07] yeah [18:59:09] which is fine [18:59:15] which is good [18:59:17] i didn [18:59:18] but none of these have x-cs? [18:59:20] yeah [18:59:35] started to say, I'm not immediately sure that we should be alarmed [18:59:40] can we look at the None, 'M' ones and see what hte IPs are? [18:59:42] but the lack of x_cs seems to be the problem [18:59:47] yup [19:00:08] why don't you send your findings to the ops list and ask them for feedback? [19:00:48] sure, I'm a little shy about ops list stuff, but that's fine [19:01:30] well, not yet [19:01:32] let's check it out [19:01:38] we have more sleuthing we can do first [19:01:50] yeah, that is my attitude [19:01:59] we should email ops when we get stumped [19:02:07] we got questions we can answer ourselves first [19:02:38] well we have been spinning our wheels on this topic for quite a while [19:02:40] can you grab the M lines and put them in a file? [19:02:43] naw, we haven't [19:02:45] we haven't been thinking about it [19:02:47] maybe it's a deadened, maybe it's something obvious [19:02:48] ottomata: drdee: check out hte second gistfile: https://gist.github.com/embr/012e97997cd2153b8fb8 [19:02:51] we haven't spun the wheel [19:03:17] if you want the whole lines, i can do that too this just seemed easier to read [19:03:25] yeah, can you put them in a file on stat1 [19:03:29] sure [19:03:31] all the lines? [19:03:40] i'll check the udp filter stuff for orange kenya against htem [19:04:01] oh kenya is a good one [19:04:02] easy range [19:04:02] 212.49.88.0/25 [19:04:21] great, ok [19:04:42] acl carrier_orange_kenya { [19:04:42] "212.49.88.0"/25; [19:04:42] } [19:04:53] else if (client.ip ~ carrier_orange_kenya) { [19:04:54] if (req.http.X-Subdomain == "M") { [19:04:54] set req.http.X-Carrier = "Orange Kenya"; [19:04:54] set req.http.X-CS = "639-07"; [19:04:54] } [19:05:22] hmm [19:05:28] ok, yup, then that should be happening, either it isn't or we aren't logging it correctly [19:05:30] ooo [19:05:32] my bad [19:05:37] this is all messed up [19:05:48] ? [19:05:50] I'm using a log from before X-CS was deployed!!!!!!!!!! [19:05:51] hehe [19:05:53] ah [19:05:54] hahah [19:05:56] okay on sec [19:05:57] um [19:05:58] also [19:06:10] we are logging X-Analytics in logs now... [19:06:21] i see [19:06:27] /* Assemble X-Analytics header */ [19:06:28] if (req.http.X-CS) { [19:06:28] set req.http.X-Analytics = "zero=" + req.http.X-CS; [19:06:28] } [19:06:29] ok [19:06:29] that's fine then [19:06:34] are you suggesting we should use such a log? [19:06:41] nono [19:06:43] i would opt for a mid-feb log [19:06:44] k [19:06:45] use a late one [19:06:49] use yesterdays if you can [19:06:52] i see [19:07:07] its the same field though [19:07:19] so your script should work, it just wont' be only 'X-CS' in that field [19:13:13] ottomata: check this file: ~/erosen/share/kenya_mobile.counts [19:13:43] that has the counts of requets, grouped by (x-analystics, site, host, ip) [19:13:49] i'm making a file with the raw lines presently [19:14:16] hola erosen [19:14:23] heyo [19:14:24] sup? [19:14:29] quick qs: are you planning to join us at 2? [19:14:35] if not can we have you for 30 mins at 1.30 before Kirsten comes in? [19:14:50] i was planning [19:14:55] ok cool [19:14:56] sorry about not responding to the event [19:15:15] milimetric: are you already in cave-man mode? [19:15:19] np - I want to take a moment to review the logic of timeseries aggregation [19:15:25] a little bit [19:15:28] why what's up? [19:15:37] so we can do it at the very beginning before we move to docs mode [19:15:40] DarTar: great, that sounds very useful [19:15:53] DarTar: still at 2 or earlier [19:15:54] ? [19:15:54] milimetric: 2 mingle cards, and mobile page view definition [19:16:12] ok, which cards? [19:16:17] oh i'll come in the hangout [19:16:19] 5 min. [19:16:24] cool [19:16:30] erosen: 1 sec, hold on [19:16:33] k [19:16:34] average_drifter: can you me and milimetric in hangout in 5? [19:18:20] hm, erosen, so there are some IPs that are not being tagged properly [19:18:41] ya [19:18:51] erosen: I have stuff scribbled on the whiteboard in Chambers and the room is free after 1.30, I am blocking it just in case we want to meet there before 2 [19:18:59] did you manually check them / used your special tool [19:19:02] this is weird: [19:19:05] rfaulkner ^^ [19:19:12] at ~erosen/share/kenya_mobile.counts | grep 212.49.88.101 [19:19:18] cat ~erosen/share/kenya_mobile.counts | grep 212.49.88.101 [19:19:52] there are 3 times that 212.49.88.101 was tagged with x-cs on cp1044, and 1 time that it wasn't [19:20:05] very true [19:20:14] now that is a stumper [19:20:24] what time frame is this from? [19:20:31] ottomata: try this: cat ~erosen/share/kenya_mobile.lines | grep 212.49.88.101 | less [19:20:55] DarTar: sounds good. I don't have another hard commitment [19:21:03] DarTar: let me know if you decide you want to meet earlier [19:21:22] great, 1.45 should be fine [19:22:12] cool [19:22:15] ok here's something erosen [19:22:21] i'm comparing requests that have x-cs and those that don't [19:22:23] here's a diff in the url [19:22:36] http://en.m.wikipedia.org/w/index.php? … vs [19:22:36] http://en.m.wikipedia.org/wiki/ ... [19:22:40] index.php ones are not tagged [19:22:45] ooo [19:22:47] interesting [19:23:22] seems like a very plausible error [19:23:23] are those possibly redirects? [19:23:26] hmm [19:23:38] not always, afaik [19:23:46] but can you check the status code [19:23:51] hmm [19:24:00] text/html vs text/javascript i think [19:24:03] is the more relevant difference [19:24:13] sure [19:24:43] i think all text/html has x-cs tagged [19:25:44] image/x-icon is also tagged [19:26:54] hmm, found this: (('-', 'M', 'image/x-icon'), 22), [19:27:38] that is ther are 22 requests where (x_cs == '-' && site=='M' and mime_type='image/x-icon') [19:27:59] ok so that's not consistent then? [19:28:24] did you grep to be sure about your results? [19:28:38] i'm grepping around [19:29:03] i just mean when you said all the image requests were tagged, did you check a whole file? [19:29:33] i only see one with image/x-icon and x-cs set [19:29:39] i'm only checking the file you gave me [19:30:15] 23 image/x-icon in that file, only one of which was tagged with x-cs [19:32:28] ottomata: a little more info (counts by mime-type): ~/share/kenya_mobile_mimetype.counts [19:33:17] my script isn't quite parsing the charset stuff right [19:33:24] (just a warning, but I have the sniffles) [19:33:44] * erosen warned [19:34:05] i feel like i have been sick for a day every single week for a month [19:34:06] hmph [19:34:07] so [19:34:08] not reltaed [19:34:59] erosen, can you do the same for /index.php vs /wiki? [19:35:06] yeah, one sec [19:39:43] ping average_drifter [19:43:40] ottomata: /home/erosen/share/kenya_mobile_url_type.counts [19:44:56] POOP [19:44:57] no consistency [19:44:59] no pattern [19:45:03] ya [19:45:11] i love those sorts of problems [19:45:13] hehe [19:45:38] ok so sounds like this is an ops issue somehow then, maybe there are varnish worker threads that hang around that don't have the proper confnigs loaded [19:46:34] ottomata: in the meantime, can we consider this enough evidence to justify constructing a few udp2log instances which write lines using a disjunction of a few partner ranges? [19:47:26] yeah that's totally fine [19:47:33] k [19:48:23] ottomata: would you be willing to draft the ops e-mail? I would be happy to help format our findings [19:52:41] ottomata: i'm off to lunch for a bit, let me know if you need anything today [19:55:26] k, thank [19:55:26] s [19:55:29] i'm checking a coupel of things [19:55:30] will ask ops [20:00:06] hi milimetric [20:00:21] hi jgonera [20:00:27] I don't know what magic we did last time to set up limn locally [20:00:33] but I have problems with it again [20:00:56] I guess I must be running the data-linking command differently [20:01:01] coke --vardir ./var --data /path/to/limn-data --to example link_data [20:01:08] what is example and link_data here? [20:01:18] link_data is literal [20:01:24] example is literal [20:01:30] literal? [20:01:37] /path/to/limn-data should be the absolute path [20:01:50] yes, I remember this one [20:01:54] yeah, literal as in has to be typed exactly [20:02:06] but exactly as what? [20:02:11] "example" is the name coke will give to the symlinks [20:02:36] exactly as written, "example link_data" should be just like that [20:02:37] where are those symlinks? can I just use any name? [20:02:42] if you're linking to the limn-data repository [20:02:51] no, I want the limn-mobile-data [20:03:09] ok, then it has to be "mobile" instead of "example" [20:03:12] this sounds dumb, I know [20:03:23] but it's how it finds the datafiles if more than one data repository is linked [20:03:28] but still link_data? [20:03:54] yes, but, here, I'll break the command down so as to make it less magical [20:04:12] coke -- this is the executable installed by npm install -g coco [20:04:37] everything seems clear now, apart from link_data [20:05:03] --vardir ./var [20:05:30] that is the place it puts generated files. It's configurable so you can run more than one site on top of the same limn codebase [20:05:42] --data /absolute/path (you know this one) [20:05:48] --to example [20:06:13] I get it, link_data is coke's command [20:06:15] here, "example" is the name of the symlinks that live in --vardir [20:06:24] it's just confusing it's after the arguments [20:06:28] yep [20:06:42] it's a bit of a silly thing, it won't recognize those if I put link_data first [20:06:49] at least not the last version [20:06:50] still nothing shows up in limn... I remember we changed the branch too, right? [20:07:01] heh [20:07:13] you can try either develop or master, but lately those are pretty close [20:07:45] so if you're linking at the limn-mobile-data, which link are you hitting on your local box? [20:08:01] http://127.0.0.1:8081/ [20:08:14] oh ok, yeah, that should work. Does /datasources give you anything? [20:09:43] it just keeps saying "Loading all the datasources..." [20:09:48] gr [20:09:55] no errors in JS console [20:10:00] ok, i'm trying it on a fresh clone [20:10:05] yeah, means it can't find the files [20:10:16] i keep suspecting permissions problems but i never run into them personally [20:10:28] i've literally never had the problem you and Yuvi are having... [20:10:55] hm, I don't know, I have everything in my home dir, the only thing I used sudo for was installing coke globally [20:12:36] heh, of course it works for me on a clean clone of both. Hm... here, let me paste you the command history: [20:14:03] git clone git@github.com:wikimedia/limn.git [20:14:03] git clone https://gerrit.wikimedia.org/r/analytics/limn-mobile-data [20:14:03] cd limn [20:14:03] npm install [20:14:03] coke -v ./var -d /home/dan/publish/limn-mobile-data -t mobile link_data [20:14:18] hm, the -v -d -t things are just shortcuts [20:14:26] shouldn't make a difference... [20:14:34] the symlinks are there, I checked [20:14:39] and then npm start and I got a working instance [20:14:47] yeah, i'm not following why it does that sometimes. [20:15:02] but since I can't reproduce, I'd love to do a screenshare with you and look into it if you have time [20:15:10] (not now though 'cause i'm supposed to be in a meeting) [20:15:37] I'd do that but I'm on an old machine using open source ATI drivers which don't work well with hangout [20:15:51] erosen [20:15:55] you there? can you check one more thing for me? [20:16:10] group on $6, http status [20:16:12] miss vs hit [20:16:33] milimetric, I think I got it, Limn's readme says to run npm update too [20:16:44] no, that actually breaks things [20:16:44] I removed node_modules and did only npm install [20:16:47] and it works [20:16:56] oh crap, where is that damn npm update reference? [20:17:02] i gotta kill everything that says that [20:17:04] it's the devil [20:17:22] npm update will actually ignore the packages.json file that's supposed to dictate the versions you're interested in [20:17:45] in Install in README.md [20:18:04] yeah, I know that, that's why I wondered why it's in the readme ;) [20:18:09] k, removed [20:18:26] maybe it'll fix it for YuviPanda too ;) [20:18:28] at some point it randomly helped us with some problem we were having [20:18:33] so we put it in there for no reason [20:18:45] :) cool, hope so [20:19:10] thanks for the help [20:19:28] [travis-ci] master/fb5f91d (#123 by Dan Andreescu): The build passed. http://travis-ci.org/wikimedia/limn/builds/6164103 [21:04:44] kraigparkinson: coming to the hangout? [21:07:40] yes, just a few seconds. :) [21:08:34] ;ok, maybe another couple minutes, I'm sorry... [21:22:39] man, thatcher was so good. http://www.theatlantic.com/business/archive/2013/04/watch-margaret-thatcher-explain-why-the-euro-is-a-terrible-idea-in-1990/274768/ [21:22:47] wonderfully forceful speaker [21:29:21] growl, erosen, i can't find any consistency to why some lines are tagged and others aren't!~ [21:29:30] weird [21:30:01] have you guys tested against the distribution of cache servers? [21:30:16] because it's the cache that's applying the tagging [21:30:23] (the x-cs header) [21:30:25] distribution? yeah we're only looking at the 4 mobile hosts [21:30:27] no consistency [21:30:30] huh [21:30:38] even bucketed by timestamp? [21:30:47] i am thinking about deploy timing [21:30:50] haven't checked timestamp, but we're only looking at a single day [21:31:44] looks pretty random across the day [21:32:48] hm [21:32:49] then yeah [21:32:51]