[11:07:03] so you would like a benchmark on a n270 atom ? [11:07:13] ops wrong channel... [13:55:59] I'm at the dentist [13:56:04] they have wireless [13:56:12] it's boring [14:01:26] hey average_1rifter [14:01:32] I LOVE dentists in Romania [14:01:50] milimetric: you do ? [14:01:52] milimetric: hi [14:01:56] totally [14:02:09] milimetric: why's that ? dentists in .us are not as skilled ? [14:02:14] a guy here charged me 2000 to mess up my wisdom tooth extraction [14:02:17] my dentist is a nice blonde lady in her 40s [14:02:20] I was infected for two months [14:02:35] wow hmm [14:02:47] Then I went to another guy who charged me 600 to xray my teeth and give me some antibiotics that didn't work [14:03:14] hmm that's a lot [14:03:17] I came to Romania and for $100 (private dentist, no insurance) she saw me three times and nursed me completely back to health [14:03:20] with no dangerous drugs [14:04:03] oh she said the antibiotics that they gave me in the US were a classic mistake - waaay too strong [14:04:36] anyway - just know that doctors there overall are doing it because they love what they do and not to get paid. And that's pretty rare [14:05:12] (that's just one anecdote but I've been to many great doctors there and many shitty ones here) [14:10:46] * YuviPanda is going to a dentist for the first time after ~10 years in about 2 weeks [15:13:56] morning guys [15:14:24] milimetric, can you email me instructions on how to setup a full working instance of limn on my local machine? [15:19:51] morning ottomata [15:25:14] mornign! [15:25:40] drdee, quick checking about priorities: [15:25:48] yo] [15:25:52] when paravoid is available [15:25:56] i'm going to work on puppet [15:25:58] when not [15:26:08] oozie automatiing [15:26:13] s'ok? [15:26:15] drdee my IRC isn't pinging me anymore - I'll get those instructions up on the Readme [15:26:15] yes [15:26:24] (dropping proxy/access priority) [15:26:43] the reason i am asking i want to do demo a kraken workflow using ssh tunneling and limn on my machine [15:27:07] so i can point limn to localhost for a datafile [15:27:11] that should work right? [15:27:47] actually [15:27:58] drdee, before oozie, shoudl be a cron to copy generated reports to stat1, right? [15:28:01] stat1001 [15:28:06] so that limn can graph them? [15:28:20] the things from /wmf/public/ ? [15:28:23] right [15:28:27] oh yeah I could just sync the whole thing [15:28:28] that'd be fine [15:28:29] yes good idea [15:36:09] ottomata, can you paste one more time the command for ssh tunneling? i will write it down now :) [15:36:33] i should write a script and put it in kraken repo :) [15:36:54] ssh -v bast1001.wikimedia.org -L 8081:analytics1027.eqiad.wmnet:8888 [15:36:56] that's for hue [15:39:10] do y'all know how to put tunnel commands in the ssh/config? [15:39:22] ottomata: ^ [15:40:36] in .ssh/config, naw never done that before [15:40:42] i think i have it [15:40:47] i'll send it along when it works [15:42:19] ottomata: i think i've got the config file working [15:42:29] but I'm getting a public key denied message [15:42:41] did I lost access as a result of the security thing? [15:42:53] s/lost/lose/ [15:44:29] don't think so, just use bast1001.wikimedia.org though, to be safe [15:44:36] instead of analytics1001 [15:44:43] that's what I'm doing, i think [15:44:45] hm [15:44:50] can you just ssh into that? [15:44:51] i just copied the line you sent to diederik [15:45:04] no [15:45:19] ssh erosen@bast1001.wikimedia.org [15:45:20] Permission denied (publickey). [15:45:42] drdee: install instructions for Limn: https://github.com/milimetric/limn#install [15:46:00] hmmm, weird [15:46:03] well try analytics1001 then :p[ [15:46:22] yeah that works [15:46:44] thanks for the help [15:51:46] cool visualization: http://www.guardian.co.uk/world/interactive/2013/feb/12/state-of-the-union-reading-level [15:55:30] erosen: ping [15:57:15] geohacker: hey there [15:58:01] erosen: I've cleaned up whatever dumps you shared and have started tinkering with them [15:58:21] nice [15:58:29] was wondering whether we can find rest of them :-/ [15:58:41] edit size, interesting [15:58:55] also edits by geography, sounds a bit tricky now. [15:59:00] do you mean the number of bytes of an edit on average? [15:59:07] erosen: exactly. [15:59:28] that will show us how productive the edits were. [15:59:34] indeed [15:59:42] i don't have such a dataset sitting around myself [15:59:51] but I suspect that some of ErikZ's files do [15:59:58] let me look around wikistats [16:00:08] that will be awesome. [16:00:49] just looked and I suspect I was misremembering [16:01:02] however, one approach would be to use number of edits during a period [16:01:08] and change in the article namespace total size [16:01:14] over that same period [16:01:21] and just compute the average your self [16:01:35] hmm that sounds good. [16:01:43] now erikz [16:01:55] 's size field isn't updated for a lot of the wikis [16:02:02] yeah. [16:02:05] but I believe i have that data on gp-dev [16:02:10] okay. [16:02:17] let me look [16:02:21] cool [16:02:42] erosen: also, do you think I should scrape out rest from stats? or can we find the dumps somewhere? [16:02:52] i [16:02:58] let me look a bit right now [16:03:11] awesome. thank you so much! [16:03:11] I suspect that there is a csv sitting somewhere on a server which I use [16:03:26] and it would be good to expose more numbers through the gp-dev dashboard [16:04:17] most of the code that I'm going be writing will use d3. [16:04:30] awesome [16:04:31] I'm sure we can hook that up to the dashboard [16:04:46] :) [16:05:07] not to push framework on you, but there is already a map functionality in the dashboard software we are developing, limn [16:05:32] great. didn't know that. [16:05:36] though I suspect you might want more customization [16:05:56] let me take a look [16:06:25] milimetric: do you have the link to the limn map handy? [16:06:39] which one, the old fabian one? [16:06:48] no the new dechoon one [16:06:53] http://reportcard.wmflabs.org/graphs/editors_by_geo [16:06:55] I'm checking github [16:06:57] i mean of that data [16:07:19] geohacker: that is the one i was thinking og [16:07:51] neat [16:08:26] geohacker: looks like I have the total size handy, but for some reason I am having trouble finding the link [16:09:12] erosen: uh oh. :-/ [16:09:25] should be okay [16:10:21] that map is neat. [16:10:40] do we have some way to do it at the country level? [16:10:51] I'm going to be doing that. [16:10:57] if I get the data. [16:11:17] drdee, erosen, if you pull kraken [16:11:22] bin/ktunnel [16:11:27] map edits by geography in the Indian states. [16:11:32] easy use: [16:11:33] ktunnel hue [16:11:40] will open a tunnel to hue, etc. [16:11:44] oh nice [16:11:52] danke [16:12:39] thanks otto [16:12:41] ja [16:12:47] it'll print the url you can navigate to as well; [16:12:48] when you use it [16:14:03] so, hm [16:14:05] milimetric [16:14:20] if limn/reportcard was running on a prod host [16:14:22] stat1001 for example [16:14:29] it could access data in kraken without having to copy it off [16:14:31] in /wmf/public [16:14:52] geohacker: bad news about the db size, looks like the files are no more up to date than wikistats [16:14:56] we need to puppetize limn :) [16:15:09] someone was working on .deb, right? [16:15:11] that's the first step [16:15:15] http://gp-dev.wmflabs.org/graphs/indic_language_db_size [16:16:30] geohacker: ^^ [16:16:47] erosen: checking [16:18:05] i can run the analysis myself without too much trouble, but I'm not sure what sort of schedule i can commit to [16:18:27] erosen: eh. why does it say zero beyond a point from 2011? [16:18:42] the script must have stopped collecting the data [16:18:48] darn [16:19:02] it seems to match the point in http://stats.wikimedia.org/EN/TablesWikipediaHI.htm where the numbers disappear for the db size colun [16:19:20] I just guessed that. [16:19:56] i've made graphs like this before: http://gp-dev.wmflabs.org/graphs/ar_bytes_monthly [16:20:00] what do we do? [16:20:19] just never made it automatic [16:20:30] oh nice. [16:21:02] i can take a crack at adding graphs for all of indic languages [16:21:04] erosen: do you think we can run the scripts specific for indic? [16:21:20] yeah, i can at least [16:21:27] that would be fantastic. [16:21:29] it uses the databases at the moment [16:21:37] do you have a tool server account by chance? [16:21:46] i'm so sorry for this trouble. [16:21:46] erosen: nope. [16:21:50] no worries, it's all things that should be done eventually [16:22:24] I can create an account and request access. [16:23:04] milimetric, erosen, I forget what our status was on oozie + limnification [16:23:08] not sure it is entirely worthwhile [16:23:15] milimetric, did you build something cool into limn to graph tsvs? [16:23:17] something like that? [16:23:20] ottomata: that last comment was for geohacker [16:23:24] (aye) [16:23:30] erosen: I'll do what you suggest. [16:23:37] ottomata: i think dan did [16:23:48] tsv is graphable [16:23:50] nothing special [16:23:54] works just like csvs [16:24:03] it needs datasources yamls, right? [16:24:05] milimetric: does it pivot though? [16:24:09] the "cool" thing I built is the ability to pivot [16:24:11] but those only need to be generated once? [16:24:12] riiiight [16:24:13] nice [16:24:15] but that's just hardcoded [16:24:20] like - it pivots on column 5 [16:24:21] :D [16:24:25] ok, can we try something then? [16:24:25] hehe [16:24:31] i have a .tsv of webrequest loss hourly [16:24:34] in kraken [16:24:47] its got a buncha columns [16:24:51] yes, but is it super urgent? Right now I found two annoying bugs that got introduced somewhere in the last month [16:24:55] and I'm git bisecting my heart out [16:24:56] nawww, not at all [16:25:10] just trying to figure out the best way to make this data available to limn [16:25:21] ok, then I promise I'll give you my undivided as soon as I figure these bugs out [16:25:24] real quick though, do you thikn I should just wait until limn is productionizable (.deb + puppet)? [16:25:40] i wouldn't have to think about this if so, since it would be able to access the files in kraken from stat1001 [16:25:41] drdee hasn't updated me on when that's going to happen [16:26:12] drdee, average_drifter - where's debianization on the list? It's starting to hold things up [16:26:29] oh he said average_drifter's doin dentist stuff for a few days [16:26:32] right [16:26:33] milimetric: I am working on the package as we speak [16:26:35] ok ottomata so we shouldn't wait then [16:26:36] as we speak [16:26:42] I started writing the files in debian/ [16:26:44] cool, thanks! [16:26:45] dude take it easy :) [16:26:45] in a separate branch [16:26:55] no need to work while you're recovering from tooth stuff [16:26:59] I was to the doctor and they scheduled me next week btw [16:27:15] oh, ok [16:27:38] thanks average_drifter! [16:27:43] erosen: so how do we go about this? [16:28:02] i'm going to generate some graphs for all of the indic langauges [16:28:04] ottomata: yeah, so let's just plan on debianization and puppetization being done sometime relatively soon [16:28:12] it should be pretty easy to do as a one off [16:28:24] erosen: okay. great. [16:28:26] and if you need them to be regularly updated, we can do that later [16:28:41] erosen: sounds good. [16:46:04] first push [16:46:18] far away from the first package, but will have one soon [16:46:28] limn branch is called debianization [16:46:34] feel free to keep an eye on it [16:49:12] geohacker: here is the hindi graph http://gp-dev.wmflabs.org/graphs/hi_bytes_daily [16:49:21] should be able to do the others easily now [16:50:28] erosen: checking [16:51:12] sync-ing git log with debian changelog [16:51:19] wow there's a lot of commits in the limn repo [16:51:37] erosen: awesome. this looks good. [16:51:55] erosen: is this a lot of manual work? or do you have a workflow? [16:52:18] it's a workflow [16:52:29] perfect. [16:52:32] i had a script which was parameterizable by language [16:52:44] awesome [16:52:56] so I just set it up to take in a list of languages and hooked it up to my dashboard deploy setup [16:53:59] great. [16:54:23] erosen: so I'm checking edit size from the etherpad. [16:54:30] ottomata: can you please help me with htaccess thingie ? [16:54:32] cool [16:54:45] ottomata: I have a directory /home/spetrea/www/private on stat1 [16:54:53] ottomata: I created a .htaccess and a .htpasswd for it [16:55:01] oh yeah, sorry, gimmee fiiiiive minutes [16:55:02] sorry [16:55:03] will help [16:55:18] ottomata: but I also need from someone who has access to configure Apache and restart it so I can use my private dir [16:55:21] ok thanks [16:56:29] erosen: this is the size of the db daily right? [16:56:49] yeah let me tell you a bit about how it works (I'll include this in the graph description) [16:57:02] okay. [16:57:24] basically it iterates over the revisions and looks at the changes in the size of the article in bytes [16:57:41] it then integrates those per revision changes into the total size as of a point in time [16:57:48] the graph I gave you has daily resolution [16:58:18] however it also generates a monthly value: http://gp-dev.wmflabs.org/graphs/hi_bytes_monthly [16:58:20] erosen: hmm. I might have to aggregate it monthly. [16:58:25] ah cool [16:58:57] and a have a month to month change graph too: http://gp-dev.wmflabs.org/graphs/hi_bytes_added_monthly [17:00:07] erosen: what are those negative values? [17:00:07] geohacker: also a quick note about something I noticed: these numbers can be negative if a lot of content or articles were deleted [17:00:09] hehe [17:00:15] ah :D [17:00:23] I guess we both noticed the same things [17:00:32] *fist pump* [17:00:43] *fist pump*, indeed [17:01:17] the other languages are still churning, but shouldn't be more than another 20min [17:01:25] perfect. [17:02:15] erosen: I think if we take the daily dump, that will reflect the edit size. right? [17:02:28] for that day [17:02:51] i don't think i follow [17:02:59] what do you mean by daily dump? [17:03:11] erosen: sorry. I meant the daily graph. [17:03:39] the differences in the daily_SIZE graph should give you the amount of content added [17:03:49] I can also add a daily bytes added graph [17:03:53] if that is useful [17:04:07] erosen: yes. perfect. [17:04:33] k, one sec [17:15:05] brb [17:19:51] geohacker: here is the daily graph for Hindi: http://gp-dev.wmflabs.org/graphs/hi_bytes_added_daily [17:26:35] milimetric: Do you have another limn repo? [17:28:59] erosen: back now. hmm that table is empty. am I missing something? [17:29:03] hmm [17:29:11] it shouldn't be [17:29:17] erosen: http://gp-dev.wmflabs.org/graphs/hi_bytes_added_daily [17:29:41] oo interesting [17:29:45] :-/ [17:29:47] i must have updated it just now [17:29:54] it was working in my browser [17:30:20] one sec [17:30:26] sure :) [17:31:28] weird, now it works [17:31:34] for me at least [17:31:42] geohacker; can you check (same url) [17:33:03] geohacker: also, all of the other languages finished, I am rerunning so as to generate a daily bytes added graph (which will be much faster because the queries are cached from the first run) [17:33:04] checking [17:33:23] geohacker: i added a list of the languages I am using to you etherpad document [17:33:40] mornin [17:33:49] erosen: works now. [17:34:32] great [17:34:40] hey preilly - sorry was at lunch [17:34:46] We work on limn via forks [17:34:57] so wikimedia/limn is the only repository that matters [17:35:00] milimetric: What is limn sanity ? [17:35:02] we deploy from there [17:35:31] milimetric: I understand the fork and pull request model I was asking about another repo that I heard about [17:35:34] oh, heh, that was just a proof of concept that d3 and knockout could be used in a very simple way [17:35:41] erosen: saw the etherpad. perfect. I just have to change URL language cod right? [17:35:48] *code. [17:35:50] milimetric: where does that live? [17:36:10] it's on my milimetric github account, but it was never meant to see light of day - perhaps I should delete [17:36:23] all the limn relevant work I did got merged into limn proper [17:36:55] * preilly — Forking milimetric/limn-sanity [17:36:58] I just kind of like the name and was thinking I could use it for other proof of concepts [17:37:13] geohacker: yup, justs replace the language code in urls [17:37:14] hahaha, it really probably doesn't even work at this point [17:37:43] geohacker: all of the languages should now have a daily_bytes_added graph/data [17:37:53] preilly: you interested in working on limn? :) [17:38:00] geohacker: what were the remaining dimensions you were interesting again? [17:38:21] erosen: the ones at the bottom of the pad [17:38:23] milimetric: nope I'm just reviewing code [17:38:42] geohacker: so, edits by geography? [17:38:45] or all of the ones form wikistats [17:38:55] erosen: all of them, I just updated it. [17:39:03] k [17:39:07] preilly: ok, then. Let me know if you have any questions [17:39:08] god i love aesop rock. [17:39:24] milimetric: will do — thanks [17:39:27] fyi, kraigparkinson -- i'll be in the office after scrum [17:40:01] dschoon: have you seen Ian perform live? [17:40:21] i missed him when i saw rjd2 back in 2010 :( [17:40:31] erosen: do we have the edit size monthly as well? [17:40:38] dschoon: he is live in Oakland, CA on Apr 22 [17:40:50] geohacker: do you mean bytes added? [17:41:09] erosen: oh yes. sorry. I keep going back to what we wrote in the pad. [17:41:09] if so, yes: http://gp-dev.wmflabs.org/graphs/hi_bytes_added_monthly [17:41:09] dschoon: here is the link: http://www.thenewparish.com/event/195433-aesop-rock-rob-sonic-dj-oakland/ [17:41:32] cool, thanks, preilly [17:41:41] geohacker: however something weird is going on with that graph I just linked [17:41:59] erosen: yes. just noticed. [17:42:01] :-/ [17:42:04] ya [17:42:31] erosen: firebug tells me that n is undefined formatters.mod.js line 60 [17:42:45] yeah [17:43:46] i suspect it has to do with negative values or something funny [17:43:54] uh oh [17:43:56] milimetric: have a sec for a limn q? [17:44:26] you can still grab the csv at least:http://gp-dev.wmflabs.org/data/datafiles/gp/hi_bytes_added_monthly.csv [17:44:29] just not as pretty [17:44:35] hi [17:44:44] erosen: i bet this is somehow callout-node related [17:44:46] erosen, what's up [17:45:00] we're seeing an unexpected behavior here: http://gp-dev.wmflabs.org/graphs/hi_bytes_added_monthly [17:45:02] i think i've mostly excised formatters.js otherwise [17:45:02] in favor of d3.format [17:45:08] gotcha [17:45:39] erosen: also the link to the csv in the graph page has two ../gp/gp/.. [17:45:51] yeah, i noticed that [17:45:51] ooo [17:45:57] i think i know what is happening [17:46:04] milimetric: don't worry about it [17:46:34] geohacker: this is a result of a hack which does a sed command i run on the data files. one sec [17:47:13] erosen sure. [17:50:16] geohacker: fixed, i think [17:50:23] bleh. stupid wifi. [17:50:25] erosen, sorry. what was the issue you were seeing? [17:50:36] erosen: checking [17:51:22] erosen: awesome. [17:51:25] thanks! [17:51:57] erosen: should we start looking at the other items in the list? [17:52:04] dschoon: it was a js error which had to do with a formatter and some "toFixed" method [17:52:24] dschoon: but it turns out it was just because the datasource was pointing at the wrong path for the datafile [17:52:27] geohacker: sure [17:53:14] erosen: also, if we can document this somewhere, at least the workflow, I think that will make things easy next time. [17:53:40] ahh. yeah, toFixed is a method of Number, so you're right that it was probably n == null [17:54:25] fyi messing with namenode conf stuff [17:54:30] things funky for a sec [17:54:32] geohacker: i'm a fan of documentation, but I'm not sure I know what you mean [17:54:44] geohacker: do you mean the code that generates these graphs? [17:54:57] erosen: both the code and the process. [17:55:01] I'm planning on setting them up to run on a cron job [17:55:09] erosen: right. [17:55:11] i can show you the code though, in case you are interested [17:55:26] would love to take a peak! [17:55:40] erosen: are these erik's perl scripts? [17:55:55] I might have seen them. [17:55:57] these are my python scripts which interact with the databases [17:56:11] erosen: ah clean. would love to see them. [17:56:34] * geohacker is not a fan of perl. never. [17:57:08] hehe [17:57:29] here it is on gerrit: https://gerrit.wikimedia.org/r/gitweb?p=analytics/global-dev/dashboard.git;a=blob;f=db_size/total_size.py;h=8705e3f98cb36ef7f8e43a2146bc9f1fc13b1837;hb=HEAD [17:59:22] erosen: thanks. this looks neat. [18:00:22]