[01:09:39] drdee: I'm awake now, I had dinner and was waiting for the review on the code and I fell asleep [01:10:07] drdee: my phone's alarm went off for some reason just now [01:10:38] drdee: reading the review now [01:23:34] drdee: you there ? [01:34:12] ok, I'm out to get smokes and some beer(I know it's not perfectly heathly at this hour but meh..) [01:34:28] when I come back, I'll make a new remote branch and push my changes there [01:36:56] don't sweat it :) tomorrow is a new day [04:04:39] hey guys, is the cluster up atm? [04:14:07] No, it's not. [04:14:22] otto was switching servers over to precise at the end of the day, ori-l [04:14:30] gotta run. back in a bit. [12:52:45] good morning guys [12:54:45] moin [12:54:51] ori-l's sleeping i guess [12:55:20] so, i barely read anything yet about this new feed, but... how about pubsubhubbub? [12:55:24] drdee: morning [12:55:34] morning everyone [12:55:44] jeremyb: or-l is pst [12:55:53] drdee: pdt!!!! [12:56:01] pst is (for now) the devil! [12:56:09] morning average_drifter [12:56:14] jeremyb:? [12:56:45] drdee: there is (AFAIK) no part of the country currently observing PST. there are MST though [12:57:05] why can't people that don't care (which is fine if you don't) just say something DST agnostic like "pacific"? ;-( [12:57:13] oh is pst winter time on the west coast? and pdt summertime on the west coast? [12:58:17] winter time is always normal and summer is (if you observe it) DST. this rule holds in israel, eastern and central europe, britain, and the whole US [12:58:28] and probably even australia! [12:59:28] anyways he lives on the westcoast :) [12:59:34] but i don't care so much if you memorize all that or get it right. i just care that you don't say pst or pdt when you don't know which you mean. so say pacific ;) [12:59:43] average_drifter: any luck with the remote branch? [12:59:53] drdee: I will make the remote branch now [12:59:56] (the P in pdt/pst stands for pacific in case you didn't get that) [13:00:13] no reason to be snarky jeremby [13:00:35] drdee: i don't follow? [13:00:44] drdee: is it possible to have the git repo for wikistats so I can merge my changes there somehow and maybe start a review for them as well ? [13:00:54] drdee: I would require the git repo url for the wikistats for that [13:01:04] average_drifter: yes there is already a git wikistats repo [13:01:28] drdee: ok, I don't know the url. I must have overlooked it in the e-mails ? [13:01:29] git clone ssh://@gerrit.wikimedia.org:29418/analytics/wikistats.git [13:02:07] also, wow, that's a new misspelling of my name that i think I've not seen before. (people do misspell it quite often) ;-) [13:02:15] drdee: thanks [13:02:51] but others surely have it worse than I do! [13:08:53] moin to other side of kings! [13:10:03] morninnnn [13:10:10] moooooooooorning [13:14:53] ottomata, question about wikistats [13:15:20] could you move the current wikistats folder to wikistats_bak (on stat1:/a) [13:15:43] and run git clone ssh://ottomata@gerrit.wikimedia.org:29418/analytics/wikistats.git [13:15:52] so we have a wikistats that is deployed from git? [13:16:03] um, you sure that's ok with EZ? [13:16:44] i spoke with him about the need to have wikistats on the server match wikistats in version control [13:16:52] right now the two have different folder structures [13:17:04] which makes it very hard to deploy [13:17:07] certainly [13:17:11] ez agrees we need to fix that [13:17:16] as long as EZ ok with it [13:17:17] ok [13:17:25] 202G? [13:17:32] maybe not, but then we can reverse it very easily [13:17:41] wikistats in git is not 202G [13:17:44] ok [13:17:47] probably the out files [13:17:58] wikistats in git is very small [13:18:05] just the perl code basically [13:18:07] would be nice if you changed wikistats not to write to itself? :) [13:18:10] aye [13:18:12] yes [13:18:27] i have learned that lesson from you and i will work on that with average_drifter [13:18:36] i'm going to give group write perms [13:18:39] to it [13:18:41] average_drifter ^^ [13:18:42] and you can move whatever you need [13:18:51] that would be great! [13:19:07] just give me the permissions and erik z as well [13:19:14] then we will do the move ourseles [13:19:16] yeah [13:19:20] it is owned by erikz [13:19:23] and group owned by wikidev [13:19:25] try now [13:20:28] yep works [13:20:40] ty [13:24:32] ottomata: can't you just do `git clone -n URL foo && mv foo/.git /a/wikistats/` ? [13:24:46] instead of moving it first [13:25:43] what is the -n switch? [13:26:03] that means don't do a checkout after clone is finished [13:26:28] hm, maybe, but i think we don't know what changes have been made to wikistats...right? [13:26:42] right. so then git status ;) [13:26:44] would git be confused if a buncha stuff was different or moved? [13:27:01] that's kinda cool, btw, would never have thought of doing that [13:27:20] drdee, can I try that? I want to see what happens [13:27:22] * jeremyb usually does `git status -sb` [13:27:50] yeah but let's make a backup of wikistats first :) [13:27:53] ottomata: before you joined: 21 12:55:20 < jeremyb> so, i barely read anything yet about this new feed, but... how about pubsubhubbub? [13:27:56] hah [13:28:02] hah [13:28:04] backups are automated? ;) [13:28:12] because there is a huuuuge misfit between wikistats in git and wikistats on stat1 [13:30:19] if it is just in file changes, it will probably work [13:30:25] if there are deletions of file moves or renames [13:30:26] not so sure [13:30:38] it won't hurt to try, just wanna see what git status says [13:30:39] trying now... [13:31:03] drdee: http://i.imgur.com/Kmrb0.png [13:31:11] drdee: what can I do about this ? [13:31:22] hmm, jeremyb, doesn't look so good [13:31:25] just shows lots of deletions [13:31:34] no modifications [13:31:35] drdee: uhm, I could rebase them, but.. is it ok to have 2 commits per review ? [13:31:59] drdee: the second one is just what we talked about yesterda: removing filter2 and that stuff.. [13:32:04] uhmmmm, i think i need to abandon your first commit [13:32:11] ottomata: maybe it's the wrong part of the tree? is it all in a subdir or something? [13:32:14] drdee: so I should rebase ? [13:32:25] yeah probably [13:32:30] multiple subdirs [13:32:36] average_drifter: why don't you just push to a sandbox first so we can see what you have? [13:33:17] jeremyb: well, I could but, honestly, I talked with drdee and this is what I understood so far about the workflow [13:33:25] average_drifter: a given change (or review) can only have a single commit. but that could be a merge commit [13:33:27] jeremyb: I'm nto sure if I'm allowed to have my branch with my stuff in it [13:34:05] try to rebase [13:34:23] ok [13:34:48] ottomata: did you already undo the git thing? [13:35:05] * jeremyb points to http://lists.wikimedia.org/pipermail/wikitech-l/2012-July/061514.html [13:35:44] thanks for the link [13:37:22] read [13:37:30] nice, I'll make one of these sandboxes then [13:38:02] yes [13:38:16] git thing undid [13:38:32] mornin [13:38:40] moin [13:41:22] mooooonin [13:42:59] morning milimetric [13:43:05] howdy [13:43:48] average_drifter: looks like (from your screenshot) you have a really long first line for a commit msg. FYI, general git rules: first line should be a short+complete thought. (when printed alone without the rest of the msg it should make sense) my vim complains if it's >50 but 60 is probably ok too. (not longer). second line must be completely blank. 3rd and later lines should be normal email length (wrap around 70 or 75 chars) [13:46:11] jeremyb: alright I will try to conform [13:47:20] jeremyby: do you have some pointers on how to fix submitting two commits for average_drifter? [13:47:36] drdee: would help if i could see them. is there a sandbox? [13:48:22] just `git push origin HEAD:sandbox/driftersname/doublecommit` or something like that [13:48:42] obviously replace driftersname with your username [13:50:10] anyway, assuming he just forgot to reset to master before making a second unrelated change then it shouldn't be too hard [13:53:05] trying to rebase, the older commit doesn't want to go away [13:53:16] erm I mean the new one [13:53:38] I want to rebase it into the old one. I suspect that's how it should work [13:54:56] * jeremyb heads off to breakfast. will be back from there in ~10 mins [14:04:27] ottomata: i don't know a thing about npm and very little about node... but if it's anything like python then I don't see why you couldn't globally install both at once [14:04:53] i know very little about it too [14:05:13] maybe you can? [14:05:18] i mean it's not a problem from the deb perspective [14:05:23] right [14:05:33] the code just needs to be able to resolve the dependency properly [14:05:48] so if there's sometjhing like 'require foo-1.2' [14:05:49] maybe that's ok? [14:05:54] i think you'd have a virtual package and then have per version package names and have those Provide: the virtual one [14:06:13] ohhh [14:06:44] what you guys talkin 'bout? [14:06:50] ottomata: i was thinking different node versions for the different lib versions... idk. with python you could essentially just rename the folder and it would just work [14:06:52] .deb packages for nodejs stuff [14:07:14] does node have an equivalent of python's `import foo as bar` ? [14:07:18] ah, no, we're talking about different versions of the dependencies [14:07:19] i dunno [14:07:25] node is just javascript [14:07:35] well it's also sugar and glue [14:07:37] but serverside [14:07:40] and C/C++! [14:07:45] i mean like what's the problem [14:07:54] aye aye [14:08:09] uh, the problem is figuring out how to create .deb pacakges for nodejs apps [14:08:10] like limn [14:08:20] see the email thread started by ori about his websockets edit feed [14:08:20] ottomata: npm knows to pull the deps versions it needs for each individual package [14:08:25] milimetric: multiple versions of the same node lib installed on the same machine. how do you do the install and how do you choose at runtime which one to use? [14:08:44] average_drifter: we can't use npm [14:08:46] so does it help that checking out limn and going "npm install" pulls all the dependencies in? [14:08:57] not really [14:09:04] um, yes [14:09:07] drdee: I tried to rebase, how do you rebase ? I'm clueless.. I read some guides on SO and couldn't figure it out [14:09:08] it does, but not for what we are talking about [14:09:19] there are two solutions in that thread (read the thread :) ) [14:09:24] average_drifter: you could just push a sandbox... ;) [14:09:39] one is doing npm install and then creating the .deb package with all the deps included in the source [14:09:55] the other is creating separate .debs for each dep and having them as real debian dependencies [14:09:58] sorry I think I missed you pasting the thread [14:10:11] milimetric: it's just the most recent thread on analytics-l [14:10:13] its in the analytics email list (you are on that, right?) [14:10:25] hm, hopefully [14:10:40] i don't have anything today... when was it? [14:10:42] now [14:10:43] hehe [14:10:46] https://lists.wikimedia.org/mailman/listinfo/analytics [14:10:56] ok, analytics not analytics-l whoops. http://lists.wikimedia.org/pipermail/analytics/2012-September/000141.html [14:12:18] drdee: https://gerrit.wikimedia.org/r/24544 [14:12:23] it finally worked ! [14:12:28] git rebase -i HEAD~2 [14:12:56] ohhhhh, you're spetrea [14:13:23] jeremyb: yes, it is me, also known as wsdookadr on CPAN or github [14:13:47] jeremyb: nice to meet you Jeremy :) [14:14:06] i just know you from [[developer access]] ;) [14:15:29] average_drifter: so... you should delete the 2nd Change-Id: line and move the first one to the end (in that last push) and then repush [14:15:48] and then abandon the new one that was just created (that you just linked too) [14:15:52] err, to* [14:15:57] yep, i wasn't on that list :( but now I understand what you guys are saying [14:16:10] heh [14:16:18] milimetric: are you on other lists yet? [14:16:37] i'm on like 5 or 6 lists [14:17:30] so I take it ConvertingNodeNPM didn't work? [14:17:51] i think probably it's not been tried [14:18:01] whether it works isn't really relevant [14:18:07] jeremyb: so you mean I should amend the current commit right ? [14:18:14] average_drifter: yes [14:18:43] yeah, we're just discussing, nobody has really tried anything yet :p [14:19:00] but really it's irrelevant. /me is typing [14:19:33] hm, the problem is npm install resolves all the dependencies and installs them into subfolders thus allowing multiple versions on the same system to co-exist. Moving all the dependencies to .deb would eliminate that [14:20:29] milimetric: forget about npm and forget about .deb packaging. let's say you want to manually install a node lib on a system for all users of that system. where do you put it? now lets say you want 2 copies of that lib (at different versions) installed for the whole system. how do you do that? [14:21:31] maybe you just need a local structure for each app that's using them that mimics the way that they would look if installed by npm (in whatever subdirs) but instead of being new copies they are symlinks to the global copies [14:21:38] i think dschoon is going to have a much better answer but I'll think on it. Good problem statement, thanks [14:21:59] the global ones don't even have to be in a place that node would normally know to look [14:22:09] and puppet can make the symlinks for you [14:22:29] or your .deb can [14:22:51] just thinking out loud. i still don't really know anything about npm [14:23:40] well, the way i'd do it manually is to clone the repo into two separate directories, npm install, coke server, pass different ports and boom, two versions are serving on two ports [14:24:40] oh, no. it's not 2 versions of the same app [14:24:49] i believe a defining feature is that npm doesn't care about conserving storage at all. So it just happily downloads the same version of the same dependency into 20 different places [14:24:55] it's 2 independant apps that have a shared dep but that dep is different versions for each [14:25:00] but same solution would work [14:25:12] right, so npm doesn't try to share those deps ever [14:25:31] it's like - space is cheap, i'm my own package manager, i do what i want [14:25:37] right. so would it just work (tm) if we used symlinks instead? [14:26:03] and didn't even call npm. just created our own structure that looked like what npm produces [14:26:11] well, so would you set up the symlinks and then run npm install? [14:26:13] except symlinks instaed of real copies [14:26:18] no [14:26:21] or just use the packages.json and set up the symlinks and hope you did it right? [14:26:24] not run npm at all ever [14:26:26] k [14:26:37] * milimetric bad at typing and reading at the same time [14:26:39] well you could do it once manually [14:26:44] but not on each install [14:26:56] right, to get the stuff *to* symlink [14:27:03] yea, that seems to me like it would work [14:27:12] just to learn what it does. and then hard code it into the .deb [14:27:14] jeremyb, aside from elegance, what does using symlinks gain over just having multiple local copies installed by npm? [14:27:21] i got this slated for Oct. as soon as I finish d3 conversion with dschoon btw [14:27:58] ottomata: well among other things it means that you're diverging less from debian packaging policy [14:28:16] ottomata: i'm sure someone else (faidon?) could give a better answer though [14:28:36] not that we have to follow every part of the policy [14:30:16] but i think we should try to follow most of it by default rather than just make up our own rules from scratch or just have no rules/conventions at all. when we run into areas where following the rules is painful or we can't make it work then diverge at that point. my $.02 [14:30:29] aye, cool [14:30:36] i agree, [14:30:55] i'm not sure because I haven't tried, but I think that this might be one of those points where following the policy would be really painful [14:31:22] well i'm interested in working on it myself. maybe beginning of next week [14:31:23] npm wants to install the deps locally to avoid version mismatches and to keep things isolated [14:31:46] yep, my gut agrees with otto [14:32:02] you can't have a version mismatch if it's exactly the version that the app said it needed [14:32:07] unless the app is wrong! [14:32:21] well then both ways would break :) [14:32:35] yes [14:33:00] also (idk if it applies here or might sometime in the future but) if you diverge less from policy then you have an easier time getting the package into debian/ubuntu [14:33:03] so can you take a directory and make it into a deb? [14:33:15] e.g. if there were some 3rd parties that used limn and wanted it packaged [14:33:17] local symlinks from inside of ./node_modules/ to global locations *should* work, I just think it will be tough and annoying to fight debian/ stuff to make sure that it works all of the time [14:33:22] but it shoudl be possible, so meh? [14:33:25] no that applies for sure, I think limn has high aspirations [14:33:57] ottomata: why are you thinking it would only work sometimes? [14:34:01] but that'd also mean that we'd be responsible for creating .debs for all of it's node module deps [14:34:19] because I am I debian/ noob and wrestle with it a lot [14:34:28] i don't have a problem with that as long as it's automateable [14:35:01] but jajajja, I agree [14:35:02] IF [14:35:04] it does work [14:35:09] and is not a nightmare to maintain [14:35:16] then separate .deb packages are better [14:41:34] but what about release schedules and releasing major or breaking changes in minor version bumps? a la ruby. do we have to worry about that? or is the node community more sane? [14:42:37] and also what about the 1.2 version (of your mythical package). if 2.1 is already out then is 1.2 still being maintained? or do they only support the latest release? [14:42:57] (say e.g. for security bugs or backports) [14:43:13] brb [15:20:19] drdee: https://gerrit.wikimedia.org/r/24551 [15:21:04] drdee: I had to make another one of those. the patch set just kept getting bigger, so please abandon all other un-merged code reviews(except for the one above) [15:21:14] drdee: I mean except for https://gerrit.wikimedia.org/r/24551 [15:35:43] ok [16:14:53] average_drifter: it does not seem that https://gerrit.wikimedia.org/r/24551 is pushed against a remote branch [16:19:32] average_drifter: are you around? [16:23:03] drdee, dschoon, YAY! DSE is working and I ran my pig job [16:23:13] awesome!!!! [16:23:27] would love to play around with pig and hive as well [16:23:32] what data is available? [16:23:45] you can load in whatever you want, its not automatic right now [16:23:55] but, i have all of the sampled logs copied to analytics1001 [16:24:03] so you can copy those over and load them in if you want [16:24:05] sweet [16:24:09] lemme get you an account on the cluster [16:24:12] then I can show you how [16:24:13] one sec... [16:27:27] ottomata, do we need a separate kraken-script repo with pig and hive scripts? [16:31:34] we ahve one [16:31:39] https://github.com/wmf-analytics/kraken [16:31:44] its a working one [16:31:49] so not authoritative [16:31:55] kinda a scratch place for us to share stuff [16:32:01] mmmm, ok [16:32:04] but the pig job I ran is there [16:32:10] https://github.com/wmf-analytics/kraken/blob/master/src/pig/project_language_counts.pig [16:32:25] make it part of the wikimedia account? [16:32:38] ? [16:32:43] github wikimedia? [16:32:46] we don't have create-repo there [16:32:49] also: hi. [16:32:51] heya [16:33:00] and, naw, this should stay as wmf-analytics [16:33:03] it is for us only [16:33:09] i agree. [16:33:22] its just a place to hold buncha crap [16:33:22] i think our basic idea was exactly what otto said [16:33:32] that wmf-analytics is for team stuff [16:33:37] and wikimedia is for publishing things [16:33:45] things we think are ready for public consumption [16:33:52] but we are spreading our resource too thin, there are too many places we put stuff [16:33:53] ottomata: dse!! [16:34:00] drdee: i disagree. [16:34:04] we basically don't use less.ly any more [16:34:08] not since we got the nod for github [16:34:22] so our stuff is only in two places: github and gerrit [16:34:36] wikimedia doesn't count. it's our publications channel. [16:38:28] ok, drdee, wanna run through using hadoop w me real quick? [16:38:36] sure [16:38:44] ok log into analytics1003.eqiad.wmnet [16:38:46] i need to read the log, heh [16:38:52] i will start a screen there [16:38:59] dschoon, all nodes are up except an02 and an07 [16:39:04] still having install problems with those [16:39:06] cool. [16:39:07] an03 and an04 are seeds [16:39:15] haven't bothered with ops center yet [16:39:15] what seems to be the issue? [16:39:19] *nod* [16:39:22] i can look at that later. [16:39:28] dunno, they just hang when I try to make them do things [16:39:33] also, an01 has not been reinstalled [16:39:45] it is still lucid, but I am not running DSE there [16:39:46] *nod* [16:39:53] right, makes sense [16:39:58] since it's the proxy, bastion, whatever [16:40:00] brb a moment [16:40:02] aye k [16:40:10] what was the link with the ssh config again? [16:41:14] ottomata. i am in [16:41:31] cool [16:41:41] how do i join your screen? [16:42:15] agh, actually, it might be easier for you to start it [16:42:16] do this [16:42:18] screen -S hadoop [16:42:21] i will join yours [16:45:06] ok tell me what to do [16:46:29] hmm, i can't see your screen erg [16:46:44] you started a screen? [16:47:02] drdee? [16:47:09] on analytics1003? [16:47:18] yes [16:47:49] there it is [16:49:06] phew there finally [16:49:20] can you see what I'm typing [16:49:22] i am waatching : [16:49:24] :) [16:49:30] yes try again to be sure [16:50:02] mmm nothing atm [16:51:56] ottomata? [16:51:58] so [16:52:02] yes [16:52:08] ok cool [16:52:08] so [16:52:10] i can see that [16:52:23] the main interface is called 'dse', and it has a buncha options [16:52:42] you use 'hadoop' to interact with the hadoop file system [16:52:49] with the fs arg [16:53:09] let's go ahead and make a user directory for you to put stuff in [16:53:36] cool [16:53:41] back [16:54:11] ok, drdee, I guess i'll show you my pig stuff then [16:54:15] and then you probably know as much as me [16:54:16] git flow tutorial at drdee request [16:54:16] http://etherpad.wikimedia.org/FG3gMpB1qc [16:54:16] yes [16:54:28] dschoon! [16:54:29] hello [16:54:34] question [16:54:38] so you can see I loaded a day of sampled data in [16:54:39] howdy milimetric! [16:54:40] shoot [16:54:52] ottomata;yes [16:55:06] i've been scratching my balding head for a half hour or so - where is dygraphs hard coded as the chart type [16:55:17] I replaced everything grep was so kind to find for me [16:55:22] heh [16:55:30] we'll talk after scrum [16:55:35] and in the render of the GraphDisplayView it like CHANGES! [16:55:35] :) [16:56:01] k, this one would be interesting to understand, hopefully I can screen share [16:56:10] oh, btw, anyone have a good screensharing solution for linux? [16:56:41] woot woot [16:56:47] hangout works pretty good [16:57:12] drdee! [16:57:14] psh error! [16:57:18] yes [16:57:21] i notice :( [16:57:35] ERROR 2997: Unable to recreate exception from backed error: Error: Java heap space [16:57:35] i didn't get that when I ran it :) [16:57:54]