[13:20:21] morning [13:30:42] morning milimetric [13:30:46] morning ottomata [13:30:49] hi [13:30:50] :) [13:30:54] guys bat cave to talk about kriipke? [13:31:05] brb bathroom, but yes [13:31:08] k [13:34:16] i'm in the batcave [13:38:37] moorning! [13:38:42] drdee, mind if I reboot stat1001? [13:39:01] no [13:39:12] no you don't mind? :) [13:39:33] i'm lonely in the batcave [13:40:23] no i dno't mind [13:40:33] ottomata, bat cave :D [13:41:26] baattcaave, but i don't wanna, i'd have to move so as not to disturb my coworkers! [13:41:29] what's up? [13:42:33] kripke [13:53:21] milimetric; i pushed a change to fix the version info [13:53:25] can you review it [13:53:25] ? [14:09:43] milimetric; https://gerrit.wikimedia.org/r/64252 [14:09:56] sure [14:11:27] i don't have +2 on that repo of course [14:11:30] but i fixed it [14:11:36] *reviewd it [14:20:10] you don't have +2 ? [14:20:23] ohh right of course [14:20:29] sorry i should have asked ottomata [14:20:38] ottomata: can you merge https://gerrit.wikimedia.org/r/64252 [14:21:20] cool! milimetrics, i'm playing with the puppet thing [14:21:24] once you have self hosted puppet setup [14:21:32] it looks like its pretty easy to use a node definition rather than labsconsole [14:21:33] if you want to [14:21:40] so, I made a file [14:21:45] misc/limn0.pp [14:21:56] node 'limn0.pmtpa.wmflabs' { [14:21:56] file { '/tmp/node.txt': content=> $hostname } [14:21:56] } [14:22:07] anything that include or done in that node will be applied to limn0 [14:22:13] (file name is arbitrary) [14:34:00] milimetric: drdee are you guys using git review on user-metrics? [14:34:21] yes [14:34:32] i did for most things [14:34:42] but a couple weeks ago i had some problems [14:34:44] so i just pushed [14:34:54] sorry, i know that makes it confusing [14:35:02] hehe [14:35:03] but i prefer it to spending hours fighting with gerrit [14:35:18] yeah, I'll give it a try [14:35:27] i've never actually used git review [14:35:37] any specific words of warning? [14:37:08] don't [14:37:14] :) [14:37:19] i suspected you might say that [14:37:27] so from what i understand, you can do everything it does without the nightmares [14:37:32] milimetric: do you actually think I shouldn't [14:37:43] yeah [14:37:51] nahhhh [14:37:57] as soon as I have a minute i'm going to head over to that tutorial that explains how to do without git review [14:37:59] let's do best practice here [14:38:04] from what i understand most people have stopped using it [14:38:12] gerrit or git review? [14:38:15] git review [14:38:19] ok [14:38:26] gerrit is a bitch, but of course we want to use that [14:38:28] but we should not push direclty [14:38:33] yea [14:38:35] agreed [14:38:37] we should review each other's code [14:38:42] it was git review that was giving me troubles anyway [14:39:01] i agree we should, though i think pull requests would literally save us a day per month minimum [14:39:04] i guess I'm wondering how i should push the fix which breaks upload of user_ids [14:39:13] ok [14:39:16] so for now [14:39:26] as long as you're clean and only have one commit ahead of origin [14:39:33] (if not, you should rebase) [14:39:42] you can do this: git review [14:40:21] oh, do you have it installed? [14:40:26] yeah I have it installed [14:40:52] so that is part of my concern, i do have more than one commit [14:41:29] btw, drdee, that is merged [14:41:31] milimetric: [14:41:52] how many of those limn instances on kripke need to run on different limn code bases? [14:41:53] all of them? [14:42:04] i think they should [14:42:13] that way people can upgrade independently [14:42:16] ok [14:42:37] milimetric: I've four commits: 1c9264c (HEAD, master) fixes bug which breaks upload when using user_ids [14:42:38] f9455e0 Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/analytics/user-metrics [14:42:39] 07e8297 Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/analytics/user-metrics [14:42:40] 24f9d92 adds unicode test file which includes a user_id [14:43:00] none of them ottomata [14:43:13] but maybe erosen is right [14:43:26] as in, right now, all of them are upgraded basically to the latest as far as I know [14:43:42] your call, I was just thinking about the situations where different people needed different configurations temporarily [14:44:14] different configs should be possible, we should factor out the config from the src [14:44:24] but yeah, maybe for now it's best to keep them separate [14:44:32] i guess I menat config loosely [14:44:43] like enabling anonymous graph editing or something [14:45:00] hm, I think if someone really wanted to go their own way they should spin up their own labs instance [14:45:39] but that's not easy yet, so we should make that easy [14:45:48] i guess you would know best whether you've had different versions running on different dashboards, intentionally [14:46:12] so far everyone's just said - just bring me to latest [14:46:24] gotcha [14:46:41] so my opinion is that if two people want it to behave slightly differently, we should build that into the app when we create a new feature [14:47:09] so like we're ready to launch anonymous graph creation - if someone doesn't want it, we disable it by default and make it configurable [14:47:25] gotcha [14:47:28] that sounds good [14:47:29] k, sorry, back to your git review [14:47:31] ok it would be much simpler right now [14:47:37] for me to have a single deploy [14:47:39] yep [14:47:40] do that [14:47:44] milimetric: that would mean when you deploy [14:47:49] to this new limn0 labs instance [14:47:53] you upgrade all of the limn instances there [14:47:57] yep, course [14:48:01] ok cool [14:48:06] great [14:48:10] i like that, i think it's better [14:48:41] and i'll update the limn deployer to only do data deployments by default [14:49:48] ok erosen so your four commits are just local right? [14:49:54] haven't been pushed anywhere [14:50:35] yeah [14:50:36] just local [14:50:47] ok, git rebase -i, then pick, squash, squash, squash, then git review [14:50:47] oyy [14:50:48] okay [14:50:52] i'll give it a try [14:52:35] milimetric [14:52:38] can you try deploying? [14:52:43] shoudl I just git clone? [14:52:47] at the source directory? [14:52:51] and let you handle deploys later? [14:52:51] to the new instance? [14:52:54] yes [14:52:55] it's limn- [14:52:56] uh [14:52:58] limn0 [14:53:11] limn0.pmtpa.wmflabs:/usr/local/share/limn [14:53:14] gerrit folk, have you ever seen this error message: remote: ERROR: missing Change-Id in commit message footer [14:53:20] ye [14:53:21] in git review [14:53:21] yes [14:53:23] yes [14:53:24] just do [14:53:29] git git commit -a --amend [14:53:42] hm, ok, but are there instances set up for each instance on kripke? [14:53:42] but also make sure you have installed the hook [14:53:56] drdee: based on: Hint: To automatically insert Change-Id, install the hook: [14:54:01] scp -p -P 29418 erosen@gerrit.wikimedia.org:hooks/commit-msg .git/hooks/ [14:54:03] yup [14:54:05] k [14:54:16] so first install hook [14:54:20] gotcha [14:54:22] right, sorry erosen, forgot that [14:54:26] then the git commit -a --amend [14:54:48] ottomata: are all the instances set up? [14:55:03] New patchset: Erosen; "adds unicode test file which includes a user_id" [analytics/user-metrics] (master) - https://gerrit.wikimedia.org/r/64315 [14:55:03] so like /var/lib/limn/instance-name [14:55:10] no, 'im just messing with dev-reporcard right now [14:55:14] it isnt' quite righ tyet, but it should be there [14:55:27] ok, so i'll deploy that one [14:55:30] /var/lib/limn/dev-reportcard/ [14:55:33] k [14:55:33] k [14:55:49] drdee: milimetric: looks like it worked [14:55:53] thanks for the help [14:56:16] yep, i'll review it shortly [14:58:36] npm's not installed ottomata [14:58:48] do we need npm? [14:58:53] i thought we could deploy with out it [14:59:06] nodejs we need though [14:59:08] no, that's what we were talking about earlier :) [14:59:20] we definitely need npm for the current deploy method [14:59:36] still confused about that I guess [14:59:44] to not-need it, we should create an npm mirror or partner with parsoid to use theirs [14:59:53] for production, yeah [14:59:56] but for labs, why? [15:00:06] npm install installs in local dir, right? [15:00:10] can't we npm install local and rsync? [15:00:17] local like my box you mean? [15:00:24] hm... [15:00:36] well [15:00:38] local as in local dir [15:00:41] ./node_modules [15:00:47] but yeah, i guess [15:00:50] wherever you are deploying from [15:00:54] so fabric is running on my box, right [15:01:00] hm, yeah, i could do that [15:01:17] this'll take a bit [15:01:25] what does fabric do right now? [15:01:32] rsync + remote npm install? [15:01:33] just runs npm install over ssh [15:01:55] so i'll have to git clone to some temp directory locally, npm install, then rsync the tmp/node_modules [15:02:40] and hm, i might have to change the cokefile to use the local coco [15:02:41] k [15:03:33] should we not deal with this now? [15:03:35] I can install npm [15:04:19] milimetric: ^ [15:04:19] ? [15:04:36] hm [15:04:42] it's better the way you say it [15:04:54] and it might get us much closer to be able to do this in prod [15:05:02] it shouldn't take more than 30 min. [15:05:14] but if you want, you can just install npm for now, and i'll tinker on this in the background [15:05:35] hm, ok, i guess that won't hurt [15:05:39] if I install npm now you can deploy? [15:05:43] yep [15:07:14] ok done, let's try it [15:07:17] deploy if you can [15:07:34] yep, it's running [15:07:46] hm, i think we might also need coco globally installed [15:08:06] but it'll work without for dev [15:08:21] that might be a little trickier, 'cause the Cokefile assumes global install [15:08:50] hm [15:08:53] that is probably ok [15:08:59] if there is a coco.deb [15:09:08] that's not a big deal since coco isn't a package manager [15:09:15] there's definitely not a coco.deb [15:09:29] we just do npm install -g coco [15:09:43] oh what's the job called [15:09:46] not dev-reportcard? [15:09:51] limn-dev-reportcard [15:10:14] k, rerunning [15:10:19] yeah, not quite working [15:10:30] no it's not done yet [15:10:34] oh [15:10:34] k [15:10:52] it says unknown instance limn-dev-reportcard [15:11:13] Requested: stop limn-dev-reportcard [15:11:29] [limn0.pmtpa.wmflabs] out: stop: Unknown instance: [15:11:37] May 17 15:10:12 limn0 kernel: [ 3897.858525] init: limn-dev-reportcard main process (21990) terminated with status 1 [15:11:39] eh? [15:11:49] i dunno why its failing but what says unknown instance? [15:11:58] how are you trying to stop/start it? [15:12:02] stop limn-dev-reportcard [15:12:47] yeah, i ssh-ed into it and it says the same [15:13:07] its not running yet [15:13:10] [@limn0:/usr/local/share/limn] (develop)[24877dd] 130 ± sudo stop limn-dev-reportcard [15:13:11] stop: Unknown instance: [15:13:11] [@limn0:/usr/local/share/limn] (develop)[24877dd] 1 ± sudo start limn-dev-reportcard [15:13:11] limn-dev-reportcard start/running, process 23326 [15:13:17] but won't start up, not sure why yet [15:13:19] there is not vardir? [15:13:20] hm [15:13:25] ± ls /usr/local/share/limn/var [15:13:25] ls: cannot access /usr/local/share/limn/var: No such file or directory [15:13:40] yeah, gets created after it tries to stop it i htink [15:13:50] to start? [15:14:08] no it does stop, generate var, start [15:14:22] so the fact that stop is erroring is messing up the other step [15:14:27] i'll do it manually but we should fix that [15:14:35] it's so odd right - why would it say unknown instance [15:15:42] i think its saying that there is no process running [15:16:16] hm [15:16:16] http://stackoverflow.com/questions/9641996/upstart-node-js-app-stop-unknown-instance [15:16:40] naw, not related [15:16:41] hm [15:17:42] had to install coke globally, i'll fix after [15:19:15] hm ok, still not happy though [15:19:38] yeah, just finished [15:20:11] but you're right, it's stopped/waiting [15:20:17] where are error logs? [15:20:25] i'm looking at [15:20:29] sudo tail -f /var/log/syslog /var/log/limn/limn-dev-reportcard.log [15:20:32] but not seeing anything but [15:20:35] May 17 15:20:06 limn0 kernel: [ 4492.633952] init: limn-dev-reportcard main process (27041) terminated with status 1 [15:21:33] Failed at: ./server/server.co [15:21:33] Error: Cannot find module './config' [15:21:33] at Function.Module._resolveFilename (module.js:338:15) [15:21:33] at Function.Module._load (module.js:280:25) [15:21:33] at Module.require (module.js:362:17) [15:21:33] at require (module.js:378:17) [15:21:33] at Object. (/usr/local/share/limn/server/middleware.co:20:10) [15:21:44] ah! [15:21:45] gr [15:21:48] where's that log [15:21:58] i had to run coco server.co manually [15:22:05] sudo -u limn ./node_modules/.bin/coco ./server/server.co [15:22:10] hmmm before you fix [15:22:17] let me see if I can get that log into a log file [15:23:06] so that .config file is interesting, you could create it in puppet i think [15:23:23] it has a secret that can be randomly generated, and two properties that just depend on the address it's serving on [15:23:46] then a list of emails, which could default to empty and people could edit themselves [15:28:12] cool, milimetric, stderr should now to to the limn log file [15:28:17] cool [15:28:23] ok so wha? [15:28:26] need ./config? [15:28:28] one sec, i'll move the config in [15:28:31] k [15:28:37] is this config global to all instances? [15:30:31] nope, and that's a problem [15:30:45] oh yeah, crap, that's a fairly big problem [15:30:49] i'll have to fix that [15:31:05] that's ok, it'll work for the most part [15:31:08] conifg can go in /var/lib/limn/dev-reportcard/config? [15:31:08] except the signing in [15:31:29] i already put it in the right spot [15:31:32] ok [15:31:34] it's in server/config [15:31:39] *server/config.json [15:31:45] i'll try startig it [15:32:00] heyayyy [15:32:02] that looks better [15:32:09] k, cool [15:32:19] so, my todo: fix deploy to do npm locally [15:32:27] fix config to go in var [15:32:53] you should make config dir configurable via env var too [15:33:07] isn't it ok that var is configurable? [15:33:13] hmmmMmmmmMMMmmmmm [15:33:22] if faidon was here he would say to put config in /etc [15:33:27] separate from /var [15:33:30] but hmmm [15:33:33] i think its fine for now [15:34:11] oh i see [15:34:27] buuUuuuut, its ok [15:34:37] do it in var_dir for now [15:34:40] um, ok, this thing is running, right!? [15:34:55] as far as i know, yeah, we'd have to switch the ips and all that of course [15:36:28] yeah [15:37:00] oof so many domains [15:37:00] ha [15:37:02] k [15:50:03] [travis-ci] develop/c160f68 (#135 by milimetric): The build is still failing. http://travis-ci.org/wikimedia/limn/builds/7254924 [15:50:03] milimetric, where did you send me that list? [15:50:34] of limn instances we should port over? [15:57:31] also, milimetric, is this necessary? [15:57:32] LIMN_DATA="./var/data" [15:57:39] for dev-reportcard [15:57:43] nope [15:57:45] ok cool [15:57:47] that's artifact [15:58:08] wait, you should only be adding like one line to a puppet config for each instance right? [15:58:09] ok so ya gimme final list of limn instances we need again [15:58:11] yes [15:58:16] https://gist.github.com/milimetric/5599167 [15:58:26] great, danke [15:58:43] danke YOU [15:58:44] :) [15:58:52] you said no global-dev, right? [16:00:45] no global-dev right [16:00:48] gp took its place [16:01:00] oops! sorry i forgot it on there [16:01:18] ok [16:01:18] ottomata: ^ [16:01:20] :) [16:11:11] milimetric: , what's this? [16:11:11] http://kripke.wmflabs.org/backbone-base.html [16:11:19] no idea [16:11:25] looked like something stefan was playing with [16:11:29] but hadn't been updated in a while [16:11:33] i think it's safe to ignore [16:11:35] hm ok [16:11:43] i wouldn't kill kripke or anything, just take away its ip [16:11:46] and tell people [16:11:59] ok cool [16:12:14] all limn instances are ready for deploy [16:12:52] check it: [16:12:52] https://gist.github.com/ottomata/5600160 [16:12:52] so easy! [16:40:22] ottomata, I'm not sure if domas uses IRC, but perhaps you know - are page view stats for foundation wiki included in the data dumps and if they are where would I be able to find them? [16:41:15] ottomata: if I'm connected over ssh, is there a general way to refer to my own laptop? [16:41:27] so basically I'm trying to rsync node_modules from local to remote [16:41:38] but i don't have permissions to do certain things that way [16:42:03] so i'd like to rsync over fabric's ssh so I can treat the remote box as local [16:42:13] then i can do sudo [16:43:33] Thehelpfulone: see https://gerrit.wikimedia.org/r/gitweb?p=analytics/webstatscollector.git;a=blob;f=filter.c;h=907636cbfd5acc986b4fdc34f1aa73c733ae5704;hb=HEAD [16:44:01] btw, ottomata, your gist with the limn instances has a type-o: test-reportcard should be on port 8091 I think [16:44:10] foundation wiki should be included in dumps.wikimedia.org/other/pageviews/ [16:44:12] drdee: any idea on my rsync q above? [16:44:19] 1 sec [16:45:28] drdee, I asked Henrik who runs that tool, https://en.wikipedia.org/wiki/User_talk:Henrik#Page_view_stats_for_foundation_wiki any idea which of those it is? [16:46:24] maybe en.f? [16:46:51] yes en.f sounds likely [16:49:12] Thehelpfulone: check lines 69-81 [16:49:14] of https://gerrit.wikimedia.org/r/gitweb?p=analytics/webstatscollector.git;a=blob;f=filter.c;h=907636cbfd5acc986b4fdc34f1aa73c733ae5704;hb=HEAD [16:49:20] yeah just found it, thanks [16:49:24] there you see how the acronyms match to the projects [16:51:26] Change merged: Milimetric; [analytics/user-metrics] (master) - https://gerrit.wikimedia.org/r/64315 [16:52:30] ja, milimetric, saw that typo too [16:52:31] fixed it [16:53:00] so any idea how i can rsync a folder to the remote without it yelling at me? [16:53:15] oh i guess i could rsync it to /tmp remotely and then move that over with sudo [16:53:36] eh? [16:53:38] for deploying? [16:53:53] (sorry, my irc client crashed sec ago, missed some chats) [16:56:08] np, yeah, i think i might have it, fiddling... [17:01:59] guys scrum [17:02:07] milimetric: ^^ [17:02:07] ottomata: ^^ [17:02:13] doh! [17:03:07] doh! [17:18:52] so! milimetric, i think limn0 is ready to go! [17:19:00] as soon as you've got the deployment stuff all worked out [17:19:07] i think I can move the IP [17:19:50] cool, i'll ping you as soon as i get them all staged out and deployed [17:20:13]