[00:01:30] hey milimetric [00:01:49] erm, we can talk tommorow, but if you have 1minute, I have a q [00:19:35] average: shoot [12:55:35] !card 4711 [13:01:44] !card is https://mingle.corp.wikimedia.org/projects/analytics/cards/$1 [13:01:44] You are not authorized to perform this, sorry [13:52:20] yööööööööööööööööö [13:52:35] that's the austrian way of saying yoooooooooooo ;) [13:53:05] ooooooooooooooooooʎ [13:53:37] Woops. That's backwards too. This is more like it: ʎoooooooooooooooooo [13:53:58] Lol. I read "Australian" [13:54:33] drdee: Are you still in Toronto? [13:58:14] aight [14:00:15] lambda 000? [14:01:12] drdee: ? [14:01:42] aight == yes [14:03:44] Oh. I was going to bro-fist for being early risers -- at least for PDT. [14:04:14] Is it just Dario & Toby on the west coast? [14:05:18] yup [14:05:46] qchris_away do you know a gentle vim tutorial for someone like me? [14:10:27] aight !== yes [14:10:29] just for the record [14:11:09] aight === I agree [14:44:01] drdee: Yes, Austrian use the Metal-Umlauts :-D [14:44:51] drdee: about vim... I guess the best wim tutorial is "apt-get install emacs", or somethings like that. [14:45:06] ROFLOL [14:45:08] dear lord [14:45:38] No, I am not really passionate about it, but I suck badly at vim. [14:45:57] I can use it, but I not know about good tutorials [14:47:45] i learn about one useful vim trick a month it seems, but I am not so good at it either [14:47:53] if i'm not editing locally, i use vim [14:49:11] I typically just use nano on remote machines. It's installed on many of the wmf machines, and it's easier for me than vim. [14:49:24] oh you aren't a vimmy, what do you use qchris? [14:49:26] emacs? [14:49:40] type vimtutor in the console. go through that tutorial. then consider Steve Oualine's book for further exploration ftp://ftp.vim.org/pub/vim/doc/book/vimbook-OPL.pdff . That is pretty much enough of Vim itself. Vim isn't that good if you don't use some of its plugins. Some useful ones: NERD commenter http://www.vim.org/scripts/script.php?script_id=1218 , NERD tree http://www.vim.org/scripts/script.php?script_id=1658 , Comm [14:49:54] * qchris blushes ... yes. [14:50:17] naw that's cool, i have no probs with emacs [14:50:23] * YuviPanda uses Emacs with Vim keybindings [14:50:23] it was the first editor I learned [14:50:26] hahahha [14:50:34] YuviPanda: why?! [14:50:37] best of both worlds! [14:50:41] haha [14:50:45] :-D [14:50:45] they just removed joe class from puppet generic-definitions :p [14:50:53] haha [14:50:55] i said they should have added Brion as reviewer.hehe [14:50:56] Evil mode in emacs has come pretty far now [14:51:01] and vimscript isn't really a programming language [14:51:17] A good alternative to Vim these days is Sublime Text 2 which comes with Vintage mode (which is a big subset of Vim). And Sublime Text 2 has a lot of plugins, and a plugin manager, it's an awesome editor, probably the best I've seen out there [14:51:23] and you can script Sublime in Python too [14:51:27] but emacs can make coffee [14:51:38] not open source, which is my biggest pet peeve about ST [14:51:47] remember TextMate? :D [14:51:52] btw, coffee.wikimedia.org :p [14:52:08] i made sure it survives until we have it implemented again :p [14:52:13] Mandatory reference: http://xkcd.com/378/ [14:52:24] mutante: D: [14:52:25] err [14:52:26] :D [14:52:30] I still use TextMate [14:52:35] for puppet and other things [14:52:40] for big projects I like sublime better [14:52:46] but TextMate does puppet better [14:53:00] YuviPanda: i once ran across it when cleaning up redirects.conf,it was a remnant from ancient office times when there actually was a webcam there showing the coffee maker [14:53:30] YuviPanda: and i just couldn't kill it.. Brion said $one_day we'll get it back [14:53:34] sigh, i guess we're not cool enough to have such a thing [14:53:36] anymore [14:59:42] And I forgot to mention. Most people who use Vim tend to save their .vim and .vimrc in a git repo. A rather "strange" habbit but a useful one. You wouldn't want to reconfigure your stuff each and every time on each machine you go on. Just scp-ing the files would be enough. [14:59:57] ottomata: did you reach any conclusion regarding disk space? [15:00:40] no, still doing ulsfo stuff :/ i need to verify my suspicions [15:00:44] but I am less worried right now [15:00:51] i'm pretty sure we'll be able to handle the mobile traffic with what we have [15:01:13] yeah, [15:01:26] if mobiles generate ~200G per day (uncompressed) [15:01:36] we have 18T for kafka storage [15:01:51] okay, goodie [15:01:54] (that 200G estimate is udp2log text data) [15:02:11] so if we were to do uncompressed udp2log mobile in kafka [15:02:13] that's 9 days [15:02:15] more than enough [15:02:36] so, if json is double the size [15:02:41] and snappy compresses at 30% [15:02:49] that's ~120G per day of json data [15:02:50] so we should be fine [15:04:42] how long do you need the data? [15:09:01] we were going to do the default of 1 week [15:09:05] but we can adjust if we need to [15:09:45] qchris: can you verify whether the w0 traffic issue has been resolved? [15:11:07] I looked ad the csv files this morning, and the sums did not look convincin. [15:11:21] I have not yet had time to dissect them. [15:14:06] mmmm yeah but w0 deployed a whole bunch of other stuff [15:14:10] yesterday as well [15:14:47] That's why I preferred to leave the bug open :-D [15:15:21] I'll do some more detailed checks now... [15:29:18] i sent an email about this [15:40:57] Mhmm? Without the analysis? [15:41:16] Ok :-/ [15:45:31] no; about the process [15:45:40] not about your findings [15:45:44] no worries [15:45:56] see " IPs and Redirects are live!!!" [15:46:27] qchris: ^^ [15:46:51] Ok. [16:43:40] average around? [16:43:56] milimetric: batcave? [16:49:07] coming drdee [16:50:42] (PS1) Milimetric: test gerrit gitflow [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86273 [16:50:43] (PS1) Milimetric: Merge branch 'hotfix/test1' [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86274 [16:57:41] (Abandoned) Milimetric: Merge branch 'hotfix/test1' [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86274 (owner: Milimetric) [16:57:50] (Abandoned) Milimetric: test gerrit gitflow [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86273 (owner: Milimetric) [17:01:26] qchris, ottomata: scrum [17:07:54] ottomata: what's the hostname of stat1001 please ? [17:08:02] ottomata: I need to login to it, and I forgot the name [17:08:04] stat1001.eqiad.wmnet [17:08:06] no [17:08:09] .wikimedia.org [17:08:14] ok , trying now [17:08:14] stat1001.wikimedia.org [17:09:36] ottomata: https://gist.github.com/wsdookadr/628744ab2a5f6c2fcf39 [17:09:44] can't connect to it [17:09:59] you don't need to proxy to get to it [17:10:07] you only need to proxy if you are logging into private hosts [17:10:23] so you probably don't need *wikmedia.org in your .ssh/config proxy list [17:16:51] (PS1) Milimetric: Merge branch 'release/datetimepicker' [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86276 [17:17:38] (CR) Milimetric: [C: 2 V: 2] "I'm slowly getting more and more confused :)" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86276 (owner: Milimetric) [17:19:45] drdee drdee_ ping [17:23:47] average, still having proxy problems? [17:23:52] i'm going to run and get lunch soon [17:23:55] gist your .ssh/config [17:26:40] ottomata1: now [17:27:30] ? [18:05:49] ottomata: hey [18:05:57] average: remove thesee [18:05:58] stat1001.wikimedia.org stat1002.wikimedia.org [18:05:59] ottomata: I linked you to a gist of my .ssh/config [18:06:02] ok [18:06:39] ottomata: still doesn't work [18:06:44] ottomata: wanna hangout ? [18:10:13] paste me gist of ssh -v with those hosts removed [18:10:38] average: ^ [18:12:55] ottomata: https://gist.github.com/wsdookadr/108d2de7b554bf96982a [18:14:13] oh average, you don't have an account on stat1001 :p [18:14:17] oh !? [18:14:21] hm [18:14:24] erm [18:14:35] ottomata: what can I do about this ? [18:14:45] ottomata: can I contact someone to add me ? [18:14:58] rt ticket, + 3 days wait :/ [18:15:13] ok [18:15:14] maybe i can just help you solve this right now though? [18:15:19] you have cron jobs running here? [18:15:36] I think so yeah. But I used to have access to that machine [18:15:44] I don't know why I don't have access now [18:15:49] you sure? [18:15:56] yo drdee [18:16:00] yo [18:16:07] ottomata: let me dig that gerrit review for puppet when we set up the cronjob [18:16:10] can we private chat about nuria? [18:20:15] ottomata: https://gerrit.wikimedia.org/r/#/c/72128/1/manifests/misc/statistics.pp [18:20:26] ottomata: this is the cron job merged together [18:23:01] that cron job runs on stat1002 [18:23:03] not stat1001 [18:23:04] right? [18:24:55] I'm not sure [18:25:02] one of stat1 , stat1001 or stat1001 [18:25:05] stat1002 [18:25:13] I suspect stat1001 [18:25:23] now, stat1002 [18:25:26] naw [18:25:30] uhm, ok [18:25:35] I'll try to connect to stat1002 then [18:25:37] /a/wikistats_git/pageviews_reports/bin/stat1-cron-script.sh [18:25:41] yeah you def have account there [18:25:44] so, just a summary: [18:26:20] stat1: public IP, no private data [18:26:21] stat1001: public IP, no private data, intended as just a production webserver host. [18:26:21] stat1002: private IP, private data, intended for number crunching and general computation [18:26:46] want me to run that script as the stats user and see what happens? [18:27:04] yes please [18:27:28] ottomata: but I'd still need access to it probably [18:28:47] (PS1) Milimetric: testing only [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86283 [18:28:48] (PS1) Milimetric: Merge branch 'hotfix/test2' [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86284 [18:29:12] (CR) Milimetric: [C: 2 V: 2] testing only [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86283 (owner: Milimetric) [18:29:37] (CR) Milimetric: [C: 2 V: 2] Merge branch 'hotfix/test2' [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86284 (owner: Milimetric) [18:30:40] Can't locate local/lib.pm in @INC (@INC contains: /root/perl5/lib/perl5 /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl .). [18:30:51] BEGIN failed--compilation aborted. [18:31:00] and lots of rm: cannot remove `/tmp/pageviews-full-cron/map/sampled-1000.tsv.log-20130628.json': Permission denied [18:31:09] also [18:31:10] rsync: getaddrinfo: stat1002.wikimedia.org 873: Name or service not known [18:31:14] stat1002.wikimedia.org is not a real addy [18:31:16] classic, I need access there [18:31:22] that's on stat1002 [18:31:29] ottomata: what's the real address [18:31:37] stat1002.eqiad.wmnet [18:31:42] thanks, trying with that now [18:31:47] but you probably want stat1001.wikimeidda.org, right? dunno what you are trying to do there [18:32:04] It worked ! [18:48:57] drdee_: report is running under my user [18:49:02] this does not fix the problem however [18:49:08] qchris: can we discuss monitoring please ? [18:52:02] ok average, let's figureo out why it doesn't run under stats user [18:52:30] average: Sorry. Still in the meeting. And I am bad at multi-tasking :-/ [18:52:33] looks like a path issue [18:52:35] right? [18:52:36] local/lib.pm [18:52:37] where is that? [18:53:04] qchris: that's ok, please ping me when you have some time [18:53:13] ottomata: yes, there are modules missing from the stats user [18:53:17] ottomata: let's try this [18:53:32] average: Ok. [18:55:17] ottomata: please add -I/home/spetrea/perl5/ as a switch to the perl command in the script you ran(the one that failed above) [18:56:13] ottomata: and then try running it again [18:56:25] oooooooook, but that means that this command isn't really productionized, but i will try it [18:56:49] wait, the script i ran is [18:56:54] /a/wikistats_git/pageviews_reports/bin/stat1-cron-script.sh [18:56:57] you want me to edit it? [18:57:04] yes please [18:57:15] and then if it works, we know what the problem is [18:57:30] and then we'll fix it by adding the modules to the stats user in a way or another [18:58:09] (a way: install them through cpanm on stats user which is a package manager for Perl. Oooor we use Debian packages and we should have all of them available in Ubuntu) [18:58:23] like this? [18:58:26] /usr/bin/env perl -I$MOBILE_PAGEVIEWS_DIR/lib -I/home/spetrea/perl5/ \ [18:58:26] ? [18:58:38] ottomata: yes [18:58:56] yeah seemed to help, different error now [18:59:03] /a/wikistats_git/pageviews_reports/bin/stat1-cron-script.sh: line 18: /tmp/cperlerr2: Permission denied [18:59:03] cp: cannot stat `/tmp/pageviews-full-cron/PageViewsPerMonthAll.csv': No such file or directory [18:59:17] rsync: getaddrinfo: stat1002.wikimedia.org 873: Name or service not known [18:59:49] ottomata: wipe out files /tmp/cperl* [19:00:22] as a root user if you don't have rights [19:00:24] ah, sorry, no [19:00:29] still have same error [19:00:30] i just missed it [19:00:31] Can't locate local/lib.pm in @INC (@INC contains: /root/perl5/lib/perl5 /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl .). [19:03:30] /usr/bin/env perl -I$MOBILE_PAGEVIEWS_DIR/lib -I/home/spetrea/perl5/ -I/home/spetrea/perl5/lib/perl5/ [19:03:35] how about this one ? [19:04:49] same error [19:04:50] but [19:04:50] notice [19:04:51] Can't locate local/lib.pm in @INC (@INC contains: /root/perl5/lib/perl5 /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl .). [19:04:55] @INC isn't changing [19:04:56] right? [19:05:29] unusual [19:05:33] ok so we have to install some modules [19:05:55] eek, fingers crossed that there are .debs [19:06:33] (PS1) Diederik: Fix determining latest commit from git. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86293 [19:06:56] (PS1) Milimetric: test 4 [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86294 [19:06:57] (PS1) Milimetric: Merge branch 'hotfix/test4' [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86295 [19:07:23] (CR) Milimetric: [C: 2 V: 2] test 4 [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86294 (owner: Milimetric) [19:07:39] (CR) Milimetric: [C: 2 V: 2] Merge branch 'hotfix/test4' [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86295 (owner: Milimetric) [19:08:41] (PS2) Diederik: Fix determining latest commit from git. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86293 [19:12:21] (PS3) Diederik: Fix determining latest commit from git. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86293 [19:19:00] (CR) Milimetric: [C: 2 V: 2] Fix determining latest commit from git. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86293 (owner: Diederik) [19:20:19] (PS1) Milimetric: Merge "Fix determining latest commit from git." [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86301 [19:20:20] (PS1) Milimetric: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/analytics/wikimetrics [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86302 [19:25:04] (Abandoned) Milimetric: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/analytics/wikimetrics [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86302 (owner: Milimetric) [19:25:10] (Abandoned) Milimetric: Merge "Fix determining latest commit from git." [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86301 (owner: Milimetric) [19:32:30] ottomata: ok I think I got it [19:32:33] yeah? [19:32:38] please delete line 6 in that file [19:32:47] in that bash script [19:32:50] and rerun it [19:33:09] ok... [19:33:30] same errors, but it looks like it is doing more stuff [19:35:34] hm, hangout ? [19:36:27] k [19:37:07] https://plus.google.com/hangouts/_/91a126200b88c976fe46039ab10c5573ed5e42ba [19:39:58] average: ^ [19:40:03] can't enter that [19:40:06] not allowed [19:40:10] calling you [19:40:13] hm, [19:40:54] let me add you to my circles [19:40:58] then maybe I'll be allowed [19:41:37] ? [19:41:44] can you start a hangout and send me the url [19:41:56] here, try this one: [19:41:56] https://plus.google.com/hangouts/_/c0f873d149cf0cd982e85fce7fa00438114a3347 [19:42:39] https://plus.google.com/hangouts/_/2a25c3513285d9c9c0ffda4ea3a3b4721f636d29?hl=en [19:42:43] ottomata: try this one ^^ [19:48:07] (PS1) Milimetric: test 5 [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86307 [19:48:08] (PS1) Milimetric: Merge branch 'hotfix/test5' [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86308 [19:48:41] (CR) Milimetric: [C: 2 V: 2] test 5 [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86307 (owner: Milimetric) [19:48:56] (CR) Milimetric: [C: 2 V: 2] Merge branch 'hotfix/test5' [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/86308 (owner: Milimetric) [20:05:45] average: are you still around? [20:06:58] qchris: yes sir [20:07:18] average: So if you want to talk about monitoring ... [20:07:25] average: (We're done with the other thing) [20:08:28] qchris: https://plus.google.com/hangouts/_/2a25c3513285d9c9c0ffda4ea3a3b4721f636d29?hl=en [20:08:44] Ok. [20:14:17] batcave ? [20:14:31] naw you guys were hogging the batcave [20:14:35] we are in our own cave [20:14:36] milimetric, qchris [20:14:38] what kind of cave? [20:14:41] slug cave? [20:14:42] noooo [20:14:45] glow-wormcave [20:15:05] http://nexttriptourism.com/wp-content/uploads/2012/11/light-Waitomo-glowworm-caves-is-beautiful.jpg [20:15:20] qchris, milimetric [20:15:22] please : [20:15:23] :) [20:15:35] average has qchris' attention [20:15:40] you wait your turn patiently. [20:15:43] :D [20:16:34] NO NO NO [20:16:47] * drdee_ is stampfeeting [20:23:06] coming drdee_ [20:26:52] drdee_: Andrew helped me solve the problem [20:26:55] drdee_: commit pushed [20:27:08] drdee_: it was depending on unpigz (which was basically uncompressing stuff) [20:27:20] and that binary was only present in my $HOME on stat1002 [20:27:55] and so Andrew installed the pigz package which provides pigz (and if you run that with -d you get the same thing as unpigz, in other words uncompressing gzips) [20:28:00] and uhm, yeah, that fixed it [20:28:07] now I'm gonna write some stuff to monitor it [20:31:26] average: nice [21:52:43] hey drdee_, I just deployed a datetime picker UI [21:52:55] card! card! card! car! [21:53:06] i had no card for it, it was kind of a pet project since Jessie said it'd be nice [21:53:20] let's get credit for it :) [21:53:22] sure [21:53:28] check it out though, let me know if you hate it [21:53:50] I thought it was the best one I've seen so far [21:53:58] k, i'm off for the weekend - have a nice one everyone! [23:37:13] bye