[07:44:16] hello drdee [07:44:38] drdee: I have had some conversation with a friend who uses git also [07:44:51] drdee: I would like to make a proposal for the workflow [07:45:29] drdee: I would basically like to propose that I use a sandbox for my development, and then merge --squash all commits of my sandbox to a branch specifically made for review [07:46:27] turning back to wikistats for the moment [07:47:46] drdee: I basically understood why the workflow you told me uses git commit -a --amend, because that effectively does not create a new commit, it's the same commit instead, it is modified with the new changes, and the old one is lost [07:55:27] drdee: please ask Erik if he could join this channel [07:55:35] drdee: I am missing a file called SquidDataSavesPerCountryMonthly.csv [08:50:09] solved [08:53:11] drdee: my mistake last week when we had the many abandoned reviews was that I was so used to git commit -am ".." [08:53:26] that I was using it without thinking it would be interfering with the current workflow [12:34:11] average_drifter: mornign [12:34:36] drdee: hey [12:34:52] drdee: I'll be here, take your time :) [12:34:56] did you submit your hours on oDesk? [12:35:07] drdee: no, sorry, I need to look how to do that [12:35:14] you should :) [12:36:53] it takes a bit to figure git out [12:37:03] i mean gerrit [12:37:11] so sandbox is not necessary [12:37:16] just push to a branch [12:37:25] and keep using amend to improve the commit [12:37:40] but keep every single functionality in separate branch [12:38:26] alright [12:47:47] what is the status of webstatscollector? where you able to push it to a remote branch? [12:52:34] alright [12:52:36] damn [12:52:41] drdee: yes [12:52:45] drdee: please see the last review [12:53:14] drdee: is decoding the %B5 stuff desirable ? did that too [12:53:25] how? [12:53:33] drdee: a function called decode_title [12:53:37] brb [12:53:46] wait [12:54:51] back [12:54:54] here [12:58:01] morning milimetric :) [12:58:15] morning drddee [12:58:25] how's your weekend [12:58:39] average_drifter: please explain the lookup table [12:59:05] milimetric: just baby stuff buying mode :D [12:59:14] carseat, check [12:59:27] oh cool :) that must be exciting [12:59:38] veryyyyyy [13:00:09] i am sure your weakend was more exciting :D [13:00:23] drdee: lookup table is used to convert a lower and upper A-F to 10-15 and digits to their normal form [13:00:43] drdee: that is actually used to convert %B5 to a an actual char [13:01:22] so this only applies to ascii control characters? [13:02:06] drdee: well they are converted to actual characters. reading them is a different issue [13:02:15] drdee: remember that sample file that was in the initial task ? [13:02:21] yep [13:02:21] drdee: it has some requests for chinese pages [13:02:25] right [13:02:33] drdee: turns out with the decode_filter, we get the actual chinese characters [13:03:05] ok, so that way it is human readable [13:03:09] yes [13:03:16] that sounds like a good plan [13:03:24] thx [13:06:28] the max title length is 2000 characters, right? [13:07:08] drdee: yes because each non-ASCII letter weighs 3bytes [13:07:18] drdee: and if you look in the sample you will find very long titles [13:07:39] yes so i am worried that 2000 might be too short [13:07:54] drdee: because you have 3 bytes per non-ASCII letter, but that takes(for example %B5) actually 3 bytes to encode, so for each non-ASCII char we have 9 bytes [13:08:16] drdee: alright I can set it to 10000 [13:08:22] to be on the safe side [13:08:32] http://en.wikipedia.org/wiki/Wikipedia:Naming_conventions_(technical_restrictions)#Title_length [13:08:42] so the title length is max 256 [13:09:28] ok, but those 256 are bytes ? or just letters not depending on encoding ? [13:09:30] so can we calculate the maximum length when it is decoded? [13:09:37] drdee: yes [13:10:04] drdee: what do we do if it's over 256 ? truncate ? [13:10:29] yes, truncate it although it is not ideal [13:10:42] i rather know the maximum decoded length of a title [13:16:46] morning ottomata [13:17:03] average_drifter: i added comments to your code [13:17:06] https://gerrit.wikimedia.org/r/#/c/24551/ [13:17:17] morning! [13:17:35] you didn't hug the tiger in the bronx zoo, right? [13:18:08] hug the tiger [13:18:09] no [13:18:11] no tiger hugging [13:18:15] not that I remember anyway [13:18:15] i mean, i *know* you love panda's ;) [13:18:43] you didn't hear it on the news? [13:22:12] nope [13:22:23] drdee: thanks, what branch should I pull from ? [13:22:36] drdee: oh, I'll look at the link first [13:22:53] guy went to the zoo, wanted to 'be one' with the tigers so i climbed into their domain [13:23:14] tiger != lolcatz [13:23:27] he did survive but was pretty beaten up [13:23:30] anyways [13:23:59] haha [13:26:22] would it be an idea to start setting up udp2log and stream data from the firehose into kraken for a 24-48 hour period to just see what happens and what kind of issues we would face [13:26:24] ? [13:28:05] if you're asking me, I'd totally be interested in taking the filter for a real-world scenario [13:28:12] but I think you asked ottomata not me [13:28:24] :) [13:28:33] yes webstatscollector we will deploy this week [13:29:00] drdee: how do I access stat1 ? [13:29:03] yeah we'd like to do that as soon as things are more up and stable [13:29:09] i still have 2 machines that won't boot [13:29:21] there is an RT ticket for RobH to check out one of them today [13:29:25] gotta try one more thing with the other [13:29:43] average_drifter: you can't yet, i requested access for you [13:30:29] alright [13:32:13] drdee: Sumana answered by saying I should not send my pubkey through mail [13:32:18] drdee: but through a link [13:32:24] ottomata, right but are those two final machines on the critical path for udp2log? would also be nice to just add them to a working cluster to see how that would go [13:32:25] drdee: should I gist it and send the link to him ? [13:32:40] drdee: I'm not sure if this is an impediment due to regulations [13:33:05] average_drifter: well that's for SVN access and we are already working on the git migration so maybe it's not worth pursueing [13:33:06] that's true [13:33:18] we should be able to spawn up machines while the cluster is running [13:33:19] so yeah [13:33:20] would be fun [13:35:26] http://vagrantup.com/ [13:35:27] i just want to get real experience asap :) [13:39:13] ottomata, on what machines did we start experimenting with webstatscollector, was that build1? [13:40:25] bwerrrr [13:40:27] log1? [13:40:29] build1? [13:40:36] probably log1 [13:40:43] it has mediawiki and other log stuff set up [13:41:52] nope, it is build1 [13:42:01] i am gonna add average_drifter, okay? [13:45:56] average_drifter: what is your username on labsconsole.wikimedia.org? [13:46:14] yeah sure [13:49:07] average_drifter: ^^ [13:49:07] drdee: spetrea I think [13:49:22] drdee: my ssh username is spetrea [13:49:26] drdee: the other username is stefan.petrea [13:50:11] what username do you use to login to https://labsconsole.wikimedia.org/wiki/Special:NovaProject? [13:50:58] checking [13:51:17] drdee: stefan.petrea [13:51:25] grumble [13:51:48] incorrect ? [13:52:33] drdee: Stefan.petrea [13:52:54] it is correct but i need an additional right, hold on [13:53:03] ok [13:53:57] average_drifter: fill out https://labsconsole.wikimedia.org/wiki/Help:Contents#Requesting_Shell_Access [13:54:20] mention that you are a WMF external contractor and work with Diederik (drdee) with the analytics team [13:54:32] milimetric: did you also fill out https://labsconsole.wikimedia.org/wiki/Help:Contents#Requesting_Shell_Access as part of your onboarding? [13:55:01] drdee I don't think so, I'll do that now [13:55:05] cool [13:55:54] drdee it says I don't have access to edit the Shell_Access_Request page [13:56:02] oh, sorry [13:56:05] nvm [13:58:01] drdee: done [14:04:25] ty [14:34:32] brb guys [15:35:24] average_drifter: are my comments regarding webstatscollector clear? [15:41:35] drdee: yes, merging them in [15:45:22] brb, update restart [15:49:43] cool [16:01:29] drdee: how do we test to see if the collector knows how to read these characters ? [16:01:49] drdee: we might need to use some UTF-8 api [16:01:53] feed it enough test data, and check the outputted of collector [16:02:03] ok [16:02:26] (the csv files that are written from the berkeley db) [16:24:55] hey ottomata, it seems that the edits filter is no longer running or syncing to stat1, if you have some spare cycles could you have a look? [16:28:38] edits filter on stat1? [16:28:47] oh syncing [16:28:48] i see [16:28:49] k [16:40:10] mornin! [16:41:25] mornin [16:41:34] have a good weekend? [16:42:10] eh. [16:42:23] looking forward to next weekend, when all this moving is finally done. [16:43:51] you got people doing it? [16:43:58] every time I move I swear I'm gonna do that [16:45:01] morning dschoon! [16:45:11] yeah, i have a guy. [16:45:24] but more importantly to me, it's just a source of stress and distraction [16:48:36] drdee: collector says Got the message: ja 1 25535 大友克洋. [16:48:48] drdee: still waiting for it to write to disk [16:48:52] looking good! [16:48:53] drdee: so I can see what it's actually writing [16:48:57] nice [16:53:46] milimetric: you might want to pull dsc. i pushed some stuff i was fiddling with over the weekend. [16:54:11] cool, doing so now [16:55:35] writing zero to disk [16:55:36] dunno why [16:56:06] drdee: do you use screen ? I'll set up a screen for us two so we can look at this [16:56:08] drdee [16:56:09] edits sync? [16:56:13] whatcha talking bout? [16:56:22] ottomata: yes the edits.log files [16:56:31] they were synced until august 31 to stat1 [16:56:39] or are they only synced every month? [16:56:56] maybe i just did it manually once [16:57:00] because I don't have a cron job for that [16:57:05] ahhhh… [16:57:20] could you add those to the cronjob? [16:57:45] https://plus.google.com/hangouts/_/2e8127ccf7baae1df74153f25553c443bd351e90 [16:58:23] average_drifter: yeah i can watch in about 15 minutes [17:01:00] hey very cool, was doing the same stuff but manually to get more xp with d3 [17:01:14] https://plus.google.com/hangouts/_/2e8127ccf7baae1df74153f25553c443bd351e90 [17:08:48] https://github.com/embr/userstats [17:19:32] yo average_drifter [17:20:00] what machine do you want me to connect? [17:27:22] hello ? [17:27:32] network problem, storm is building up outside [17:30:59] drdee, what's the cleanup? [17:31:39] low priority, and i need to delve into it myself a bit more but it came from erik m [17:33:46] mk, soo, request! [17:34:00] maybe twould be better not to assign tasks to me unless they are tasks? [17:34:01] hehe [17:34:15] i know what it means, it is low pri [17:34:23] assign it to you then? [17:34:28] sure [17:34:30] and when you want me to do it, assign it to me [17:34:37] otherwise when I look at my list I see a buncah stuff I can't actuallyw ork on [17:35:18] k [17:35:34] mmmk, danke [17:38:02] back [17:38:34] yoyo [18:38:00] cascading failures: http://i.imgur.com/M4E9u.gif [18:47:59] oohhhhh and as this is monday….. could everybody update / add and close relevant asana tasks? [19:01:29] milimetric: i will add you now to a labs group so you can join us in our digital sandbox [19:01:47] drdee: cool [19:03:59] milimetric, average_drifter: try connecting to labs bastion like: [19:04:09] ssh -A bastion.wmflabs.org [19:04:52] permission denied public key [19:05:41] grumble [19:05:56] have you set up your public key with ops yet? [19:06:04] aka, "requesting commit access"? [19:06:07] yes [19:06:14] ...i mean, milimetric [19:06:17] yep [19:06:21] he was answering for me i think [19:06:25] heh [19:06:26] milimetric: did you paste your public ssh key into labsconsole [19:06:28] ? [19:06:29] (i'm in between bites) [19:06:43] ryan_lane said he had added you to bastion as well [19:06:44] drdee: no [19:06:47] ohhh ok [19:06:54] login to labsconsole [19:07:06] left side 'manage your ssh keys' [19:07:16] upload your public ssh key [19:07:31] then try again [19:08:06] drdee: thx, trying [19:08:32] yo welcome [19:14:16] drdee: it gives me the same error [19:14:38] mmmmm [19:14:54] you are using the right username, right? [19:16:04] I tried ssh -A bastion.wmflabs.org and ssh -A Milimetric@bastion.wmflabs.org where Milimetric is my username on labconsole.wikimedia.org [19:16:28] but my key has dan@DAndreescu-ThinkPad-T420s [19:16:34] so I'm not sure? [19:17:34] try ssh -A -vvv [19:17:41] and paste output in pastebin.org [19:18:44] what's -vvv? [19:18:53] very very verbose mode [19:18:55] extensive debugging ouput [19:19:03] oh with the address :) [19:19:09] yep [19:19:25] remember, milimetric is a Total command line noob, I assume Nothing :) [19:19:43] :D [19:19:50] you should join #wikimedia-labs [19:20:23]

09/24/2012 - 19:10:53 - Creating a home directory for milimetric at /export/keys/milimetric [19:20:35]

09/24/2012 - 19:15:54 - Updating keys for milimetric at /export/keys/milimetric [19:20:42] try milimetric (all lower case) [19:21:07] as username [19:21:19] http://pastebin.com/k3pbwr0e [19:21:56] that worked [19:21:59] lowercase [19:22:02] confused - why? [19:22:22] because your labs shell username != labs console username [19:22:33] a username on a wiki always starts with a Capital [19:22:45] oh ok, so they just auto-caps it on the wiki [19:22:54] yes [19:22:55] cool, thx [19:23:11] now try ssh reportcard2 (from bastion) [19:23:31] the -A stands for autoforwarding public key [19:24:09] if that works then read https://labsconsole.wikimedia.org/wiki/Help:Access#Using_ProxyCommand_ssh_option [19:24:10] a [19:24:11] n [19:24:12] d [19:24:14] also, dan [19:24:19] and setup proxy stuff [19:24:22] this is my .ssh/config [19:24:23] https://gist.github.com/4e35873bd8734ce4d07c [19:24:38] you might only need the pit at the bottom [19:24:56] hm, no, ssh reportcard2 fails on the public key (tried it with all the combinations I could think of) [19:25:17] your key is at $HOME/.ssh/id_rsa, right? [19:25:31] this is a minor issue [19:25:35] he already got into bastion [19:26:22] ottomata: thanks, what's in id_rsa-wmf? is that your pub key? [19:26:36] dschoon yes [19:26:49] that is the private key [19:26:49] dschoon, well I don't know on bastion [19:26:52] yeah, he has multiple keys [19:27:00] pub key ends with .pub [19:27:05] no, you shouldn't have a key on bastion [19:27:15] yes, I have id_rsa.pub and id_rsa both in my local computer's ~/.ssh/ [19:27:18] that's what -A is for -- it forwards your local key [19:27:23] right. [19:27:45] it's asymmetric -- .pub is the public key, the other is the private [19:27:52] is it really useful that all 3 of us trying to help milimetric? [19:28:00] probably not :) [19:28:06] i'm just answering questions :) [19:28:19] to free up you smarty-pants to *actually* help [19:29:06] (because i'm mostly waiting for a labs instance to build) [19:29:19] so... I don't know what to do next [19:30:49] so [19:30:52] if you are forwarding [19:30:54] on bastion [19:30:55] run [19:30:57] ssh-add --list [19:31:03] it will tell you if your key actually forwarded properly [19:31:21] AHhhh ops meeting over [19:31:22] that was long [19:31:24] I GOTTA PEEEEEEE [19:32:48] ok, I'm confused, after I added that to my ~/.ssh/config I couldn't connect to bastion any more so I deleted that fine and I'm ok just doing ssh -A milimetric@bastion.wmflabs.org [19:33:33] i guess i have no idea what i'm *trying* to do so I can't creatively think here [19:33:47] haha, ok, i took a while to figure this out recently, lemme see [19:36:06] k, lemme know what info you need [19:36:22] does your ssh key have a passphrase? [19:36:56] no [19:37:08] well, it's blank I guess [19:37:14] that's fine [19:37:18] (that's a no) [19:37:34] so you can log into bastion [19:37:51] i can ssh -A milimetric@bastion.wmflabs.org [19:38:07] ok [19:38:09] when you do that [19:38:09] once I'm there, I'm not sure if I have any keys set up or anything [19:38:11] what is the output of [19:38:13] ssh-add -l [19:38:19] anything? [19:38:25] The agent has no identities [19:38:49] ok, i'm trying to remember what I did differently, I think I have something set up for my mac, [19:38:52] ok back on your localhost [19:38:53] do [19:39:00] ssh-add .ssh/id_rsa [19:39:04] or whatever path you have [19:39:46] k [19:39:52] it said identity added [19:39:55] but that's my private key [19:40:03] that's fine [19:40:09] they're a pair -- they go together [19:40:10] then ssh into bastion again [19:40:18] same way as before [19:40:20] does [19:40:23] yep [19:40:24] ssh-add -l [19:40:25] now show anythign? [19:40:30] now yes [19:40:36] try reportcard2 now [19:40:47] k, that works [19:40:59] ok, lets try one more thing [19:41:08] i'm not exactly how to get it set up so you don't have to do the ssh-add part [19:41:09] lets see [19:41:18] open a new terminal in your local [19:41:30] is the key saved through sessions? I think it is: [19:41:31] it should run automatically on your local machine [19:41:33] ssh-add -l [19:41:45] right, but it didn't add his key when he just ssh-ed [19:41:53] maybe ssh-agent has to be manually started? [19:42:12] you mean that ssh-add I just did on my local won't stick around? [19:42:12] oh, this is ubuntu. [19:42:14] i have no idea. [19:42:46] if i exit out of reportcard2 and bastion I can go back in, I just checked [19:42:59] right [19:43:05] it will def stick around for your session [19:43:09] open up a new terminal window [19:43:10] on your local [19:43:13] run [19:43:15] ssh-add -l [19:43:15] in it [19:43:18] see if the key is still there [19:43:46] Ah, i found the document I read to understand all of this: [19:43:47] http://www.unixwiz.net/techtips/ssh-agent-forwarding.html [19:43:55] ssh-add -l on my local shows me that same key I see on bastion [19:44:14] in a new window? [19:44:16] yep [19:44:19] the issue might be that milimetric has a number of private keys, i never had to use ssh-add-l myself [19:44:20] then cool, it'll stick for sure [19:44:22] (i always do as I'm told) [19:44:24] maybe after reboot you have to run that again [19:44:38] ok, so ssh-add my private key if I can't connect [19:44:49] question: how does ssh-add on my local impact my session on bastion? [19:44:51] pretty sure his key is at the default location [19:44:59] i only ever made one key [19:45:00] good enough, there should be a way to do this so you don't have to manually run ssh-add [19:45:02] not sure what it is for you [19:45:08] ok, cool [19:45:20] http://www.unixwiz.net/techtips/ssh-agent-forwarding.html#fwd [19:45:20] no prob, i'll try to figure it out if it bothers me [19:45:24] milimetric: if you try to `ssh -A milimetric@reportcard2.pmtpa.wmflabs` -- does that work? [19:45:36] dschoon: from my local? [19:45:51] yes [19:45:58] check his debug output: [19:46:00] debug1: identity file /home/dan/.ssh/id_rsa-cert type -1 [19:46:01] debug1: identity file /home/dan/.ssh/id_dsa type -1 [19:46:02] debug1: identity file /home/dan/.ssh/id_dsa-cert type -1 [19:46:03] debug1: identity file /home/dan/.ssh/id_ecdsa type -1 [19:46:04]