[00:16:42] mutante, not that I can see, but do you want me to go poke him in meatspace and get him online? [00:17:35] Ironholds: well, he should be able to SSH to stat1002, and i see how he is succesful logging in on bast1001, but never on the stats host [00:17:49] so what is needed is ProxyCommand setup [00:17:50] that's..strange. [00:17:53] or agent forwarding [00:17:53] * Ironholds looks at his setup [00:18:43] i'm almost certain it must be client issue [00:18:49] because the key exists on both [00:18:57] and i see in logs how he can login on bastion [00:19:20] my guess is he has a ProxyCommand line in his .ssh/config [00:19:29] where the user name is not set [00:19:39] and then it tries to login as "work" or something [00:19:41] nope, he's setting user [00:19:48] looks like he forgot to forward that email to you, though [00:19:51] hangon, I'll fwd it [00:20:14] i see it now [00:20:18] it's an RT.. hold on [00:20:37] kk [00:20:46] ProxyCommand ssh -a -W %h:%p bast1001.wikimedia.org [00:20:52] it looks like he's providing ProxyCommand settings before user, which isn't how I'd do it, but I don't know if it is itself a problem [00:20:57] User sahar [00:21:02] let him try: [00:21:29] Host stat1002 [00:21:32] ProxyCommand ssh -a -W %h:%p sahar@bast1001.wikimedia.org [00:21:34] ssh stat1002 [00:21:47] (he used stat2 there, which i think is wrong) [00:22:07] nope, it's fine [00:22:08] only further down it says stat1002, after the relevant ProxyCommand [00:22:11] argg.. [00:22:19] yeah, the host definition is just the alias your machine hunts for in the config file. [00:22:27] I've been using stat2 in mine and can confirm it works a charm [00:22:47] HostName has to be [real name], Host can be [alias to real name]. Although really it's an alias to that proxycommand setup. [00:22:47] yet, i see you in lastlog [00:22:52] but i don't see him [00:23:07] and the keys he has on both machines are the same? this is odd. [00:23:27] yes, setup by puppet and same [00:23:30] I can't see any reason why his shouldn't work, unless it's something stupid like the order of declarations. [00:23:38] hmn; let me wander over to his desk and try something. [00:24:00] thanks! [00:24:18] root@bast1001:/home/sahar/.ssh# last sahar [00:24:19] sahar pts/2 wiki.static.monk Wed Feb 19 00:16 - 00:16 (00:00) [00:24:38] root@stat1002:~# last sahar [00:24:42] [00:24:55] also not in auth.log [00:25:27] mutante, so, he's being rejected by bastion, looks like [00:25:31] which explains why he doesn't hit stat2 [00:25:36] speak of the devil! :p [00:25:45] hello [00:25:59] sayhar: hi [00:26:13] Feb 19 00:16:39 bast1001 sshd[8069]: pam_unix(sshd:session): session opened for user sahar by (uid=0) [00:26:16] Ironholds: [00:26:26] sessions are being opened [00:26:44] Feb 19 00:24:46 bast1001 sshd[8398]: Invalid user work from 192.195.83.38 [00:26:50] but we _also_ have the invalid work user [00:27:18] hhuh [00:30:04] wait still? [00:30:08] that's so strange [00:30:22] what client are you using? [00:30:32] can you think of anything that's different from Ironholds setup? [00:30:46] he's on a mac, I'm on a xubuntu machine [00:30:50] that's..about it, to my knowledge. [00:31:07] wait, possible stupid thing... [00:31:19] The rest of my config is for other wmf machines, one uni machine [00:31:21] (stupid as in "if I'm right I totally should've spotted this earlier..") [00:31:26] and it says this up top: VerifyHostKeyDNS ask [00:31:27] IdentitiesOnly yes [00:31:36] wait, no, nevermind. Ignore me. [00:31:58] can you login on bast1001 directly [00:32:05] without trying to proxy [00:32:22] and then just type "ssh stat1002" there [00:32:33] and let me see if at least i see that in logs [00:32:55] or does a direct login on bast1001 fail? [00:33:57] Ironholds: is your local user equal to the remote user but his is not? [00:34:32] sayhar: try adding sayhar@ to the ProxyCommand [00:34:47] I was thinking of that [00:34:50] okay going to try that [00:34:51] i mean [00:34:53] ProxyCommand ssh -a -W %h:%p sahar@bast1001.wikimedia.org [00:34:55] that [00:34:59] the Username line after that [00:35:10] will influence the user you use on the bastion [00:35:22] typing in ssh stat1002 on bastion gives me this error: [00:35:26] Permission denied (publickey). [00:35:33] editing the proxy command and trying that now... [00:36:07] well, now [00:36:09] Feb 19 00:35:49 bast1001 sshd[8742]: Failed publickey for sahar [00:36:16] editing proxy gives me this: ssh_exchange_identification: Connection closed by remote host [00:36:28] but before i saw you open sessions [00:36:47] on bast1001 that is [00:37:06] i still don't see anything failing on stat1002 [00:37:09] mutante, yeah [00:37:14] just ezachte logging in .. [00:37:16] etc [00:37:16] well, mine is, at least, I can't confirm his. [00:37:56] what now? [00:37:59] does it work with agent forwarding? [00:38:12] ssh -A sahar@bast1001.wikimedia.org [00:38:16] ssh stat1002 [00:39:44] hmm [00:40:05] YES [00:40:07] ok, session opened on bast1001 [00:40:07] Yes it does [00:40:09] works! [00:40:10] i see it [00:40:14] so... [00:40:36] something about the ProxyCommand line and/or Mac OSX [00:42:16] is IdentityFile ~/.ssh/wiki_rsa really correct? [00:43:08] ehm, yea, i'm not a Mac guy, but it works for Ironholds and it works for me on other hosts ...hrmm [00:43:33] so i guess something is different about the ssh version? [00:47:51] hmm [00:48:16] Here's my ssh version: OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011 [00:49:15] maybe I can see if I can use home-brew to get a better/newer ssh version? [00:50:13] sayhar: hmm, if it's the Mac default then we would have noticed this way earlier, there are enough others using that [00:50:32] okay [00:50:34] it is the default [00:52:00] should ForwardAgent be "yes" instead of 'no'? [00:52:15] well, there is the old version [00:52:21] where people had to do [00:52:38] ProxyCommand ssh username@pryor.lshtm.ac.uk exec netcat -w 5 %h %p [00:52:58] but that's not going to be it either.. hrmmm hrmm [00:53:12] sayhar: no, it should be 'no' [00:53:37] well, you can try [00:54:13] ok [00:54:18] i don't mention it in my config [00:54:39] but the point is to not forward them [00:54:45] that's why ProxyCommand [00:54:54] instead of just using what works for you now [00:55:27] right [00:55:28] um [00:55:32] maybe let's go line by line? [00:55:47] Host stat2 [00:56:11] User sahar [00:56:12] ProxyCommand ssh -a -W %h:%p bast1001.wikimedia.org [00:56:13] ForwardAgent no [00:56:14] IdentityFile ~/.ssh/wiki_rsa [00:56:15] HostName stat1002.eqiad.wmnet [00:56:25] Ironholds said Host stat2 is ok, because it's just an alias [00:56:34] but to simplify things, we can [00:56:38] Host stat1002 [00:56:43] ProxyCommand ... [00:57:50] let's make it as simple as possible and try that version first, ok? [00:57:55] oh yeah [00:57:59] sorry I thought you were going to keep going [00:58:01] that doesn't work either [00:58:07] like this: [00:58:14] Host stat1002 [00:58:31] ProxyCommand ssh -W %h:%p sahar@bast1001.wikimedia.org [00:58:35] User sahar [00:58:42] (and nothing else in the file) [00:58:54] what does that look like then [00:59:08] if you type "ssh stat1002" [00:59:13] on your local computer [00:59:15] that works [00:59:19] heh [00:59:21] success [00:59:23] :) [00:59:31] that's kind of from my own config [00:59:34] just with another bastion [01:00:01] I don't understand why it doesn't need the identityfile [01:00:04] Ironholds: ^ heh [01:00:38] sayhar: what do you get when you type "ssh-add -l" [01:01:09] 1024 ... /Users/work/.ssh/id_dsa (DSA) [01:01:09] 2048 ... /Users/work/.ssh/wiki_rsa (RSA) [01:01:13] sayhar: so .. you do want to load the key from the agent, to connect to the bastion [01:01:34] the part you don't want to do .. is forwarding that to the second host [01:01:57] if it's loaded in the agent, you dont need to specify identity file [01:02:31] or , you can try instead: [01:02:49] ssh -i /path/to/identifile sahar@host [01:03:12] well, it works now,, right, resolved?:) [01:03:40] mutante: you lost me [01:04:14] Is this something about not wanting to sent my private key over to Bastion? [01:04:17] sayhar: well, if we agree it works now, it's ok [01:05:00] mutante: but I ran ssh-add -D [01:05:04] and now it doesn't work. [01:05:16] that's normal [01:05:26] you have 2 options: [01:05:46] - load it into the agent with ssh-add and use ProxyCommand (just not -A) [01:06:00] - do not use any agent but specify the key each time with -i [01:06:03] but the first should be ok [01:06:28] so if you do what just worked for you, we should be done [01:06:30] so it'll be a 2-step process every time, instead of ssh stat2 [01:06:36] no [01:06:43] it will just be [01:06:48] "ssh stat1002" [01:06:56] oh, you mean loading the key into the agent [01:07:05] yea, well _once_ per your computer booting [01:07:34] as opposed to typing the key passphrase on each connect to stat1002 [01:08:22] okay [01:08:36] not ideal of course but at least it's possible. [01:08:37] i am still not entirely sure which line broke it.. to be fair, the alias or the identify file or the others [01:08:48] but i can confirm it works with the minimal config [01:08:59] sayhar: what is the difference to before? [01:09:12] that it's "ssh stat1002" vs. "ssh stat2"? [01:09:25] ssh-add ~/.ssh/wiki_rsa each boot [01:09:28] i may also be confused now to how you have it on other hosts [01:09:42] how do you do that for other hosts [01:09:47] you have to load it once somehow [01:10:03] or you type the passphrase each time? [01:10:10] ..or ..no passphrase on key ? [01:10:43] though maybe I could put in a .bash_profile or something be done with it [01:10:45] i'd recommend having a passphrase, but then it'll be annoying to type it each time, hence the ssh-add solution [01:10:56] that's like what it was made for [01:11:07] convenience / security balance [01:11:19] it's a mac thing, I think [01:11:35] imho your current solution can only work because you have no key passphrase [01:11:42] which means if i get to copy it.. [01:11:45] i have access [01:11:50] normally I'd attempt to connect to something without worrying about ssh-agent, and there'd be a shiny macosx prompt that works as a GUI to ssh-agent [01:11:54] with just a "thing" [01:12:00] as opposed to a thing and a password [01:12:01] ah brb [01:12:07] phone call [01:12:13] so, it's like 1-factor vs. 20factor wuth [01:12:15] ok [01:12:43] well, i also kind of have to run, if we can continue it later because you have access now [01:12:50] yes [01:12:54] great [01:13:17] ok, cool, then ..i'll update ticket and ttyl [01:44:00] +1 [02:10:53] (CR) Edenhill: [C: -1] "Since openlog() is called with LOG_PERROR (print to stderr) you can remove the old fprintf(stderr, ..) calls." [analytics/kafkatee] - https://gerrit.wikimedia.org/r/114078 (owner: Ottomata) [09:26:37] (CR) Hashar: "What is your flake8 version? It works for me when invoking flake8 directly at the root of the repository or when using "tox -e flake8"." [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/114086 (owner: Milimetric) [12:30:43] (PS10) Nuria: [WIP] Changes tu support wikimetrics in vagrant. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/109676 [12:30:57] (CR) jenkins-bot: [V: -1] [WIP] Changes tu support wikimetrics in vagrant. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/109676 (owner: Nuria) [14:59:24] average, csalvia_, nuria, ottomata, tnegrin: fair warning, we're starting at exactly 10:00 [14:59:30] which is in 30 seconds or so [14:59:44] due to the notes from yesterday's retro [15:00:09] AHHHGHHGHHHH [16:29:06] (PS11) Milimetric: [WIP] Changes tu support wikimetrics in vagrant. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/109676 (owner: Nuria) [17:46:10] (PS12) Milimetric: [WIP] Changes tu support wikimetrics in vagrant. [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/109676 (owner: Nuria) [18:21:00] (CR) Milimetric: "my only other comment besides the patches I added" (1 comment) [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/109676 (owner: Nuria) [18:51:09] (CR) Milimetric: "Flake 8 version was:" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/114086 (owner: Milimetric) [18:51:17] (CR) Milimetric: [C: 2 V: 2] Fix tox ignoring flake8 config [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/114086 (owner: Milimetric) [19:34:19] Ironholds, et al, Is anyone else coming to the Research & Data showcase? [19:34:47] halfak, ^ [19:35:21] Bah! there isn't one today. [19:35:28] We moved it to next week. [19:35:37] The mpeople in the meeting room seem to disagree! [19:35:48] * halfak shakes his fist at DarTar [19:36:09] ok, they agreed and dropped the call. [19:36:31] never mind! back to real work for me. >.> [19:37:01] Sorry for the trouble. The event was moved, but apparently it didn't affect everyone's calendars. :( [19:37:32] yeah. I still see it in my calendar. [19:37:39] There were 3 friendly people in the meeting room, and 3 of us remote. I didn't hear names properly, so I canna tell ya who it was! [19:39:08] Nay worries. This is the least of my gcalendar confusions this week! [19:39:47] I hope we'll see you next week. :) Do you see the new event on your calendar? [19:40:18] I see it. [19:44:58] yup [19:47:45] OK. At least we have that. :) [20:38:53] (CR) Hashar: "Dan : yeah that got changed in pep8 1.4.6" [analytics/wikimetrics] - https://gerrit.wikimedia.org/r/114086 (owner: Milimetric) [21:32:59] milimetric: thanks! [21:33:55] np AndyRussG, looking forward to helping out with anything I can [21:34:10] note: the mediawiki-vagrant work is not quite 100%, though it's very close [21:34:13] cool, likewise! [21:34:28] if you'd like, I can ping you and adam once it's done [21:39:45] milimetric: ah thanks... So far I haven't used Vagrant, though maybe now'll be a good time to start :) [21:40:05] yeah, ideally for wikimetrics dev, it'd be something like this: [21:40:07] (once we're done) [21:40:18] git pull http://...vagrant.git [21:40:23] cd vagrant [21:40:24] vagrant up [21:40:34] vagrant enable-role wikimetrics [21:40:48] and that's it, you'd have a checkout you can dev on and the site would be running on your localhost [21:45:34] + vagrant provision [21:46:48] right, thanks otto :)