[00:00:35] restarting puppetmaster gets me 'Could not prepare for execution: Got 1 failure(s) while initializing: change from absent to directory failed: Could not set 'directory on ensure: File exists - /etc/puppet/manifests' [00:00:45] Which, maybe I should just rm that directory [00:02:30] oh [00:02:35] seems the repo didn't check out [00:03:09] I'm cloning it into the directory [00:03:41] the clone must have failed for some reason [00:04:24] ok. it's working now [00:05:17] yep, looks better. [00:05:20] thanks! [00:05:23] yw [00:31:17] New patchset: Ryan Lane; "Fixing variable name for script user" [labs/private] (master) - https://gerrit.wikimedia.org/r/19973 [00:34:10] Change merged: Ryan Lane; [labs/private] (master) - https://gerrit.wikimedia.org/r/19973 [00:34:49] ^ now if only we could actually see those changes [00:38:15] meh [00:38:21] I'll fix that at some point [00:38:40] the private repo isn't amazingly necessary [01:38:24] Ryan_Lane: what does cas mean? [01:38:35] eh? what do you mean? [01:39:06] idk, what do you mean? [01:39:06] 16 01:32:37 <+gerrit-wm> New patchset: Ryan Lane; "labs puppetmasters should be cas..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/19979 [01:39:13] CAs [01:39:18] read the diff [01:39:22] oh... [01:39:28] i guess i could do that ;) [01:39:34] * jeremyb was really stumped [01:40:17] i see in the diff now ;) [01:44:28] \o/ [01:44:34] well, instance creation is fixed [01:48:08] Ryan_Lane: so $ldapconfig relies on global scope? [01:50:06] * jeremyb rereads again in a different editor (with different hilighting!) [01:53:19] oh, i see where they're used [01:53:52] Ryan_Lane: why does production use the labs passwd? [01:54:01] or am i missing something? [01:54:10] it's all production [01:55:10] so there's ...::config::labs and ...::config::production [01:56:07] seems like a lot of repitition [01:56:07] yes [01:56:10] and yes [01:56:14] which is why I have a todo to removeit [01:56:16] *it [01:57:38] so, current master starting at line 18 all passwords are under labs. [01:57:55] from 47 they are production except 1 [01:58:00] that's correct? [01:58:11] as I said, it's all production [01:58:19] production and labs are the same directory [01:58:32] ok... [02:04:24] Ryan_Lane: well i still don't understand how it works or what it does but i'll live with it. anyway, I28f4806cdd7f51 is for you ;) [02:04:38] can you link to the change? [02:04:42] !g I28f4806cdd7f51 [02:04:42] https://gerrit.wikimedia.org/r/#q,I28f4806cdd7f51,n,z [02:05:56] ty [02:06:39] ty yourself ;) [04:29:36] any known problems with ssh keys? i periodically have to restart machines or wait for puppet runs to regain access to instances. my configuration and keys aren't variable, but my ability to access instances is. are there any known bugs that could be causing this? [04:31:57] i forced the instance to restart and am now able to ssh in again. ugh. no idea what would cause this. [09:13:07] hello :) [09:14:36] hi hashar [15:27:01] paravoid: When a command launched by puppet generates stdout or stderr, where does it go? [15:27:06] Is there some way I can capture it? [15:27:34] it gets in the log [15:27:38] what do you mean by capture? [15:27:42] act on it? [15:27:57] Nope, I just want to see it. [15:28:39] I guess I was thinking that when I run puppet from the commandline it did something different with logging. Maybe that's silly. [15:29:05] oh right, you want logoutput => true [15:29:21] That's a property of the exec? [15:29:26] ori-l: that sounds weird... [15:29:30] In the manifest? Or a cmdline switch? [15:29:38] ori-l: did you see my question about updating phab? [15:30:19] exec { "foo": command => "...", logoutput => true } [15:30:53] ok. And, where does that log go? I'm looking in /var/log/puppet but it is weird… it has a rails.log and a masterhttp.log, neither of which seem to be what I want. [15:31:15] /var/log/daemon.log on our systems [15:31:22] or stdout when you do puppetd -vt [15:31:37] s/puppetd/puppet agent/ ;-) [15:31:38] Hm, ok. [15:32:04] you will have to change eventually ;) [15:32:33] it's missing from the 3.0 rc in experimental (at least according to the changelog) [15:32:46] we don't build our own puppet, just use the one from ubuntu? [15:34:07] jeremyb: we "build our own" (backport) for lucid, use the stock one for precise [15:34:30] hrmmm, k [15:34:53] lucid has some ancient version, precise has 2.7.something so no need to change it yet [15:35:13] but I'd say that if we really want features from a newer version, not being present in Ubuntu wouldn't stop us [15:36:06] right, just wondering [15:36:28] does ubuntu use urgency at all? [15:36:37] just looking at http://changelogs.ubuntu.com/changelogs/pool/main/p/puppet/puppet_2.7.11-1ubuntu2.1/changelog [15:37:16] the ones that were not =low seemed to be from debian. but there were more recent security releases [15:39:56] * jeremyb runs away [15:58:12] paravoid: OK, I'm stumped. Every time I run this command it succeeds, but every time puppet runs it it fails. "git clone -b master https://gerrit.wikimedia.org/r/p/mediawiki/core.git /srv/mediawiki" [15:58:28] fails how? [15:58:40] timeout exceeded perhaps? [15:58:44] "RPC failed; result=22, HTTP code = 502" [15:58:49] It's not a timeout failure, I've already addressed that. [15:59:17] That error turns up in google a fair bit, but it's generally described as being intermittent. [15:59:28] where's that? may I see it? [15:59:49] You mean the context for the command? Or forum posts about the error? [15:59:58] the first [16:00:03] which machine is it so I can login [16:01:20] paravoid: http://pastebin.com/8dxp6RXe [16:01:29] I'm running it on labs instance i-000003be [16:01:39] which is using self-hosted puppet [16:05:39] paravoid: If you want to re-run puppet, I'll have to clean up some files to force it to check out again. [16:05:51] (Having just run the command by hand and, ergo, successfully.) [16:47:10] paravoid: Looking back at my link, I get 'unknown paste ID'. Did that happen to you too, or did it work and then expire? [16:49:45] anyway, take two: http://pastebin.com/qHKCZ1PL [17:12:52] Hello guys, could anyone tell me how does one create a project on labs ? [17:13:33] chughakshay16__: You have to request one [17:13:36] I'm not sure where [17:13:38] @search project [17:13:38] Results (Found 9): sudo, deployment-prep, project-access, instanceproject, puppet-variables, pl, projects, blehlogging, wikiversity-sandbox, [17:13:45] !projects [17:13:45] https://labsconsole.wikimedia.org/w/index.php?title=Special:Ask&q=[[Resource+Type%3A%3Aproject]]&p=format%3Dbroadtable%2Fheaders%3Dshow%2Flink%3Dall%2Fsearchlabel%3D%E2%80%A6-20further-20results%2Fclass%3Dsortable-20wikitable-20smwtable&po=%3FMember%0A%3FDescription%0A&limit=500&eq=no [17:13:53] Ah that's just the list of projects :S [17:13:59] !instanceprojec [17:13:59] $instanceproject is a variable used in puppet to determine which project an instance is in. [17:14:00] !instanceproject [17:14:00] $instanceproject is a variable used in puppet to determine which project an instance is in. [17:14:02] RoanKattouw: :) [17:14:05] *sigh* [17:14:17] chughakshay16_: I can make you a project; what's it for? [17:14:44] andrewbogott: for testing my gsoc work .. [17:15:09] chughakshay16__: Which involves mediawiki development, I take it? Or is otherwise wikimedia-related? [17:15:19] Sorry if I should already know all this :) [17:15:19] andrewbogott: Maybe the project creation process could be documented better? I can't find anything on labsconsole or in the bot brain about this [17:15:49] RoanKattouw: It could be, yes. I think part of the idea is that we are a 'closed beta' right now, and will have a more direct self-serve process sometime. [17:15:54] andrewbogott: its an extension which runs atop of mediawiki [17:16:09] andrewbogott: https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/ConventionExtension [17:16:16] chughakshay16__: OK. What would you like the project to be called? And, any other people you want included as members? [17:16:50] andrewbogott: yes i would like to have my mentor as a member as well.. freakolowsky [17:17:49] andrewbogott: and i would like to call it conventionextension.. [17:17:59] andrewbogott: if thats not an issue :) [17:20:19] chughakshay16__: Sorry, multitasking. One moment... [17:20:26] chughakshay16__: You have a labs account already, I take it? [17:20:32] andrewbogott: yeah no problem. [17:20:39] andrewbogott: yes.. [17:22:05] chughakshay16__, I don't see you in the Bastion project. Do you use a different username? Or maybe you have gerrit access but not labs access yet? [17:22:53] andrewbogott: my username on labs is Chughakshay16 [17:24:27] andrewbogott: does creating an account on labs give you access ? If yes then I do otherwise I dont.. :) [17:24:49] you didn't have access but you will in a moment. [17:25:46] andrewbogott: ok [17:26:43] OK, your project should be ready. You can go ahead and create an instance now... [17:26:53] bastion access might take a few minutes to percolate. [17:27:01] For your test setup you will probably want to use this: https://labsconsole.wikimedia.org/wiki/Help:InstanceConfigMediawiki [17:27:27] BUT, that is currently somewhat broken. I'm working on it, but i the meantime I can talk you through the breakage. [17:28:30] chughakshay16__ ^^ [17:30:56] andrewbogott: got dc [17:31:31] I'll repaste privately... [17:42:14] paravoid: hey, do you mind checking out instance-00000271 on virt7? [17:42:16] it won't boot [17:42:31] not sure if that went through before I was booted [17:44:00] I will. can you check labstore2 -> virt0 ldap? [17:44:11] it's broken and sends cronspam. there's nothing in iptables to allow that [17:44:28] ah. it was likely hitting nfs1 before [17:44:29] and used to work afai, so I'm figuring it's something that recently changed? [17:44:57] basically everything is hitting virt0 rather than nfs1 now [17:45:02] since the ldap changes in puppet [17:45:38] I should just open ldap access to everything in our network [17:48:05] that was changed with what? the mega-commit? [17:48:49] yes because I didn't make a "production" client [17:48:58] :/ [17:49:11] "production" == "labs" anyway, from the ldap pov [17:49:18] it's the same directory [17:49:24] with the risk of being tiring, this is why I don't like mega-commits... [17:49:41] this has *nothing* to do with the size of the commit [17:49:54] if the commit was small, what would you do? revert it? [17:50:12] chughakshay16: The docs you're following are very new; I'm always interested in suggestions as to how to clarify things. [17:50:15] it's not about reverting, it's about reviewing, doing incremental steps and understanding what they do [17:50:26] you reorganized *and* changed the behavior in the same commit [17:50:34] and? [17:50:40] so it was basically impossible to see the behavioral change without reading through all the reorganization [17:51:03] I think you misunderstand why the change was so large [17:51:14] it's not because I like huge commits [17:51:15] it's different to see a diff of three lines changing than three lines changing among all of its surroundings moving into a different file [17:51:29] it's because it was impossible to do otherwise without breaking live systems [17:51:34] andrewbogott: well there isnt anything written about creating a project.. so that is something which could be improved [17:51:40] * andrewbogott nods [17:51:47] I hate going over this again, but I kinda guessed this was done my the mega-commit, tried to read through it again to find the bug [17:51:52] then failed again [17:52:23] you can complain about the mega-patch as much as you'd like [17:52:32] it doesn't change the fact that it was impossible to do otherwise [17:52:39] Really we just need a lab or labs to do stuff like the manager updates knowing it's not going to break everything and actually work when openstack is updated. [17:52:46] this is the problem with spaghetti code [17:52:59] Damianz: we do that in labs right now [17:53:22] the only things that broke when I merged this change are things that are difficult or impossible to test in labs [17:53:43] andrewbogott: some snapshots would definitely help for a newcomer [17:53:56] screenshots of the console, you mean? [17:53:58] Nothing should be impossible to test if it's in production, but yeah some things are gonna suck. [17:54:28] Damianz: well, for instance, this issue with labstore2 not being able to talk to the ldap server [17:54:32] it's specific to production [17:54:36] and firewalls [17:55:17] the other thing that I broke temporarily (which is why I reverted the large patch twice) was the puppetmaster on virt0 [17:55:22] err [17:55:36] no, the puppet config on labs instances [17:55:43] still not totally sure why that broke [17:55:46] andrewbogott: no. screenshots of the process that one needs to go through while creating an instance [17:55:48] I had to put a hack in [17:56:05] While I know it's never possible to fully test everything in a simulation before hitting prod, my idealisim side says let's replication prod entirly to test for labs which also replications prod entirly in a more stable fashion [17:56:06] the puppet config on virt0 I didn't revert for [17:56:09] * Damianz hides [17:56:36] chughakshay16: Ah, sorry, when I say 'the console' I mean 'labsconsole' which is the web interface. [17:56:47] Damianz: testing for ip/firewall issues means you need to have the same IPs [17:56:51] andrewbogott: ok :) [17:56:52] that's not doable ;) [17:57:18] Hmm, questionable [17:57:31] Public ips no, public ips internally probably - but you'd need seperate routers [17:57:49] And total isolation while using the prod network is rather more near impossible than easy [17:59:16] andrewbogott: of all the group rules which one shall i choose ? [17:59:56] Um… none, I think. You should make a new 'web' rule. [18:00:11] andrewbogott: Yes I made a new one [18:01:00] chughakshay16: OK, that looks right. So just make sure you check that one (and 'default') for your new instance. [18:01:49] andrewbogott: I would have loved to improve the docs.. but there is not much I know about [18:03:10] paravoid: any thoughs/suggestions about my git failure? [18:03:19] which one? [18:03:24] didn't see that [18:04:06] Um… the thing we were discussing earlier, different behavior when puppet does the checkout vs. when I do it on the commandline. [18:04:42] oh right, sorry [18:04:57] np; it's unclear if it's a git problem or a puppet problem. [18:05:32] which machine is that? [18:05:52] i-000003be [18:06:41] paravoid: You'll need to rm -rf /srv/mediawiki to reproduce the failure. [18:08:30] andrewbogott: Installing puppet class 'role::mediawiki-install::labs' gives an error.. [18:09:28] chughakshay16: Yep! That's the problem I'm bugging paravoid about. (It's my fault, but I'm stumped as to how to fix it.) [18:09:33] Try running 'sudo git clone -b master https://gerrit.wikimedia.org/r/p/mediawiki/core.git /srv/mediawiki' [18:09:42] After that, I suspect that puppet will be happier. [18:10:08] andrewbogott: sec, it's swift deployment window time and I have to divert my focus a bit [18:10:16] ok [18:10:49] andrewbogott: where do you want me to run this ? [18:11:05] Shouldn't matter, all paths are absolute. [18:11:20] I mean, on your instance. But, cwd doesn't matter. [18:16:30] andrewbogott: I really dont get you.. [18:18:13] No one does :) [18:19:32] * Damianz goes and does something useful like making food [18:20:11] andrewbogott: My git bash doesnt recognize this command :) [18:43:46] andrewbogott: I was able to successfully install a puppet class to my instance.. [18:44:59] andrewbogott: now what exactly am I supposed to do.. [19:00:29] chughakshay16: sorry, I had to step away. [19:00:40] Can you tell me where you're at? [19:00:56] andrewbogott: you kind of left me hanging here.. :) [19:01:12] Are you able to ssh to your instance? [19:01:15] andrewbogott: I have created the instance and have installed puppet class to it.. [19:01:45] andrewbogott: could you give me the exact command that one needs to use to access it with ssh ? [19:02:10] That's a 'no' I assume? [19:02:45] The general access information is here: https://labsconsole.wikimedia.org/wiki/Help:Access [19:02:45] andrewbogott: yes [19:02:59] Getting your keys and such set up depend to some extent on what you're running on your local machine. [19:03:16] If linux, it should be straightforward. If windows, then… it's possible but I'm not the right one to get advice from. [19:03:30] andrewbogott: its windows :) [19:03:39] Apologies; earlier when you told me the result of your puppet run, I assumed that you were logged into your instance and running puppet there. [19:03:49] I guess you were getting that from the system log on the web interface? [19:04:20] andrewbogott: yes that error log was from the interface [19:04:26] ok [19:04:37] Let's start at the beginning... [19:05:05] Do you know how to use a linux shell? Do you have a terminal program on Windows that you're somewhat familiar with? [19:05:06] andrewbogott: ok, I need to figure this out tonight... otherwise i wont be able to sleep :) [19:05:17] andrewbogott: this thing has scared for sometime now.. but no more [19:05:29] andrewbogott: git bash [19:05:57] In my world, 'git' and 'bash' are two different things. Are you using a windows program that's called 'git bash'? [19:06:07] andrewbogott: yes [19:06:19] OK. I have no idea what that is or if it will help you. [19:06:31] I think people mostly use putty as their ssh client. [19:06:37] andrewbogott: I have been using it to push my code changes to gerrit.. [19:06:46] I'm sorta hoping that a windows user will appear and coach you on this. [19:06:52] andrewbogott: and to even clone mw repos [19:07:59] andrewbogott: just to be clear from the beginning , which repo is this instance using ? [19:09:04] https://gerrit.wikimedia.org/r/p/mediawiki/core.git [19:10:32] andrewbogott: so how does one put their extension in this repo ? [19:10:52] No idea. [19:11:12] andrewbogott: :) [19:11:19] Maybe we should go back to you telling me what you want a labs project for. [19:11:53] You want a server to run a mediawiki server on, right? [19:11:55] andrewbogott: I want to demo a use of the extension that I created... [19:12:06] andrewbogott: I'm looking at your issue, I think it just takes too long for git clone to run... [19:12:09] and the server timeouts [19:12:24] paravoid: OK, but why would it timeout when puppet does it but not when I do it from the commandline? [19:12:50] andrewbogott: yes i need a server that runs mediawiki , but in that mediawiki copy i need my extension too.. [19:12:55] the puppet server, not the git server [19:13:25] paravoid: I have the timeout set to 50 minutes. And, also, I've seen timeout failures and they say 'timeout' rather than that cryptic http thing... [19:13:36] Which isn't to say that you're wrong, just that this is weird. [19:14:00] note that we're running with ::self, which runs with webrick [19:14:06] not a real webserver [19:15:24] paravoid: I must not understand how this works :( Isn't puppet just running the 'git clone' command? Is there some middleware involved that doesn't enter into it when I run 'git clone' myself? [19:16:11] chughakshay16: So, labs can provide you with a server to host your instance. It can also preconfigure that with a ready-made mw install. [19:16:13] puppet agent runs the git clone command, while keeping an http/puppet session open with the puppet master [19:16:43] But after that you're on your own. I don't personally know even a single thing about how to install extensions in mw. Doesn't each extension live in its own repo? [19:17:24] andrewbogott: yes they do, but in order for them to work they need to be loaded along with a mediawiki copy [19:17:26] paravoid: Hm… ok. Do think that my attempt to manually set a timeout is affecting the agent but not the puppetmaster session? [19:17:51] yeah maybe... [19:17:57] that's just speculation at this point [19:18:01] chughakshay16: Ok. So once you have a mediawiki server running (of any sort) then you can add your extension to that server, can't you? [19:18:38] paravoid: Unfortunately, that puppet class needs to do things after the git checkout completes. So we can't really just spawn it and forget it :( [19:18:46] andrewbogott: yes we can, thats what i asked how to push my code changes to that mediawiki server copy ? [19:18:51] and we shouldn't do that anyway [19:19:24] chughakshay16: OK. Well, basically, step one is that you need to be able to do anything at all on the server. [19:19:45] Which will require access, for starters. [19:20:01] And also a modicum of linux and mediawiki-config skills. [19:20:11] andrewbogott: you mean by setting up a public ssh key [19:20:12] Labs people can help you with getting access (that's what ssh is for.) [19:20:31] But once you're connected, I can't really help you with what to do there… maybe your gsoc mentor can coach you on those parts. [19:21:21] chughakshay16: Yes, you will need a private/public keypair in order to get ssh access. Do you have that already? [19:21:48] andrewbogott: I have that for gerrit , but apparently I dont have one for labs. [19:22:12] You say 'apparently' because... [19:22:26] * andrewbogott can't remember if gerrit and labs keys are the same [19:22:28] andrewbogott: because I have pushed it just now :) [19:22:36] Ah, great. [19:23:22] andrewbogott: which timezone are you in ? [19:23:46] People mostly use Putty and Pageant to connect to labs machines. So, you should probably download and install them, and have a look at this guide: http://www.howtoforge.com/ssh_key_based_logins_putty [19:24:00] I'm in CDT, but I'm also going away for the weekend in ~an hour. [19:24:11] But, as I said, I'm not the right person to help you with windows access anyway :) [19:24:38] andrewbogott: even my git bash can add ssh-keys [19:25:23] andrewbogott: so i would say git bash is nothing but a sweet client for operating with git and has almost all those features which you would normally find in a shell console [19:27:16] So, can you ssh to another system with it? What happens when you type 'bastion.wmflabs.org'? [19:27:42] andrewbogott: yes i can connect to other machines via ssh.. [19:28:00] andrewbogott: thats how i clone mw repos using ssh proto [19:29:06] Can you log into bastion.wmflabs.org? [19:29:30] from my git bash ? [19:30:01] This is confusing, because ssh is two things. [19:30:46] Yes, from git bash, or from anything. Do you have a way to get shell access to bastion? [19:32:26] andrewbogott: i presume that git bash could do that [19:32:47] I just ran this command from my gitbash - ssh -A Chughakshay16@bastion.wmflabs.org [19:32:56] And? [19:32:57] it says permission denied [19:33:53] OK, that might be good! It's at least trying :) [19:35:20] You're following this? https://labsconsole.wikimedia.org/wiki/Help:Access#Using_agent_forwarding [19:35:45] andrewbogott: yes [19:35:58] chughakshay16: Hey, me too working on the same stuff ;) [19:36:22] nischayn22: Hi there, [19:36:36] nischayn22: any luck with this .. :) [19:37:02] you guys aware of /var/log/glusterfs/data-project.log ballooning to fill up entire disks? dunno if it's just my instance. not sure if this is sufficiently interesting to report. [19:37:11] chughakshay16: consider using VMware Git Bash is good normally but has limitations [19:37:23] i have 7gb of "[2012-08-15 07:44:55.582456] W [fuse-bridge.c:4006:fuse_thread_proc] 0-glusterfs-fuse: read from /dev/fuse returned -1 (Resource temporarily unavailable)" [19:37:38] andrewbogott: may be a silly question, but why cant i access my instance using a browser ? [19:38:05] chughakshay16: instances don't have publicly-accessible addresses by default. you need to request one and assign it to this instance [19:38:19] chughakshay16: but the good news is that it's fairly simply and easy to do, if your use-case requires it [19:38:50] ori-l: I have already set up a specific puppet class for that.. [19:39:20] chughakshay16: Do we know if your instance is even running a web server? [19:39:21] ori-l: 'role::mediawiki-install::labs' [19:40:14] chughakshay16: once you're able to log in, you can try hitting http://localhost/ using "curl" or "wget" on the command line. i suspect you were successful in getting mediawiki to install but it's simply not exposed to the outside world. [19:40:46] ori-l: no i wasnt successfull with anything, i m still stuck at step one :) [19:41:34] ori-l: I have created my instance , but i cant seem to understand how to access it ? [19:42:09] chughakshay16: try running the same command you ran a few minutes ago, but add "-vv" between 'ssh' and '-A' and paste the log on dpaste.com [19:42:22] ori-l: ok [19:42:46] ori-l: I can log into my instance, then when doing the foxy proxy thing localhost never loads [19:43:49] ori-l: http://dpaste.com/787429/ [19:44:08] nischayn22: i don't use a proxy for http -- my instance has a public-facing ip, so i haven't configured that setup myself, sorry. [19:45:35] chughakshay16: can you run "cat /c/Users/akshay/.ssh/id_rsa" and verify that it matches *exactly* the key you declared in labsconsole? you don't need to share it here, just confirm. [19:45:54] ori-l: even my instance seems to have a public ip (I didn't request for that though), but that doesn't load either [19:45:59] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Greensmw [19:46:23] nischayn22: false advertising :) that's not really a public ip [19:46:33] lol [19:47:22] ori-l: you know other methods to access it? [19:47:57] ori-l: labconsole wont show me the complete key [19:48:50] chughakshay16: okay, so it doesn't have the whole key. go ahead and erase the fragment that's in labsconsole and paste the key as outputted by that command [19:48:59] and save it [19:51:03] ori-l: but the command that you gave me prints out the private key and not the public key that i have shared on labs [19:52:38] * ori-l slaps forehead. [19:52:48] yes, you're right, i forgot the .pub. horrible mistake. [19:53:48] the basic principle applies: make sure the private key matches the public key, and the public key matches what you have on labsconsole, and make sure it is being offered by your ssh client, by inspecting the output of ssh -v and manipulating your config / command line options as necessary [19:54:46] ori-l: and yes both keys are same.. the one that i have shared on labs and the public key that i have on my system. [19:55:44] ori-l: its the same key that i have shared on gerrit as well [20:52:05] chrismcmahon: you should retire that key now and generate a new one (assuming you sent the private key to labsconsole0 [20:52:25] gah [20:52:29] ? [20:52:36] that was for chughakshay16 [20:52:38] sorry [20:52:43] ch [20:57:01] maplebed: see what i just did here? ;) [20:59:06] lol