[00:03:01] If the ldap setup a bit funky? I swear sudo wants my pass at least 3 times before it actually works. Unless I can only type my pass on the 4th attempt anyway.... [00:03:15] mistypes, likely :) [00:03:21] it works for me the first time every time [00:03:27] Hmmm [00:03:37] Maybe it just doesn't like me [00:04:31] Anyway, on a seperate note, for this gerrit ticket can I add the nagios::monitor:: stuff into a misc host pp file rather than nagios.pp? [00:04:54] I don't really think that using puppet for nagios would work [00:05:04] ? [00:05:15] New patchset: Ryan Lane; "Password login inside of labs is bad. Let's ensure this is only set for production." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1765 [00:05:19] because nagios is completely independent, I am working on a way to insert various services by users [00:05:29] New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1765 [00:05:30] Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1765 [00:05:39] hmmm [00:05:49] I don't really care, but puppet seems the most sensible way [00:05:51] it download a list of instances from nova and generate configs using c++ program [00:06:02] then it reload the service [00:06:09] Damianz: in production we do use puppet [00:06:21] unfortunatelly it doesn't work here [00:06:27] hmm [00:06:35] but, unfortunately, we haven't fully puppetized nagios [00:06:47] ah [00:06:51] also, we are likely to make some changes to how we use puppet in labs that will make it not work anyway [00:07:01] it can recognize which service you need from the configuration of instance so we could create a class with variables where you could specify what should be monitored [00:07:06] (for instance, getting rid of the central server) [00:08:12] it know what services it should watch from the classes you checked in nova [00:08:29] I was kinda hoping to create some dpkgs for cluebot's parts + have a puppet config for it so we can trash/rebuilt the server and just drop the mysql details in place. [00:08:46] Damianz: how is it related to nagios [00:08:50] Damianz: sounds good [00:09:15] Vivek: ok, in about 30 minutes to an hour you won't be able to ssh in with your password [00:09:24] so you need to get that ssh agent thing worked out [00:10:00] :o [00:10:03] petan: It should also be monitored! [00:10:05] Or be lazy and use ssh -i [00:10:08] Ryan_Lane: that's not a good change [00:10:17] yes it is :) [00:10:19] keys only [00:10:23] password for sudo [00:10:27] ok [00:10:27] I can only ssh with password from other than bastion [00:10:28] that was the original intent [00:10:43] that makes no sense [00:10:45] because I have private key only on bastion [00:10:56] I don't like idea that someone with sudo steal my private [00:10:59] add the public key to your keys [00:11:08] and use my account :o [00:11:20] you can have more than one public key [00:11:26] I know [00:11:29] no one can sudo on bastion [00:11:31] I made a private key only for labs [00:11:46] of course, but if I wanted ssh from bots-apache to bots-sql [00:11:49] Ryan_Lane: Yeah ryan, I'm not too crazy about having a private key on a machine I don't own [00:11:53] I would need to have private key on apache [00:11:53] then forward your key :) [00:12:04] Forwarding? [00:12:08] forward what is it [00:12:12] yes. ssh key forwarding [00:12:23] What was the issue with logging in via password once you're inside bastion? [00:12:25] I can ssh to anything from anywhere as soon as I login, thanks go my keychain getting unlocked then my client forwarding life. [00:12:29] it's insecure [00:12:53] forwarding is insecure? [00:12:56] no [00:12:59] password login is [00:13:04] Can we have krb tickets? :P [00:13:05] sort of [00:13:13] krb would be nice, ues [00:13:17] but copying private key everywhere is even worse [00:13:19] it's in the list of proposals [00:13:31] petan2: why would you copy your key? [00:13:33] ever> [00:13:41] it doesn't even need to be on bastion [00:13:42] where do I find how to forward key [00:14:02] !access [00:14:08] @search access [00:14:08] No results found! :| [00:14:11] heh [00:14:32] @search ssh [00:14:32] Results: bastion, socks-proxy, [00:14:35] @regsearch ..* [00:14:35] Results: puppet, instance, morebots, git, bang, nagios, bot, labs-home-wm, labs-nagios-wm, labs-morebots, gerrit-wm, wiki, labs, bastion, extension, wm-bot, projects, putty, gerrit, change, wikitech, revision, monitor, alert, password, unicorn, help, $realm, bz, os-change, instancelist, instance-json, leslie's-reset, damianz's-reset, amend, credentials, bug, queue, socks-proxy, sal, info, security, logging, ask, sudo, [00:14:41] I wish macs had a better keychain manager :( Really need to re-install this thing with ubuntu. [00:14:41] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [00:15:11] you make an agent, add your key to it, then forward your agent [00:15:37] !access is https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances << someone logged, do that :P [00:15:37] You are not autorized to perform this, sorry [00:15:45] if someone compromises the bastion host, they can use your key, if you have your agent attached, but otherwise not [00:15:54] !access is https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [00:15:54] Key was added! [00:16:40] and they key itself can't be stolen [00:16:41] Btw is labs "test" or "production" in git or both depending which project? [00:16:42] looks pretty cool to me [00:16:53] !realm [00:16:56] I'm ok with the agent thing [00:17:04] !$realm [00:17:04] either labs or production [00:17:09] labs is test [00:17:13] "I'm okay with this!" [00:17:19] production is production [00:17:21] !realm alias $realm [00:17:21] You are not autorized to perform this, sorry [00:17:29] !realm alias $realm [00:17:29] You are not autorized to perform this, sorry [00:17:35] !realm alias $realm [00:17:35] :P [00:17:35] Successfully created [00:17:37] \o/ [00:17:39] * Damianz shoows wm-bot [00:17:41] Damianz: need wikipedia cloak [00:17:51] or mediawiki [00:17:52] @access [00:17:55] @help [00:17:55] Type @commands for list of commands. This bot is running http://meta.wikimedia.org/wiki/WM-Bot version wikimedia bot v. 1.1.4 source code licensed under GPL and located in wikimedia svn [00:17:59] What's wrong with my wikipedia cloak! [00:17:59] @trusted [00:17:59] I trust: petan!.*@wikimedia/Petrb (2admin1), .*@wikimedia/.* (2trusted1), .*@mediawiki/.* (2trusted1), .*@wikimedia/Ryan-lane (2admin1), [00:18:16] wikipedia cloak isn't trusted. heh [00:18:17] ah [00:18:32] I don't really do wikimedia stuff so asking for that cloak seemed silly [00:18:46] do @trustadd .*@wikipedia/.* trusted [00:19:01] I think only staff can get wikimedia cloaks now [00:19:13] I got one too :o [00:19:17] heh [00:19:34] @trustadd .*@wikipedia/.* trusted [00:19:34] Successfuly added .*@wikipedia/.* [00:19:54] @whoami [00:19:54] You are admin identified by name .*@wikimedia/Petrb [00:20:02] :o [00:20:03] @whoami [00:20:03] You are trusted identified by name .*@wikipedia/.* [00:21:36] !realm [00:21:36] either labs or production [00:21:51] hm I think this could be explained more [00:21:53] :D [00:22:04] !keys [00:22:05] !$realm del [00:22:05] Successfully removed $realm [00:22:43] !$realm is $realm is a variable used in puppet to determine which cluster a system is in. See also $site. [00:22:43] Key was added! [00:22:49] !keys is http://bots.wmflabs.org/~petrb/db/ list of infobot keys [00:22:49] Key was added! [00:23:29] !$site is $site is a variable used in puppet to determine which datacenter a system is in. See also $realm. [00:23:29] Key was added! [00:23:37] !site alias $site [00:23:38] Successfully created [00:23:39] !realm [00:23:39] $realm is a variable used in puppet to determine which cluster a system is in. See also $site. [00:27:00] !ask [00:27:00] Hi, how can we help you? Just ask your question. [00:27:09] :P [00:28:25] Ryan_Lane: I am now to able to access the vivek-puppet instance via password. [00:28:48] Vivek: does it still require it [00:28:48] Ryan_Lane: Neither am I able to use ssh keys as I was trying to configure them just now [00:28:52] "General consensus that we need to deprecate the EC2 APIs (because it does not help OpenStack long term to maintain EC2 APIs over its own)." That makes me happy. [00:28:57] gimme a sec ;) [00:29:01] ok. [00:30:05] you can ssh into vivek-puppet from bastion via password? [00:30:07] right now> [00:30:25] I'm not sure I believe you [00:30:32] Damianz: +1 [00:30:48] unfortunately, I haven't started adding support for the OS API yet [00:30:52] Vivek: I'd tell you more about keys but I just found out that I don't know really much since few minutes ago I didn't even know about forwarding [00:30:52] :S [00:31:14] Hmm does labs use swift for compute storage or gluster for everything? [00:31:15] I can't ssh [00:31:23] with the password [00:31:33] Damianz: gluster. swift is object storage [00:31:41] doesn't help much in our situation [00:31:42] I was trying to configure the key but it requires ssh right ? [00:31:53] Vivek: you used the key to ssh into bastion [00:31:58] I thought you could use Glance or w/e it's callsed as a proxy to swift for serving vm images? [00:32:03] Ryan_Lane: Yes [00:32:06] you need to start an agent on your computer, not on bastion [00:32:13] ok [00:32:14] and you need to add your key to it [00:32:16] Puppet will sort out the ssh keys on bastion [00:32:21] ok [00:32:29] s/callsed/called/ [00:32:52] puppet only sourt out keys you have on wiki, not all in agent [00:32:54] this is all in the very first paragraph of !access [00:32:56] !access [00:32:56] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [00:34:01] On my local machine what equivalent of ssh -A @bastion.wmflabs.org should I give ? [00:34:20] ssh -A vivek@my_puppet_instance ? [00:34:22] s//your username/ [00:34:25] no [00:34:40] yeah, just replace with your username [00:34:42] Ryan_Lane: I am able to reach bastion [00:34:49] I know that [00:34:55] Then from bastion just do ssh [00:34:58] ok [00:35:13] Ryan_Lane: I reach bastion. [00:35:20] Ryan_Lane: now what ? [00:35:27] * Ryan_Lane sighs [00:35:34] Vivek: ssh [00:35:37] please go read up about ssh agent forwarding some [00:35:52] it'll make this much clearr [00:35:54] *clearer [00:35:59] vivek@bastion1:~$ ssh vivek-puppet [00:35:59] Permission denied (publickey). [00:35:59] vivek@bastion1:~$ [00:36:05] then you didn't forward properly [00:36:26] Vivek: easier way is to upload your private key on bastion [00:36:40] you shouldn't do that [00:36:42] petan2: Please let Ryan_Lane do the talking :) [00:36:47] meh [00:36:48] but it's a bit unsecure because it's possible Ryan_Lane will steal it :D [00:36:56] I am getting confused.. [00:36:58] ok, sorry [00:37:00] guys, agent forwarding is simple. [00:37:06] on your local computer... [00:37:12] you have something called an ssh agent [00:37:22] ok. [00:37:24] it is a process that will hold your keys in memory [00:37:26] Argh bugger [00:37:28] ok [00:37:39] so, when you start an agent, you need to add your key to it [00:37:45] ok [00:37:46] eval `ssh-agent` [00:37:49] ok [00:37:56] ssh-add -i [00:38:03] now your key is in the agent [00:38:27] That is not documented Ryan_Lane [00:38:34] Can you document that please ? [00:38:46] now you can ssh into the bastion host, forwarding your agent [00:38:51] it *is* documented! [00:38:53] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [00:39:04] Only ssh-agent is mentioned [00:39:13] the -i option is not mentioned. [00:39:23] ssh-add defaults to id_rsa or id_dsa [00:39:45] anyone that isn't using a default keyname knows this well enough to not need that documented [00:39:59] so... [00:40:16] to forward your agent, you *must* use -A in the options for ssh [00:40:24] ssh -A vivek@bastion.wmflabs.org [00:40:27] The cool kids use .ssh/config :D [00:40:33] Damianz: that works too :) [00:40:40] but, that assumes the person knows ssh well [00:40:53] and you shouldn't have forwarding enabled by default anyway ;) [00:41:14] Ryan_Lane: ssh -A vivek@bastion.wmflabs.org works for me [00:41:18] I do for labs+work bastion servers, apart from that the keys are mine! [00:41:27] Ryan_Lane: My issue is from the bastion to vivek-puppet [00:41:41] it still isn't working? [00:41:50] No [00:41:56] It is not working. [00:42:12] I have no issues connecting till bastion. [00:42:20] type ssh-add -l [00:42:25] on your local system [00:42:36] ok [00:42:37] I did [00:42:44] what does it show? [00:43:15] Mental note; don't try running grep on a switch =/ [00:43:22] heh [00:44:15] Vivek: ? [00:44:37] It shows me the RSA key stored in /home/vivek/.ssh/id_rsa [00:44:48] ok. now ssh -A vivek@bastion.wmflabs.org [00:45:03] make sure you are in the same window [00:45:19] now, on bastion: ssh-add -l [00:45:27] it should show the same key [00:45:48] Yes it shows the same key [00:46:19] ok, now try to ssh to vivek-puppet [00:46:31] ok [00:46:37] I am in vivek-puppet [00:46:50] I figured out the mistake [00:47:01] when you forward your agent, it creates a reverse-tunnel to the agent on your local system [00:47:18] making it seem like the agent on bastion is the same agent as on your local system [00:47:26] therefore allowing access to your key [00:47:36] There was an issue with the way I set up agent on my local machine caused the problem. [00:47:42] Sorted it out. [00:47:46] did you not run it in eval? [00:47:47] Thanks Ryan_Lane [00:48:05] yw [00:48:14] No [00:48:28] there's a reason the documentation says to use eval.... [00:48:43] running ssh-agent without eval will start the agent, but won't connect to it [00:48:49] night [00:49:09] you can run it without eval, then add the output of the command to your environment [00:49:18] but it's easiest to just run it with eval [00:49:22] ok [00:50:37] Now that the instance is set up, what next ? [00:50:41] Does labs use keystone for auth or just grab the api keys from ldap and talk to the api server? [00:50:50] the latter [00:50:53] * Damianz is trying to read the os documentation without falling asleep [00:50:59] we'll need to use keystone with essex [00:51:06] unfortunately [00:51:18] Ryan_Lane: 6.20 AM in India and I have not slept in the night :) [00:51:25] You gonna use dashboard if it becomes a project in essex? [00:51:36] we can't yet [00:51:50] OpenStackManager does some stuff dashboard does not [00:52:02] Like puppet handling? [00:52:05] (and dns) [00:52:10] DNS, Puppet, adding documentation to mediawiki [00:52:30] we are adding DNS support to nova right now [00:52:36] :D [00:52:36] puppet will likely be next [00:52:39] then editing of mediawiki [00:52:39] Via ldap I assume? [00:53:04] the DNS support we are writing is modular, but the first module will be powerdns with ldap, yeah [00:53:56] That will actually be more awesome as I assume then we can use the cli stuff in the same way as OpenstackManager now? [00:53:57] the initial puppet support will also be for ldap :) [00:54:10] yes. that's actually more my goal than using dashboard [00:55:06] but I'd also like to move some complexity out of OpenStackManager too [00:55:11] so it's a win/win there [00:55:36] http://wiki.openstack.org/Atlas-LB looks like an interesting project [00:55:45] ah. yeah [00:56:00] we may add pybal as a service, at some point [00:56:17] pybal is the bespoke mw loadbalencing/failover thing? [00:56:24] I recall reading about that a while back [00:56:28] yeah. it manages LVS [00:56:36] so L2/3 load balancing [00:56:40] atlas is L7 [00:56:57] LVS is useful, I deploy a lot with haproxy so a box can do L2/3/7 [00:57:08] pybal supports BGP, too [00:57:25] it actually has a full BGP implementation written in python [00:57:36] :o [00:57:55] What do you actually use BGP for? As I recall MWF doesn't have a ASN? [00:58:03] Ryan_Lane: I am off to get some sleep. [00:58:09] an* gah can't type. [00:58:11] so, we advertise our service IP addresses to the routers from (at least) two LVS servers, and if one dies, the routers automatically route traffic to the other [00:58:15] Vivek: night [00:58:33] Ryan_Lane: Will you be here today evening ? [00:58:43] nite Ryan_Lane, Damianz. [00:58:43] Ah :) [00:58:46] Night. [00:58:48] I likely won't be back on till monday [00:58:57] ok. [00:58:58] or tuesday [00:59:01] ok. [00:59:05] since monday is actually a holiday for me [00:59:11] ok. [01:00:53] Damianz: so, it can be used as a way to load balance L7 services in an HA way [01:01:08] without needing pacemaker [01:01:36] well, load balance traffic to the L7 services in an L2/3 way [01:01:37] Actually BGP for that sounds like a really good idea [01:01:48] it works really well in practice [01:03:41] pybal looks pretty nice, interesting in the BGP stuff as I'm actually re-writing a ruby script in python that we use to shift traffic around automagically. [01:06:45] wait wait wait, you're multihoming without an ASN? [01:06:53] what is this madness? [01:07:10] AFAIK we have ASNs [01:07:26] * Ryan_Lane is definitely not the right person to be talking about the networking stuff [01:07:37] oh, i was about to say, if WMF doesn't have ASNs, i would be concerned [01:07:55] Hmmm not checked in a while but I don't think so *shrug* [01:08:17] * hyperon is getting Jimbo's face again. [01:08:45] I use a couple of ASNs internally that actually talk to kit off site via a MPLS setup, but we only use BGP for shifting traffic betwean locations rather than any form of any/multicasting. [01:10:25] i meant multihoming, not multicasting, sorry. [01:18:05] Grrrr wtf, my roof has decided to start dripping water again... this time right onto one of the switch stacks :( Silly old building [01:59:35] Ryan_Lane: Progress! http://awesomescreenshot.com/025r2gx5e [09:49:52] PROBLEM Disk Space is now: WARNING on nova-production1 nova-production1 output: DISK WARNING - free space: / 568 MB (5% inode=86%): [12:39:52] PROBLEM Disk Space is now: CRITICAL on nova-production1 nova-production1 output: DISK CRITICAL - free space: / 286 MB (2% inode=86%): [16:14:27] anyone here can help access labs from bastion using putty ? [16:51:20] any1 here? [17:07:03] I am here. [17:07:16] You can use ssh agent. [17:08:23] !access [17:08:23] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [17:08:29] There you go. [18:07:44] Ok, is it just me, or does my home directly on bots-cb also show on bots-apache1? [18:07:51] directory* [19:20:58] OrenBochman: hi [19:50:18] > Consider your security scheme before you create an instance, you can't change the group settings of instance once it's created. [19:50:23] is that by design? [19:50:26] from https://labsconsole.wikimedia.org/wiki/Security_Groups [19:57:48] labs-home-wm: ping [19:57:53] labs-home-wm: foo [19:59:11] anyone know who sara is? [20:03:09] 12/31/2011 - 20:03:08 - Creating a home directory for jeremyb at /export/home/bastion/jeremyb [20:03:25] oooh [20:04:09] 12/31/2011 - 20:04:09 - Updating keys for jeremyb [20:07:17] hrmmm... no way to get fingerprint from the web to verify (for bastion) [21:33:53] hi happy new year [21:50:13] OrenBochman: Thanks [21:50:18] I still have an hour to go [21:50:31] 10 minutes here [22:45:50] OrenBochman: hi [22:46:15] OrenBochman: want to hack some putty? [22:46:19] yes [22:47:23] I have a saved session which connects to bastion [22:47:36] and it works with pagent [22:48:21] but once in bastion ssh labs-mw1 fails [22:48:41] the agent forwarding seems brocken [22:48:45] the agent forwarding seems broken [22:49:05] of course it fails, labs-mw1 isn't a valid place to connect to [22:49:09] it's just an example [22:49:29] so what is [22:49:45] I need to connect to the search instance [22:50:46] * jeremyb goes into SMW learning mode [22:50:46] labs-mw1 actualy seems to exsist [22:52:16] so, [[Resource Type::instance]][[Project::bots]] [22:52:20] has instances [22:52:27] [[Resource Type::instance]][[Project::search]] [22:52:31] has no instances [22:52:44] which makes me think you need to make an instance before you can connect to it [22:53:08] https://labsconsole.wikimedia.org/w/index.php?title=Special:Ask&offset=0&limit=20&q=%5B%5BResource+Type%3A%3Ainstance%5D%5D&p=format%3Dbroadtable&po=%3FInstance+Name%0A%3FInstance+Type%0A%3FProject%0A%3FImage+Id%0A%3FFQDN%0A%3FLaunch+Time%0A%3FPuppet+Class%0A%3FModification+date%0A%3FInstance+Host%0A%3FNumber+of+CPUs%0A%3FRAM+Size%0A%3FAmount+of+Storage%0A&eq=yes [22:53:16] how should I connect to it [22:53:29] you can't connect to an instance that doesn't exist [22:53:39] you need to create a new instance inside the search project [22:54:10] ok I don't get this ystem [22:55:09] can I ssh to get to the search console or not and if not what should be done [23:00:13] OrenBochman: same as i said before: you need to create a new instance inside the search project [23:00:34] OrenBochman: compare: https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bots#Instances_for_this_project vs. https://labsconsole.wikimedia.org/wiki/Nova_Resource:Search#Instances_for_this_project [23:00:43] OrenBochman: one has instances. one is completely empty [23:01:00] OrenBochman: you can't ssh to something that's not only not running but also doesn't even exist [23:01:11]