[00:02:37] !log deployment-prep j: run mwscript rebuildLocalisationCache.php --wiki=commonswiki --threads=2 [00:02:39] Logged the message, Master [00:04:59] Ryan_Lane: any chance I could get that IP quota? :) [00:05:34] yep [00:05:50] done [00:08:25] Ryan_Lane: oh do we have SSL certs at all? [00:08:34] self-signed only [00:08:37] nod [00:08:58] kinda hard to do wildcard certs if you're giving them to random untrusted people [00:11:02] indeed [08:08:00] Tanvir: that place is better for labs [08:08:11] Tanvir: are you in the 'bots' project ? [08:08:15] can you log on the bastion ? [11:00:52] http://www.bbc.co.uk/news/magazine-18892510 < shiny, shame no mention of labs but I'm happy with a mention of cbng :D [12:26:32] Hi there is a service by default in a new instance on the port 80 ? After "netstat -nap" I saw any service on it. (Problem : Failed server socket bind failed: Permission denied) [12:27:40] in the group security I inserted the port 80 :-\ [13:40:00] karima: are you root when trying to bind to port 80? [13:40:30] (sfaik, there is no default service already bound to port 80.) [13:41:36] (and the root bit is because IIRC normal users can't bind to ports below 1024) [13:44:31] maplebed : yes in theory with sudo [13:45:05] I can use other port like 9000 but I tested the port 80 [13:48:23] !log deployment-prep rebooting apache33 so it can fsck /dev/vdb [13:48:25] Logged the message, Master [13:48:45] maplebed : you are right... I can't be root on the instance. [13:49:26] that'd be why then. [13:51:31] the password root is the same my account ? [13:52:33] maybe is in relation with the role webadmin ? I 'm not web admin I think [14:04:30] maplebed : thanks I'm not netadmin It's probably the good reason bye. [14:04:59] karima: you suhold only need to be sysadmin, not netadmin. [14:05:59] maplebed : hmm... there are a firewall to config in the instance ? [14:06:15] just the security group. [14:07:22] I created the group LinkedData-Endpoint. I think it is good [14:08:29] so as a test, try this. [14:08:40] on your instance, run "sudo nc -l 80" [14:08:53] on the bastion host, run "telnet 80" [14:10:41] maplebed : run "sudo nc -l 80"... it's long ... ? [14:12:26] that's just the simplest form of a network and root access test. [14:12:50] it says "use netcat to listen on port 80 and print whatever comes in" then, on the bastion, "connect to port 80 on my instance" [14:13:06] if the connection succeeds, your sudo access and your ntework are all working and the problem lies elsewhere. [14:13:48] when (if) the telnet connects and you type stuff on the bastion host, you should see it on the instance. [14:13:52] when you disconnect, nc will terminate. [14:15:50] maplebed : waou... I resume I open another shell in bastion and in this shell I use netcat on port 80 but nc -l 80 have to run in the first shell. It's good ? [14:16:49] ummm... sure. [14:17:23] lol...wait [14:18:34] karima: for more info on netcat, check out something like http://www.go4expert.com/forums/showthread.php?t=26082 [14:19:23] !log deployment-prep root@deployment-cache-bits02:/var/lib/git/operations/puppet(git:13304/10)# git fetch anonymous refs/changes/04/13304/12 && git checkout -b 13304/12 FETCH_HEAD [14:19:24] Logged the message, Master [14:19:43] @seach cache-bits02 [14:19:56] HOLY FFKKKK SHIT [14:20:05] I am never going to end that :-( [14:20:08] hashar: try @search :P [14:20:08] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class ntp::client for i-0000031c.pmtpa.wmflabs at /etc/puppet/manifests/site.pp:55 on node i-0000031c.pmtpa.wmflabs [14:20:41] hashar: these classes are getting deleted a bit often [14:20:58] we should make some rule that when some class is created in puppet, it's not removed before notification [14:21:07] because instances that are using it get broken [14:21:32] well I would love to get a report in mail whenever puppet is broken :-] [14:21:36] would let us detect such issues [14:21:47] in this case the ntp was migrated to a module [14:21:54] when you remove a class you do that in some editor :) [14:21:59] and puppetmaster self does not have any sym links for modules [14:22:05] it's not about making it automatic it's about telling people how to do that [14:22:34] if you delete apache class it makes sense that some apaches servers will fail to update [14:22:56] unfortunatelly the classes are being deleted too often [14:23:08] since labs started it's like 10th version of apache [14:23:41] maplebed : Nothing pass between the client and server [14:23:43] @search bits [14:23:43] No results were found, remember, the bot is searching through content of keys and their names [14:24:36] !log deployment-prep fixed puppet on cache-bits02 : ln -s /var/lib/git/operations/puppet/modules /etc/puppet/modules . That was an empty directory, thus prevented puppet to find the modules and made it breaking when trying to install ntp::client [14:24:37] Logged the message, Master [14:24:44] maplebed : firewall I think... [14:27:02] petan: didn't we got a bug report about nagios being broken on labs ? [14:27:08] dunno [14:27:17] but it's still broken [14:27:25] I would like to speak with mutante-away to fix it [14:27:39] problem is that instances refuses to speak with new server [14:27:48] + there is no service for passive checks [14:28:15] I really don't remember well how nagios did work [14:28:23] I should have write it down [14:28:44] maplebed : sudo iptables -L is empty ... strange who block the port 80 ? [14:28:48] well it broke 9 days ago [14:28:54] :-/ [14:29:20] I don't think so [14:30:08] How see the config of puppet may be I would see the command to open the port ? [14:30:50] ok I don't know what I have just done, but it wasn't nagios I unbanned :D [14:30:54] heh [14:31:01] it seems it's not banned [14:31:06] I don't know why it's quet [14:34:17] nape is fun: select(5, [4], NULL, [4], {0, 32683}) = 0 (Timeout) [14:34:18] accept(4, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable) [14:36:47] so it does : pid 32333] setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 [14:36:47] [pid 32333] bind(4, {sa_family=AF_INET, sin_port=htons(5666), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 [14:36:56] then listen() [14:37:17] and the accept tell me that there is resource temporarily unavailable [14:37:18] accept(4, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable) [14:37:19] :( [14:37:26] karima: there are no host-based firewalls on labs instances by default [14:37:37] maplebed : it's not very important for the moment I can use 8080 ;) [14:38:01] if you can, that's what you should do. [14:39:35] !log deployment-prep dist-upgrade on cache-bits02, will reboot after that (bits.beta.wmflabs.org will be disabled while it reboot) [14:39:36] Logged the message, Master [14:39:50] ahh Resource temporarily unavailable is EAGAIN and that's not really an error. It means "I don't have answer for you right now and you have told me not to wait, so here I am returning without answer." [14:40:52] petan: could it be because the port 5666 is not allowed by default ? [14:41:04] yeah [14:41:10] but how it worked before? [14:41:17] the rule got broken ? ;-D [14:41:26] who broke it? [14:41:27] don't we have a default / implicit rule to allow 5666 on all instances? [14:41:41] I think so [14:41:43] I don't even know where the labs nagios is :/ [14:41:51] in nagios project :) [14:41:58] oh of course [14:42:45] Hmm port 5666 tcp is allowed from 10.4.0.0/24 [14:42:52] is that the nagios range ? [14:42:55] yes [14:42:58] should be [14:43:23] !log deployment-prep rebooting bits02 [14:43:24] Logged the message, Master [14:43:47] error: No output returned from plugin) [14:43:53] wtf it mean [14:44:05] does it mean plugin on checked host? [14:44:09] or plugin on nagios [14:44:22] eh [14:44:51] that is the nagios side [14:44:57] the logs there should give some hints [14:44:58] really? [14:45:04] hm [14:45:09] maybe the nagios instance is no more allowed to run query outside :-D [14:45:21] why would that be [14:45:26] maybe new ubuntu [14:45:27] :D [14:45:58] previous nagios was running on 10.04 [14:46:10] that's my favorite version [14:46:17] :D [14:46:26] it contains all cool packages and works well [14:46:45] for some reason I believe that ubuntu becomes usable 1 year after it's released [14:46:55] because that's a time when they put back all stuff they forgot [14:47:05] and fix what's broken heh [14:47:22] in the config of group security 0.0.0.0/0 is equivalent : anybody or everybody ? [14:47:40] uh [14:47:55] karima: everybody == anybody [14:48:03] or it sounds like that [14:48:08] sorry nobody [14:48:11] aha [14:48:16] lol my english... [14:48:20] it's everybody [14:48:42] there is no point in making a rule for nobody [14:48:57] and if the field CIDR is empty ? [14:49:07] I don't think it let you save it [14:49:50] hm in my config by default there are fields empty in this column... [14:50:10] in that case it's everybody I think [14:50:28] ok thanks [14:50:34] yw [15:05:10] karima: by default everything is rejected [15:06:29] hashar : you speak of port 80 ? [15:07:01] anything [15:07:13] for port 80 I think there is a default security group named "web" [15:07:23] that allow 80 (for http) and 443 (for https) [15:08:18] in my security group I inserted the port 80 [15:08:59] but It doen't work... localhost:80 doesn't work also [15:45:28] Somebody can delete l'instance i-00000360 (the version of ubuntu isn't not good)... the page delete doesn't work :( [16:53:29] Error with the puppet : Mediawiki-install : Dependency Exec[/bin/ln -s /etc/ssl/certs/wmf-labs.pem /etc/ssl/certs/$(/usr/bin/openssl x509 -hash -noout -in /etc/ssl/certs/wmf-labs.pem).0] has failures: true [16:54:11] I don't insert ssl in the group security may be for that ? [16:54:24] o.0 [16:54:36] That looks like puppet failed to sign cert/generator the csr [16:56:26] Damianz : and so ? I finish the install of mediawiki myself ? [16:57:20] apache is ok... only the folder mediawiki and the config is missing [16:57:46] I'd assume either a) there's a file missing that's suppose to be there or b) the manifest needs fixing to acocunt for non-ssl stuff. Would need to go poke the files but I need to go make food right now [16:58:31] ok no problem... I look for the moment. It's no urgent [16:58:56] look at... not look for [16:59:02] Try force running puppet (puppetd -tv) and see if it works [16:59:06] bye [16:59:22] it's the 3 times... [16:59:39] and the same errors [17:00:10] hmm, probably broken then. Some things do just need to run twice [17:00:26] !log deployment-prep j: clear messages after updating localization(cache/l10n) to get new messages in TMH: php MWScript.php ../php/extensions/WikimediaMaintenance/clearMessageBlobs.php --wiki=aawiki [17:00:29] Logged the message, Master [18:09:59] karima: Are you around? I'm interested in the problems you had building a mediawiki instance. [18:10:08] yes [18:10:39] andrewbogott : you want the log of puppet ? [18:10:53] sure, that'd be a good start. [18:11:11] also, what's the instance name? [18:12:48] after updating the configs on the beta cluster, how do you get the changes to actually go into effect? [18:13:24] also, can someone give me admin rights on http://en.wikipedia.beta.wmflabs.org? [18:13:32] Ryan_Lane: about yet? [18:13:38] error is not the same [18:13:49] werdna: depends [18:13:56] instance is i-00000361 [18:14:16] werdna: what's up? [18:14:18] Ryan_Lane: I don't seem to be able to log into an instance with ssh over port forwarding [18:14:27] where you want the log ? [18:14:34] Ryan_Lane: getting Permission denied (publickey). [18:14:44] but I *can* log into bastion [18:14:44] werdna: using proxy command? [18:14:57] or a forwarded agent? [18:15:27] werdna : I had the same message, I deleted the last line in the file .ssh/host... [18:15:40] neither, using ssh -L5022:echo-xmpp:22 -AMNf / ssh localhost -p 5022 -i ~/.ssh/id_rsa-wikimedia [18:15:59] werdna : do a backup before... [18:16:46] andrewbogott : Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Sshkey[10.4.0.2] is already defined in file /etc/puppet/manifests/ssh.pp at line 36; cannot redefine at /etc/puppet/manifests/ssh.pp:36 on node i-00000361.pmtpa.wmflabs [18:18:46] May be all these problems are linked... puppet, ssh, ssl... [18:19:33] Using ProxyCommand does the trick anyway [18:19:35] thanks Ryan_Lane [18:21:24] werdna: yw [18:21:38] karima: well, that error occurs occasionally [18:21:47] karima: if that happens, just try again [18:22:10] the sshkey error is kind of a BS one [18:22:14] kaldari: I am looking at the page triage ext on 'beta' :) [18:22:40] eh? [18:22:42] Ryan_Lane : I had a another problem... I cannot use the port 80. It blocked ? [18:22:59] !security | karima [18:22:59] karima: https://labsconsole.wikimedia.org/wiki/Help:Security_Groups [18:23:19] !cookies | Damian [18:23:21] :( [18:23:22] karima: port 80 is blocked from where? [18:23:35] !cookies is no cookie for you [18:23:35] Key was added [18:23:38] !cookies | Damianz [18:23:38] Damianz: no cookie for you [18:23:57] * Damianz steals Ryan's cookies from lcarr [18:24:21] I inserted the port 80 in the group of security and I have install on another instance the server apache correctly(only a prob with mediawiki) [18:24:46] kaldari: your mail about http://en.wikipedia.beta.wmflabs.org/wiki/Special:NewPagesFeed not working :-D [18:25:01] kaldari: have you send the fix via Gerrit or just as a live hack ? [18:25:12] just live hack [18:25:23] I just removed the override [18:25:37] karima: using which CIDR? 0.0.0.0/0? [18:25:52] do I need to run a script to have the configuration change take effect on all the wikis? [18:26:09] Ryan_Lane : yes [18:26:24] karima: and where are you trying to access port 80 from? [18:26:26] kaldari: maybe it is in the cache [18:27:03] Ryan_Lane : 80 also block in local [18:27:06] kaldari: that was the conf cache :-] [18:27:18] karima: then apache isn't running [18:27:20] kaldari: it is only refreshed when InitialiseSettings.php is changed [18:27:35] kaldari: http://en.wikipedia.beta.wmflabs.org/wiki/Special:NewPagesFeed seems to try to do something, at least the special page does exist [18:27:50] oh, it's working now [18:27:59] yeehaw! [18:28:14] hashar: thanks for all the help! [18:28:15] kaldari no, files are at nfs [18:28:50] Ryan_Lane : the problem with this port is in i-0000035f And apache is in the i-00000361 [18:28:51] hashar: is InitialiseSettings-wmflabs.php version controlled? [18:29:14] karima: I don't think I understand what you mean [18:29:18] looks like it [18:29:27] hashar: I'll check in my changes [18:29:40] kaldari: great :) [18:29:53] kaldari: updating the configuration on beta is a bit complex because we have SOOO many live hacks [18:30:01] I usually do a git stash to save the hacks [18:30:04] then git pull from gerrit [18:30:10] and reapply the live hacks with git stash apply [18:30:50] Ryan_Lane : I installed a instance 0000035f but the port 80 dont work. [18:31:04] karima: did you check to see if apache is running on it? [18:32:05] karima: you can put the puppet log up on pastebin.com. What is your project name? [18:32:21] Ryan_Lane : I install another instance after with puppet mediawiki and miracle the port 80 work for apache [18:33:09] I'm sorry, I don't understand what you are trying to do [18:33:44] andrewbogott : http://pastebin.com/yMnFETn0 [18:33:48] if you can't access port 80 locally, then the web server isn't running [18:35:01] Ryan_Lane : there are two instances one for the wiki test and another for the endpoint sparql [18:35:42] ah [18:35:44] karima: I think that particular error is from a race, and that if you try again a few times you'll get past it. I could be wrong... [18:35:58] so, again. if you can't access port 80, then the webserver isn't started [18:36:58] unless there's some iptables rule that's missing :P (kidding) [18:37:31] If he'd managed to firewall off localhost access then he had rather larger issues [18:37:47] nah, I'm teasing Ryan based on a discussion that we had yesterday :) [18:37:50] ignore me [18:37:55] heh [18:38:13] * Damianz SNATs paravoid to Ryan_Lane [18:40:47] Ryan_Lane : It's no important for the moment. I look at only this week the difficulty. I have not the time to debug that this week. In september, I have the time to install... firewall, analyze puppet etc apache... [18:41:00] * Ryan_Lane nods [18:42:07] I well delete the instances of test tomorrow [18:42:17] will [18:45:17] <^demon> Ryan_Lane: Cool reason we should compile our own gerrit--I can compile on labs with -Dgwt.style=DETAILED so I can properly debug JS issues :p [18:45:27] heh [18:45:55] <^demon> Kinda need the gerrit manifest overhauled first before I play with it further. [18:45:59] <^demon> *hint hint* [18:46:44] * Ryan_Lane nods [18:58:17] andrewbogott : http://pastebin.com/q8dQGT7R [18:59:03] the initial problem [18:59:55] karima: Looks like the upstream mediawiki git server was offline. Did things come up and run during subsequent puppet runs? [19:00:08] It wouldn't shock me if a failed first run means that that manifest never works properly :( [19:02:34] got the config issue fixed [19:02:51] can someone give me admin rights on http://en.wikipedia.beta.wmflabs.org? [19:02:58] andrewbogott : you can not duplicate an instance with Mediawiki ? [19:03:12] pretty please :) [19:03:47] karima: I don't think I understand your question. [19:05:04] karima: I'm not sure there's anyone around that can do so right now [19:05:08] err [19:05:09] sorry [19:05:15] kaldari: I'm not sure there's anyone around that can do so right now [19:06:23] looks like JamesA can do it [19:06:26] I'll bug him [19:06:32] when he's out of his meeting [19:07:06] Ryan_Lane : duplicate an instance is more simple than to do one puppet (script). I think... but I'm again a newbie... [19:07:18] there's no instance cloning [19:07:57] karima: Yeah, creating a scratch instance with puppet is your only option. And, it's not typically very hard, you were just unlucky. [19:08:03] Ryan_Lane: actually, I'll just change it myself in the DB :) [19:08:10] that'll work :) [19:08:16] Ryan_Lane : how are you install the containers ubuntu precise & co ? [19:08:31] they are qcow images [19:08:39] which are totally bare [19:08:59] when the instance is launched, it builds itself from puppet [19:09:38] andrewbogott: Around? [19:10:20] Jarry1250: Spread a bit thin, but around. What's up? [19:11:01] You may recall we were working through the (your?) instructions for setting up a MediaWiki installation on labs [19:11:26] Well, I ended up dualbooting Ubuntu, which got me a bit further [19:11:35] But I still can't quite get my proxy to work [19:12:35] So when/if you have a minute, it would be great if you could help me debug, [19:15:13] Jarry1250: OK, so your problem is with getting ssh access to an instance? [19:18:20] andrewbogott: Well, I actually got my proxy working - whatsmyip shows WMF HQ [19:18:42] But I still can't access anything that looks like a wiki [19:18:51] translatesvg.pmtpa.wmflabs times out [19:19:15] Thanks everybody, gerrit and the instances are great tools and I think also It's a place where magic happens ! I will be back in september to push my dev. Bye [19:23:55] Jarry1250: It times out for me too. So I think you have a server problem and not a proxy problem. [19:24:36] Unless you know of others who can access that web server? [19:30:42] andrewbogott: No, I don't know anyone who can access it, but neither do I know how to fix the server if it is a server problem, maybe you can help? :) [19:44:48] * andrewbogott knows what Jarry1250's problem is. But, too late! [20:11:54] andrewbogott: Do go on :P [20:12:25] (was called away, sorry) [20:12:56] Jarry1250: Your 'web' security group looks broken. You'll need to fix that, and then, hm... [20:13:10] Ryan_Lane: Do changes to a security group apply to instances that are already in that group? [20:13:42] kk, I just copied off of another project, loads of groups named "web" all seemed to have the same rules so I figured they were correct ones... [20:13:44] Jarry1250: So, basically, I think your instance is working, and your proxy is working, but your isntance isn't allowed contact with the outside world via the web ports. [20:13:48] andrewbogott: yep [20:14:12] I'm really hoping in the nova api that we can add/remove security groups from instances too [20:14:25] Jarry1250: No, your 'web' security group is very strange… it doesn't specify IPs for any of the rules, and the rules are just for every port rather than for specific web ports. [20:14:30] !security [20:14:30] https://labsconsole.wikimedia.org/wiki/Help:Security_Groups [20:14:51] Jarry1250: I added an example to ^ that link about how a web security group should look. [20:15:15] Jarry1250: According to Ryan, if you fix that security group then your instance should start behaving immediately. [20:15:18] Hmm, maybe the group rule copier thing doesn't copy CIDR ranges? [20:15:38] ...or maybe I didn't use it properly (more likely... [20:16:54] group rule copier? [20:17:27] Well, group rule. [20:17:35] I probably just misunderstood the concept. [20:17:45] generally don't use group rules [20:17:59] I really need to hide that in an "advanced" section [20:18:09] it does something interesting, but it's a confusing concept [20:20:28] New patchset: Ryan Lane; "Adding admin_token for keystone" [labs/private] (master) - https://gerrit.wikimedia.org/r/16670 [20:22:17] andrewbogott: Okay, so I've opened 80 and 443, would you mind testing first to save me rebooting into Ubuntu? [20:24:53] Jarry1250: I can reach Apache now. Mediawiki seems messed up somehow, not sure what's wrong yet. [20:26:13] Okay, well brb, Ubuntu here I come [20:29:36] Change merged: Ryan Lane; [labs/private] (master) - https://gerrit.wikimedia.org/r/16670 [20:30:20] jarry1250: rather than dual booting, why not run ubuntu in virtualbox? [20:30:28] or use cygwin [20:31:22] Ryan_Lane: Good shout on virtualbox, I'll have to take a look [20:31:25] Running an ubuntu server in virtualbox is what I do. Pretty fast, works great, free. [20:31:34] yep. way easier [20:31:39] and dual booting sucks :) [20:32:46] Well, the four hours literally wasted trying to get "ssh -D" to work on Windows sucked more, believe me [20:32:52] heh [20:32:59] good point [20:34:29] andrewbogott: So, it redirects to [[Main Page]] then refuses to find a server at that address? [20:35:03] Oh wait, that's me [20:35:12] *adjusts proxy URL rules* [20:35:55] I really need to start using puppetmaster::self [20:36:42] Jarry1250: The problem is that you set it up to use pmpta instead of pmtpa. so it redirects itself to a non-existent address. [20:37:54] Jarry1250: It's possible that fixing that typo in the puppet config will fix it, but equally possible that it won't. Give it a try and rerun puppet, then grep for the misspelling in the apache config and see if it helped :) [20:39:15] D'oh! [20:39:19] Thanks, I'll do that now [20:40:06] uuuuggghhhh [21:09:02] Right, at the risk of looking like an idiot, is there (S?)FTP support? [21:11:06] Jarry1250: I think that sftp uses the ssh port, so it should just work. [21:11:13] Or, should work inasmuch as ssh works. [21:11:55] Jarry1250: Did changing the host name in the puppet definition fix the redirect? (If not, I'll make a note to fix that, since I've made that same mistake several times.) [21:12:12] andrew: Doesn't appear to, no. [21:12:24] I'm trying to find the relevant apache config [21:13:22] I'll look too, will let you know if/where I find it [21:14:04] Oh, wait, I think it's a mw setting. Lemme look at how I did this [21:15:23] bah, the path is passed into install.php; who knows that happens after that :( [21:17:31] Jarry1250: It may be enough to change it in srv/mediawiki/orig/LocalSettings.php [21:17:51] incidentally, is that the same as /var/www/srv/... ? [21:17:55] (Which, unfortunately, that file is not directly created by puppet; puppet prompts the mw installer to create it. ) [21:18:00] Yeah, I think it's linked. [21:18:04] Cool. [21:18:16] Hmm, SFTP host? bastion? But how does forwarding work *aaah* [21:20:11] It is /probably/ possible to do it via tunneling. But I don't know how, offhand. Something like this maybe? https://labsconsole.wikimedia.org/wiki/Help:Access#Using_ProxyCommand_ssh_option [21:21:38] Hurray, testwiki working! Victory! [21:22:10] Jarry1250: OK, so, postmortem… the problems were [21:22:16] a) poor security group documentation [21:22:39] b) Unvalidated/hard-to-correct site address [21:22:45] That it? Or was there another roadblock? [21:22:56] Well, I needed to install Ubuntu [21:23:14] But apart from that, no :P [21:24:04] Ryan_Lane: does anyone use SFTP to access files, do you know? [21:24:51] I use scp all the time [21:24:59] so, I'd imagine sftp works just fine [21:25:56] With the config that Andrew linked to? [21:26:00] andrewbogott: http://www.mail-archive.com/openstack@lists.launchpad.net/msg14880.html \o/ [21:26:14] jarry1250: should work with that, yeah [21:26:33] a load balancer service, written in python! likely to be moved into quantum! [21:26:43] Ryan_Lane: Will that get us our web proxy as well? [21:26:54] it should. it's based on the atlas API [21:27:04] we could likely add a driver for pybal too [21:27:12] cool. I'll catch up on that thread. [21:27:27] the atlas api is kind of giant [21:29:06] is there a chance to get rid of "If you are having access problems, please see: https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances"? [21:37:27] giftpflanze: is it causing issues with your client? [21:37:37] there's a way to silence it with an option to ssh [21:38:58] which option is it? [21:41:32] -q? [21:42:13] can i make it work for direct ssh (without sshing to bastion manually)? [21:43:19] ok, works [21:43:40] It looks like I can't create new wiki accounts on http://en.wikipedia.beta.wmflabs.org (like I can on http://ee-prototype.wmflabs.org) [21:44:02] whenever I try it says "Incorrect or missing confirmation code." [21:58:22] Okay, looks like there aren't any SFTP GUI tools that support ProxyCommand [21:58:33] Or at least, none that I could find [22:28:02] sure. gimme s ec [22:29:00] j^: ok. good to go [22:29:02] Right, I'm back: andrewbogott: What's Admin's password set to?! [22:30:05] (not particularly urgent, but nice to know) [22:30:52]