[00:03:54] RECOVERY Current Load is now: OK on bots-cb bots-cb output: OK - load average: 0.65, 0.89, 4.23 [00:05:05] Helllz yeah [00:05:17] Styled Gerrit set up at http://gerrit.pmtpa.wmflabs:8080/ [00:05:20] Ah, a Krinkle ! [00:05:45] Firefox can't find the server at gerrit.pmtpa.wmflabs. [00:05:55] It's not a public IP [00:05:57] sumanah: Yeah it doesn't have a public IP, you need to proxy to it to get there [00:05:59] :) [00:06:02] Hey Roan [00:06:07] I could ask Ryan_Lane for one [00:06:13] * Damianz straightens Krinkle [00:06:17] Krinkle: You wanna help me out doing some CSS and JS tweaks to Gerrit? [00:06:25] One of the OpenStack guys found a way to do it [00:06:27] Damianz: Gooood morning [00:06:33] RoanKattouw: I read about that [00:06:46] Gerrit needs a whole new face not just a face lift :P [00:06:49] I knwo [00:06:53] Damianz: btw have you heard about https://www.mediawiki.org/wiki/Berlin_Hackathon_2012 ? [00:06:55] RoanKattouw: could you add some more squids to deployment-prep please? it's running quite slow [00:06:55] But I'm gonna do two trivial fixups [00:06:56] Can't find the exact link, but I remember seeing a mockup of some sorts that openstack ppl were proud of [00:07:05] Thehelpfulone: I'm not involved with that project at all [00:07:13] Oooh berlin :D [00:07:16] Krinkle: http://review-dev.openstack.org [00:07:20] Thehelpfulone: Not at the moment [00:07:27] I meant a blogpost, but this was the preview indeed [00:07:35] http://www.linuxjedi.co.uk/2012/03/changes-coming-to-gerrits-style.html [00:07:37] that one [00:07:46] My Gerrit looks exactly the same sans the logo [00:07:52] Oh, LinuxJedi blogged about it? [00:07:54] * RoanKattouw reads [00:07:54] bah, all these sysadmins that aren't involved in the projects! [00:08:03] I'm not even a sysadmin :) [00:08:15] you are, and a netadmin too [00:08:23] your name is catrope right? [00:08:28] Thehelpfulone: he's being modest [00:08:40] Yes [00:08:44] Deployment prep needs some other stuff doing -- like a proxy putting infront of it to distribute to the squids before it can have moar. [00:08:45] I'm sort of a sysadmin, but that's not my main job [00:08:49] The openstack skin is a bit more breathy, which is nice. The colors are bad imho though, but that's just because they're going with the puristic logo-colors only [00:08:49] anyway, RoanKattouw, wanna show me how to ... proxy ... thing? [00:08:50] w [00:08:53] which is a good start [00:08:59] RoanKattouw: how are you connecting to this via a socks-proxy? [00:09:03] sumanah: [[labsconsole:Access]] , see the section about SOCKS proxy [00:09:04] yeah, I'll need to set up that proxy as well [00:09:04] I want to see your changes :D [00:09:05] Ryan_Lane: Yes [00:09:11] it isn't working for me [00:09:16] https://labsconsole.wikimedia.org/wiki/Help:Access#Accessing_web_services_using_a_SOCKS_proxy [00:09:27] What's "it" and how is it "not working"? [00:09:44] sumanah: Essentially that means doing ssh bastion.wmflabs.org -D 8080 and then setting up some FoxyProxy rules [00:10:15] -L 8080:internalip:8080 http://localhost:8080. Screw SOCKS [00:10:22] where do I go to verify that this fingerprint is that of bastion? [00:10:28] Ryan_Lane: I really just did the top two file additions @ https://review.openstack.org/#change,5234,patchset=4 [00:10:47] I don't know where we have these fingerprints [00:10:51] RoanKattouw: I'm connected to a socks proxy on bastion-restricted [00:10:57] Puppet probably knows them [00:11:01] and when I try to connect to your instance on 8080 it doesn't work :( [00:11:25] we don't have the fingerprints available yet [00:11:32] I have some ideas on how to include them [00:11:38] we have an open bug, actually [00:11:41] Ryan_Lane: nc gerrit 8080 GET / works for me [00:11:42] with implementation details [00:11:46] from bastion [00:11:59] Oh wait, that was from gerrit [00:12:15] Same deal on bastion1 though [00:12:21] I can get there from bastion-restricted1 too [00:12:43] Wait, why did bastion-restricted1 boot me? [00:12:46] Is that because it's ops-only? [00:12:53] well... [00:12:58] (for now) [00:13:01] I haven't switched over all shell people yet [00:13:16] I mentioned I'd send a follow-up email when I do that [00:13:30] Right [00:14:19] Hmm [00:14:27] Ryan_Lane: Could I just get a public IP for the gerrit dev instance? [00:15:38] Krinkle: I'll add you to the project so you can SSH into the machine and edit the stylesheet [00:15:47] sure [00:15:56] you should really proxy to gerrit [00:16:00] from apache [00:16:07] Right [00:16:12] I'll look that up in the gerrit puppet manifest [00:16:14] I can't get the thing to work in the browser [00:16:27] That's OK, we're setting up a public IP [00:16:37] ok [00:17:20] :8080 doesn't match my foxyproxy rule :) [00:17:27] heh [00:17:42] RoanKattouw: ok, you can allocate an IP [00:17:46] Thanks [00:17:54] note, the manage addresses special page takes an eternity [00:20:14] 03/15/2012 - 00:20:13 - Creating a home directory for krinkle at /export/home/gerrit/krinkle [00:20:25] Got it working at http://gerrit.pmtpa.wmflabs:8080/ [00:20:33] yay [00:20:37] * Ryan_Lane wants to see [00:20:40] $ ssh @bastion.wmflabs.org -D 8080; and in Mac OS X proxy prefs, localhost:8080 in Socks [00:21:13] ah [00:21:15] 03/15/2012 - 00:21:15 - Updating keys for krinkle [00:21:24] I changed my proxy settings to always use labs :) [00:21:52] Krinkle: You can ssh into bastion.wmflabs.org and then from there into gerrit, the relevant files are /var/lib/gerrit2/review_site/etc/{GerritSite.css,GerritSiteHeader.html} [00:22:20] Once I set up a public IP you can ssh in directly too [00:23:36] Change on 12mediawiki a page Wikimedia Labs was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=510957 edit summary: /* Documents */ [00:24:16] Krinkle: 208.80.153.233 is the public IP [00:24:35] * Krinkle updated https://labsconsole.wikimedia.org/wiki/Help:Access#Accessing_web_services_using_a_SOCKS_proxy [00:24:47] RoanKattouw: add a hostname to it :) [00:24:56] I'm trying to [00:25:01] yeah, slow as shit [00:25:02] But like you said, that address page is hella slow [00:25:08] And I have to reload that every time [00:25:08] really poor code there [00:25:12] Actually associating it was quick [00:25:16] too many lookups [00:26:11] gerrit-dev.wmflabs.org [00:26:47] So full link: http://gerrit-dev.wmflabs.org:8080/ [00:27:08] Can I log in with something? [00:27:12] Ryan_Lane: has db replication been setup on labs yet? [00:27:13] (onto gerrit) [00:27:25] Thehelpfulone: nope [00:27:34] Krinkle: Click "become" in the top right corner [00:27:41] It's in development mode where anyone can become any user [00:27:44] we're about to have public datasets available to all instances read-only, though [00:27:44] okay, I imagine there's a roadmap somewhere that I should look for.. [00:27:50] I may or may not want to disable that now that the instance has a public IP [00:27:55] we don't have a timeframe for it right now [00:28:24] I don't think I like the red text [00:28:37] it makes me think of mediawiki red-links [00:28:43] I just stole the OpenStack stylesheet [00:28:47] Krinkle was also critical of it [00:29:00] the red is awful [00:29:13] I'm hoping he'll channel that criticism into a better stylesheet :) [00:29:23] JRWR: what error are you getting? [00:29:43] I'm gonna mess with the JS now [00:30:22] Ryan_Lane: There were no Nova credentials found for your user account. Please ask a Nova administrator to create credentials for you. [00:30:26] Ryan_Lane: okay, is there an ETA for that, is labs intended to be able to be used to host web tools that might be hosted on the toolserver? [00:30:35] JRWR: you need to logout and login again [00:30:40] that's happened to me a couple of times [00:30:42] from the wiki that is [00:32:05] yay it works [00:34:18] http://gerrit-dev.wmflabs.org:8080/ is supposed to work now?> [00:34:27] I just shut down the proxy [00:34:37] but this urls isn't working for me yet [00:34:38] Krinkle: works for me [00:34:52] and me [00:35:22] Yes, that's supposed to work [00:35:24] It works for me [00:35:55] oh, the stupid socks proxy set in mac os takes everything over [00:35:59] I need to disable it first [00:36:04] can't even access wikipedia [00:36:18] working now :) [00:36:22] ok, with testing the response times (using pingdom) the avg response time of this page: http://en.wikipedia.org/wiki/Index_of_Windows_games_(A) os 3.77s [00:36:41] My wiki with nginx cache response time is 2.32s avg [00:36:46] over 10 tests for both [00:37:07] I tested http://pcgamingwiki.com/wiki/Home [00:39:24] Looks like the nginx is able to connect faster and handle the cache lookup faster then what ever you guys are using, needs more testing, but its looking good [00:41:28] we're very unlikely to switch away from varnish for caching [00:41:57] I know [00:42:04] too much control over varnish [00:42:05] but… it's good to have the info available, especially with stats [00:42:09] !log gerrit Installing Apache [00:42:10] Logged the message, Mr. Obvious [00:42:24] its good for single server setups [00:42:24] and, even when we switch to hiphop, we still need a server for proxying [00:42:32] it may be good to use nginx rather than apache forit [00:42:46] !account-questions | paravoid [00:42:46] paravoid: I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your preferred email address. 3. Your SVN account name, or your preferred shell account name, if you do not have SVN access. [00:42:46] Apache is the replayer of Web servers :) [00:43:06] RoanKattouw: I'll try to wing it towards Vector for starters [00:43:18] OK [00:43:30] I've rewritten the JS using jQuery, now I have to install Apache so I can actually serve jQuery [00:43:45] silly question: how do I reply to the bot? 1. paravoid 2. etc.? [00:43:52] can reply to me :) [00:43:54] I plan on making a guide on how to make a mediawiki install handle a slashdotting, nowadays its called RedditDoS [00:43:54] Hm.. you're changing the actual logic? [00:43:58] it's just a helper for me [00:44:13] JRWR: cool [00:44:14] or is the logic not well separated with JS and do you need to change it in order to make other changes [00:44:35] oh and php5.4.0 is a blazin on Mediawiki 1.18.1 [00:45:48] paravoid: ok, made an account for you. it'll email you a password [00:45:53] !initial-login | paravoid [00:45:53] paravoid: https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [00:46:12] you can ignore the part about resetting your password, since it's sending you one [00:50:46] paravoid: so, I've made you a full admin on labsconsole [00:51:15] * Krinkle opens inbox to see 390 emails from toolserver because SGE is crashing (or rather, it's not up and cronsub commands fall on their face) [00:51:48] admin in the wiki, and a cloudadmin in nova. the former allows you to add people to cloudadmin, the latter lets you do all of the openstack related things in the interface [00:52:17] that said, no one except the ops team should be in either of those things :) [00:52:57] Toolserver is dodgy, has issues pretty much every week. [00:53:11] Krinkle: No, not changing logic [00:53:18] Krinkle: JS exists only to apply the "patch" class to patch views [00:54:09] RoanKattouw: There is a lot more JS in gerrit though, right? You mean that's the only change you're making. [00:54:19] Damianz: labs only has issues when I totally fuck it up! [00:54:20] :D [00:54:57] though a hardware failure would be problematic, depending on the node [00:55:02] Krinkle: No, this is separate from Gerrit's JS. Gerrit's JS is compiled from Java or something, this is injected elsewehere [00:55:16] wtf [00:55:19] ok [00:56:21] Ryan_Lane: I'd like to see our ram graphs if a node failed :D [00:58:08] well, the instances wouldn't restart [00:58:17] so, it would stay about the same ;) [00:59:14] openstack's availability is kind of crap [00:59:42] I could manually cause the instances to start on another node [01:00:07] but I'd have to modify the database :D [01:00:47] RAWR [01:00:51] SSLCertificateFile: file '/etc/ssl/certs/star.wikimedia.org.pem' does not exist or is empty [01:01:00] RoanKattouw: add it via puppet [01:01:03] oh [01:01:04] heh [01:01:06] right [01:01:10] Apparently even the Apache proxy config for Gerrit is WMf-specific [01:01:11] * RoanKattouw STABS [01:01:24] lemme turn it into a template :) [01:01:35] I'll just rip out the SSL part [01:01:39] locally [01:02:03] Wait [01:02:06] :80 redirects to :443 [01:02:26] RedirectMatch ^/$ https://gerrit.wikimedia.org/r/ [01:02:28] Of course [01:02:47] I'm fixing it :) [01:02:48] Can't that just redirect to relative paths [01:02:48] gimme a sec [01:03:04] no, it can't [01:03:25] Boo [01:05:16] [Thu Mar 15 01:05:12 2012] [warn] NameVirtualHost *:80 has no VirtualHosts [01:05:18] Is that normal? [01:05:32] hm [01:05:33] maybe? [01:05:34] :) [01:06:14] PROBLEM Disk Space is now: CRITICAL on mobile-enwp mobile-enwp output: DISK CRITICAL - free space: / 259 MB (2% inode=79%): [01:07:10] ok, pushing to prod, then cherry-picking to test [01:07:25] Meh [01:07:36] Ryan_Lane: How do I open port 80 to the world on my instance? [01:07:47] security group [01:07:54] did you read the docs? :) [01:07:56] !instances [01:07:56] https://labsconsole.wikimedia.org/wiki/Help:Instances [01:07:59] !security [01:07:59] https://labsconsole.wikimedia.org/wiki/Help:Security_Groups [01:08:07] https://labsconsole.wikimedia.org/wiki/Special:NovaSecurityGroup gives me a 404 !! [01:08:13] seriously? [01:08:24] wtf [01:08:24] PROBLEM Free ram is now: WARNING on mobile-enwp mobile-enwp output: Warning: 10% free memory [01:08:26] Something is screwed up there [01:08:38] something is screwed up on labsconsole as a whole [01:08:51] php is broken somehow [01:09:13] did someone puke on apache again? [01:09:30] Ryan_Lane: Wait, did you just DELETE files/apache/sites/gerrit.wikimedia.org ? [01:09:37] That looks wrong [01:09:41] it is. kind of [01:09:43] !g 3179 [01:09:43] https://gerrit.wikimedia.org/r/3179 [01:09:53] I didn't merge it all the way through [01:10:15] ok, what the fuck is up with php on labsconsole? [01:10:22] and why all of a sudden did it stop working? [01:10:57] check error logs [01:11:14] RECOVERY Disk Space is now: OK on mobile-enwp mobile-enwp output: DISK OK [01:11:23] Well, did you depoy 3179 to labsconsole? [01:11:28] no [01:11:39] I didn't even merge it through to production [01:11:43] via sockpuppet [01:12:00] I had this issue at some point before.... [01:12:13] php is installed... [01:12:20] ok, thats good [01:12:25] do the files exist? [01:12:30] Try gracefulling apache, see what errors yyou get [01:12:35] rc libapache2-mod-php5 [01:12:40] that's fucked up [01:12:45] whoops [01:12:58] Oops [01:13:42] how the hell did *that* happen? [01:14:02] wow, the way the core stylesheet is generated is just freaking me out [01:14:08] I didn't even try to find out [01:14:18] -size:9pt;}.GCLMTUVDOB .GCLMTUVDGG{border-left:1px solid ',X0d=';white-spa [01:14:23] that's the second time that's happened too [01:14:47] some kind of javascript interpreter because the class names can potentially change [01:15:00] from what I gather the classnames are chosen as short as possible, so if gerrit adds an element somewhere, all classnames change [01:15:43] I was wondering why the stylesheet from openstack is so generic, not styling any individual elements (like the annoying double border when you're logged one where a start is in-between when logged in), that column has a class but it's not stable [01:15:57] logged in* [01:16:04] the default green stylesheet removes that border [01:16:08] openstack doesn't [01:16:11] I'll take my chances [01:17:18] Krinkle: We can always add JS to find those elements and add classes [01:17:29] yeah, we'll see [01:17:55] http://i.imgur.com/kqPXd.png [01:21:38] hehe, easiest way to get my hands on the dynamic stylesheet is prompt('x', document.styleSheets[1].ownerNode.innerHTML) [01:21:48] can't even access it through web inspector, it's in a sub document [01:21:49] weird [01:22:00] Yay [01:22:06] http://gerrit-dev.wmflabs.org is up [01:22:14] Proxy is working, and port 80 is open to the world now [01:22:39] what happened Ryan_Lane, maybe its something that we can help prevent [01:23:42] PROBLEM Free ram is now: CRITICAL on mobile-enwp mobile-enwp output: Critical: 5% free memory [01:25:13] it's because we have something set to ensure => latest in puppet, and it didn't install cleanly [01:25:33] More boxes going down now [01:25:43] Oh, singer is back up but now ekrem is down [01:25:56] (Apache down that is [01:31:28] well, I surely hope the php on the cluster doesn't have this issue [01:31:28] heh [01:31:59] RoanKattouw: ekrem has other issues [01:32:09] its just overloaded by that AppleDictionary thingie [01:32:32] RoanKattouw: I've messed with chmod a little bit on /var/lib/gerrit2/review_site/etc so that I can edit the file as 'krinkle' in my code editor over ssh [01:32:44] 777 for now, just fyi [01:33:43] PROBLEM Current Load is now: WARNING on bots-sql3 bots-sql3 output: WARNING - load average: 4.42, 6.04, 5.45 [01:35:07] OK, sure [01:35:18] I am editing the .html file now [01:36:27] ok [01:37:13] RoanKattouw: So I reverse engineered the default stylesheet (as you might have noticed) will work from here instead. The openstack one used too many !important and #id heavy selectors to have to fight against [01:38:05] Kept a copy of the default one before I made changes so that if these class names change we can diff GerritSite -gerrit-default-extraction-tidy.css to GerritSite.css and should be able to re-create it [01:38:17] OK, thanks [01:38:33] which we'll need with classnames all looking alike "GCLMTUVDGI" [01:38:36] :D [01:38:38] So am I right that almost all classes and IDs are generated sequentially [01:38:43] RECOVERY Current Load is now: OK on bots-sql3 bots-sql3 output: OK - load average: 1.20, 2.93, 4.27 [01:38:53] Or are class names like GCLMTUVDGI actually stable? [01:39:09] I'm fine with having nondescript class names as long as they're stable [01:39:16] and map to one and only one concept [01:40:29] that's the most ridiculous class name I've ever seen [01:41:18] Hey it's shorter than gerrit-review-state-plustwo [01:41:35] So clearly the shorter class name improves performance, right? [01:42:15] Krinkle: Is review_site/etc/GerritSite.css in a non-broken deployable state? I want to restart Gerrit so it picks up my JS changes, but that picks up your CSS changes in that file as well [01:42:31] deployable where [01:43:54] gerrit-dev [01:44:04] Hmm it looks like it's already sort of on there [01:44:06] nm, restarting [01:44:18] yeah, I've been editing live for the past half hour [01:44:26] seems to apply right away [01:44:29] Yeah [01:44:36] The HTML changes don't but the CSS changes do [01:44:38] apparently [01:44:47] ok [01:46:06] sumanah: http://gerrit-dev.wmflabs.org/r/#change,publish,2,3 [01:46:16] It's buggy cause it also changes the 0 for verified [01:46:48] That's because of the way it's implemented. I'll put it on pastebin, and reading it should make you realize what a terrible terrible person that code makes me [01:50:00] Behold: http://pastebin.com/R1vG7cz4 [01:50:05] One of the most terrible hacks I've ever written [01:50:21] And the fact that it also changes the Verified box tells you why [01:56:43] PROBLEM Current Load is now: WARNING on bots-sql3 bots-sql3 output: WARNING - load average: 7.26, 7.04, 5.90 [01:58:41] RoanKattouw: weird, it seems the default stylesheet comes after GerritSite.css [01:58:51] I'm copying a rule and changing a value, it's overridden [01:58:51] *sigh* [01:59:00] argh [01:59:48] Krinkle: I'll file a bug for that with Gerrit [02:00:52] OK https://code.google.com/p/gerrit/issues/detail?id=864 exists for human-readable class names [02:01:12] let me send you a screenshot of the bug [02:01:24] Please do [02:02:28] http://i.imgur.com/zA2Ej.png [02:02:30] RoanKattouw: [02:04:00] Thanks [02:10:02] https://code.google.com/p/gerrit/issues/detail?id=1290 [02:10:05] Please star [02:10:57] And star https://code.google.com/p/gerrit/issues/detail?id=864 too [02:20:21] +3 stars on both :) [02:20:47] nice now that googlecode supports easy google account switch [02:21:55] haha [02:25:12] grmpf, this gerrit stuff is getting ugly. I'll swap the openstack one and my working copy. It's too much work to make little changes. [02:25:28] not worth the time to wing it, probably better to instead design off-code first [02:26:19] what did you change in the html? [02:26:29] added "Gerrit Code Review" ? [02:26:30] Im surpised you didnt go with something like Github [02:26:59] Yes [02:27:02] 03/15/2012 - 02:27:02 - Updating keys for andrew [02:27:05] 03/15/2012 - 02:27:04 - Updating keys for andrew [02:27:06] 03/15/2012 - 02:27:06 - Updating keys for andrew [02:27:14] 03/15/2012 - 02:27:14 - Updating keys for andrew [02:36:40] Krinkle: Yeah, added that and a bunch of JS [02:36:46] k [03:29:34] any known issues with labs bastion at the moment? [03:31:57] you guys can still login as normal? [03:44:01] !log bastion - getting my Connection closed after the "If you are having access problems.." message. just me or all ? [03:44:02] Logged the message, Master [04:14:40] PROBLEM Current Load is now: WARNING on bots-sql3 bots-sql3 output: WARNING - load average: 2.03, 4.73, 5.04 [04:19:40] RECOVERY Current Load is now: OK on bots-sql3 bots-sql3 output: OK - load average: 2.63, 3.70, 4.51 [04:33:50] PROBLEM Current Load is now: CRITICAL on mobile-enwp mobile-enwp output: CHECK_NRPE: Socket timeout after 10 seconds. [04:38:50] PROBLEM Current Load is now: WARNING on mobile-enwp mobile-enwp output: WARNING - load average: 8.60, 8.40, 8.45 [05:18:45] * jeremyb stumbles in [05:19:43] yeah, so i am being disconnected by the bastion host [05:19:51] mutante: ssh -vvv ? [05:20:01] but AFTER i already get the "If you are having access problems.." message. [05:20:08] true, lets see [05:21:00] i get [05:21:04] debug3: input_userauth_banner [05:21:04] If you are having access problems, please see: https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [05:21:07] debug1: Authentications that can continue: publickey [05:21:15] so, you seeing the banner means nothing [05:23:06] mutante: paste it? [05:23:34] i can't say stick it in my $HOME because you can't connect ;-P [05:24:19] re, distracted [05:24:23] debug2: we sent a publickey packet, wait for reply [05:24:23] debug1: Server accepts key: pkalg ssh-rsa blen 535 [05:24:23] debug2: input_userauth_pk_ok: fp 15:23:97:1a:2a:fc:c0:f3:6a:3b:f6:5f:71:d8:86:fd [05:24:26] Connection closed by 208.80.153.194 [05:26:57] that's -vvv ? [05:28:38] -vv [05:28:50] debug1: Server accepts key: pkalg ssh-rsa blen 535 [05:28:50] debug2: input_userauth_pk_ok: fp 15:23:97:1a:2a:fc:c0:f3:6a:3b:f6:5f:71:d8:86:fd [05:28:53] debug3: sign_and_send_pubkey: RSA 15:23:97:1a:2a:fc:c0:f3:6a:3b:f6:5f:71:d8:86:fd [05:28:56] Connection closed by 208.80.153.194 [05:29:07] mutante: try -vvv ? [05:29:19] oh [05:29:21] you just did [05:29:22] that is -vv now, debug3 [05:29:27] -vvv [05:32:59] $ ssh-keygen -l -F bastion.wmflabs.org | tail -n 1 [05:33:02] 2048 26:02:b9:20:f7:6c:5c:c8:2d:58:a2:4c:27:f8:f5:a7 bastion.wmflabs.org (RSA) [05:33:02] mutante: ^ ? [05:34:38] mutante: you can log into the wiki? [05:37:40] hah [05:37:40] export INSTANCENAME=bots-apache1 [05:37:41]