[00:03:51] andrewbogott: Coren Ryan_Lane can I have another public IP for proxy project? [00:11:31] Yu certainly can. If one of us upped your quota, you would. [00:12:06] But I was about to head out. Can you survive until a bit later/tomorrow early without? [00:12:54] Coren: possibly, yeah. [00:13:33] YuviPanda, I can do it, just a second... [00:13:37] andrewbogott: woohoo [00:13:38] ty [00:16:13] YuviPanda, ok, should be set. [00:17:11] andrewbogott: thank you! [00:19:29] YuviPanda: IP and hostname removed by Andrew, can you add me? [00:19:37] mutante: moment [00:19:57] !log wikistats switching from public IP to YuviProxy [00:19:58] Logged the message, Master [00:20:35] mutante: added. http://wikistats.wmflabs.org/ doesn't resolve yet but [00:21:30] o.O [00:21:34] What's YuviProxy? [00:21:40] I like the sound of that. [00:21:47] hehe [00:21:49] dynamicproxy [00:22:00] Is it faster than instance-proxy? [00:22:14] legoktm.instance-proxy.wmflabs.org is pretty slow. [00:22:23] YuviPanda: it changed to 502 Bad Gateway [00:22:27] yeah, saw that. [00:22:28] looking [00:23:27] mutante: hmm, weird. 2013/09/11 00:23:07 [error] 6782#0: *148 wikistats-01 could not be resolved (3: Host not found), client: 216.38.130.164, server: , request: "GET / HTTP/1.1", host: "wikistats.w [00:24:24] mutante: hmm, dig wikistats-01 gives me an nxdomain [00:24:40] mutante: while dig multimedia-dragons works [00:24:46] PING wikistats-01.pmtpa.wmflabs (10.4.0.9) 56(84) bytes of data. [00:25:13] hmm [00:25:43] mutante: why is dig returning NXDOMAIN? [00:26:42] YuviPanda: wikistats-01.pmtpa.wmflabs works [00:27:34] i don't know the different to multimedia-dragons yet, besides that it's an old instance [00:27:38] mutante: so it'll now work 1/3rd of the time [00:27:42] mutante: beacuse I added the IP directly [00:27:46] let me make it work 100% now [00:27:54] sees it work on reload:) [00:30:47] mutante: try now? [00:31:43] mutante: hmm, wikistats.wmflabs.org is NXDomain on my laptop, but if I set host explicitly it works [00:31:45] YuviPanda: it works:) thanks [00:31:49] mutante: woohoo :) [00:31:56] mutante: no https yet, I reverted it. [00:32:05] mutante: but it is in puppet, so should be there soon [00:32:05] wikistats.wmflabs.org has address 208.80.153.190 [00:32:14] i see https with cert warning [00:32:17] cool [00:33:51] don't tell RIPE ,hehe [00:33:58] mutante: yeah, but puppet will kill it on next run ;) [00:34:02] since those are local modifications [00:34:25] gotcha [00:35:19] so you said you needed puppet .. install_certificate{ $certificate: } [00:35:34] for the snakeoil cert? [00:36:53] mutante: not sure, I'm just hand installing certs now [00:37:02] mutante: I need to figure out how to get puppet to do that [00:37:16] mutante: https://gerrit.wikimedia.org/r/#/c/83773/ [00:37:19] still working on it [00:37:21] but it works in theory [00:37:21] so [00:37:27] that's the thing above.. [00:37:38] install_certificate, that's what we use in production to install certs [00:37:52] example: manifests/role/protoproxy.pp: install_certificate{ 'star.wikimediafoundation.org': } [00:38:04] it installs cert and key [00:38:24] if you have cert in public puppet and key in private and follow the naming convention [00:38:40] i suppose the key would be in labs/private ? [00:38:58] mutante: yeah [00:39:03] mutante: but I don't have access to labs/private, etc. [00:39:14] mutante: so I need to wait for Ryan to figure out where to put that [00:39:22] mutante: also I don't think I should have root access if we have a cert there [00:39:26] so I need to figure out how to do that [00:39:52] nod.. yep yep [00:40:38] mutante: so i think what we need is 1. get a cert, 2. add install_certificate to the *Role* (not the class), and set the key paths. [00:40:39] and that's it [00:44:22] yea, sounds like it. it's just how we deal with access to the key [00:49:57] YuviPanda: soo, here's the thing:) just talked to Ryan [00:50:30] YuviPanda: puppetize putting the cert on the instance, but NOT the key, and we will not put it in labs/private [00:50:58] we'll just put the key on the instance then and you can have root or not ,up to you [00:51:52] putting it in labs/private or even another labs/reallyprivate wouldn't be secure [00:52:11] out for dinner [02:51:56] Coren, hihi [03:06:06] http://bots.wmflabs.org/~wm-bot/dump/%23wikimedia-labs.htm [03:59:57] [bz] (8NEW - created by: 2Sarah Stierch, priority: 4Unprioritized - 6major) [Bug 54009] Magnus Treeviews not processing - https://bugzilla.wikimedia.org/show_bug.cgi?id=54009 [05:29:36] Hello all! I created a LAMP instance as per https://wikitech.wikimedia.org/wiki/Help:Lamp_Instance [05:30:28] Now I want to add a PHP file, but unfortunately I have no permission to create a file in /var/www [05:30:35] How to do? [05:42:17] oh, I have sudo, problem solved :-) [05:50:30] Now http://i-000008c1.pmtpa-proxy.wmflabs.org shows "The requested URL / was not found" :-/ Any idea what is going wrong? [05:56:47] Check the logs - tail -f /var/log/apache2/error.log [05:57:04] Then add a section to taht document called troubleshooting [06:29:56] Damianz: nothing suspicious in the log, just "File does not exist: /var/www/favicon.ico" [06:37:21] Damianz: I just created https://wikitech.wikimedia.org/wiki/Help:LAMP_issues with a few tips based on problems I encountered, and linked it from the "Troubleshooting" section. [08:21:49] @notify hashar [08:21:49] I'll let you know when I see hashar around here [08:48:01] My new LAMP instance http://i-000008c1.pmtpa-proxy.wmflabs.org shows "The requested URL / was not found" :-/ Any idea what is going wrong? [09:19:31] hello guys, I have problems with creating a new tool in Tool Labs [09:19:57] I already created a new tool "pb", but it is neihter listed in http://tools.wmflabs.org/ and nor does this work on tools-login: `--> become pb become: no such tool 'pb' .......... --> ls /data/project/pb ls: cannot access /data/project/pb: No such file or directory [09:20:10] < sorry line breaks are missing [09:20:24] second strage thing is, when I go to https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProject&action=addservicegroup&projectname=tools to (re)create it now, I can't see the old formular for new Service Group creation, just this: https://dl.dropboxusercontent.com/u/290221/new%20tool.PNG [09:20:32] can someone take a look on that? [09:27:02] @replag [09:27:03] Replication lag is approximately 46 seconds [09:27:07] :o [09:27:19] lmao [09:27:22] need to fix [09:31:19] Coren: ping [09:32:53] @replag [09:32:57] Replication lag is approximately 00:00:01.4362090 [09:33:04] better [09:33:07] @replag [09:33:07] Replication lag is approximately 00:00:01.6544030 [09:33:49] @replag [09:33:54] Replication lag is approximately 00:00:01.7826850 [09:36:32] Euku: what's name of teh tool? [09:38:10] https://wikitech.wikimedia.org/wiki/Special:NovaServiceGroup [09:39:03] !log tools petrb: started toolwatcher [09:39:06] Logged the message, Master [09:40:05] @replag [09:40:06] Replication lag is approximately 00:00:01.2117310 [09:44:15] ^ping [09:45:36] ^restart [09:45:55] petan: "local-pb" [09:46:01] Euku: try now [09:46:04] ^ping [09:46:07] @replag [09:46:07] Replication lag is approximately 00:00:00.7677590 [09:48:43] petan: thanx, it works [11:55:57] @replag [11:55:57] Replication lag is approximately 00:00:00.4078720 [12:15:42] I created a LAMP instance at http://i-000008c1.pmtpa-proxy.wmflabs.org but it shows "The requested URL / was not found" any idea? [13:24:20] I solved the problem by recreating a new instance [13:25:03] Petan: PING [13:27:01] !petan:ping [13:27:02] don't say petan: ping EVER!! If you need anything, say petan: , saying just "ping" is totaly useless [13:31:47] !petan:PING [13:31:54] how about this? :P [13:32:02] eeek...... no reply? [13:32:26] !petan [13:32:26] Petr Bena - http://enwp.org/User:Petrb (hates python) :D [14:05:33] why cant I access the webservers via tools ? [14:06:51] Betacommand: witch tool? [14:06:58] via ssh? [14:07:45] Steinsplitter: ive got a script that reads an HTML file in my webspace that im running via commandline [14:08:18] Im getting: [14:08:22] urllib2.URLError: [14:09:02] Betacommand: but you can login? [14:09:10] Steinsplitter: yeah [14:09:26] Im trying to run the script via command line on -login [14:10:21] The page loads fine in my webbrowser on my local machine [14:10:36] i dos nok know it is allowed to take content via urllib from external [14:10:59] Steinsplitter: it should [14:11:13] its not really external [14:11:19] \'http://tools.wmflabs.org/betacommand-dev/reports/pages_with_excessive_nfcc.html [14:11:24] http://tools.wmflabs.org/betacommand-dev/reports/pages_with_excessive_nfcc.html [14:11:46] Im trying to load that page [14:12:05] i dos not know, i am not specalist [14:12:12] @replag [14:12:13] Replication lag is approximately 00:00:00.7136990 [14:12:24] petan: any ideas? [14:12:35] regarding what [14:13:03] why my script cant access the above URL? [14:13:07] you mean accessing directly from within tools? [14:13:12] yeah [14:13:33] that is a bug, there is a workaround for that... on -dev it's possible because I hardcoded the dns to /etc/hosts [14:13:56] * Betacommand grumbles [14:14:01] but Coren that is not a proper way to do that, because proper way is to fix the bug in nova, so we are waiting for that to happen (I can't fix Nova I have no access to these servers) [14:14:25] petan: who has access to fix the bug? [14:14:30] "proper way" usually also means "have to wait few months / years" :P [14:14:34] in wmf language [14:14:36] anyway... [14:14:41] Betacommand: wmf ops [14:14:56] the workaround is to provide the DNS / ip of webproxy [14:15:02] instead of tools.wmflabs.org [14:15:07] you just use tools-webproxy [14:15:10] or something like that [14:15:35] petan: this is starting to piss me off, why are so many things still fucked up? [14:15:57] because there is a low number of people who have access to fix them, and that low number of people is busy [14:16:24] petan: so why dont we resolve that? [14:16:26] it's pretty hard to get access to wmf systems... they don't like to let outsiders in :P [14:16:41] Betacommand: because I simply don't have the access to resolve that? [14:16:54] petan: who is the CTO ? [14:16:59] CTO? [14:17:14] @link [[en:CTO]] [14:17:14] http://enwp.org/CTO [14:17:25] https://en.wikipedia.org/wiki/Chief_technology_officer [14:17:36] no idea [14:17:40] look at wmf pages [14:17:58] * Betacommand is going to go wrote a email [14:18:40] have fun... [14:19:00] petan: me on the war path is not a fun sight [14:19:53] btw I think they will reply to you that having more people working on this would mean having to hire more people, which means more money and they don't have that. in fact it's more about letting some volunteers in, but... that's not popular solution [14:21:15] petan: I know this is going to sound lame, but the TS has two paid server admins, and there are a hell of a lot few problems there [14:24:16] well, toolserver was on other hand configured in much more lame way... labs are using technologies so complicated (puppet in combination of gerrit and git) that setting stuff up on labs taking incredible amount of time (despite once configured the setup of new instances with same configuration is a matter of minutes) [14:24:53] toolserver consisted of few servers, each of them configured by hand (same as old bots project on labs was) [14:25:13] petan: KISS [14:25:45] that is something very easy to maintain... but that was not "proper solution" ™ by WMF [14:26:52] petan: https://en.wikipedia.org/wiki/KISS_principle is codified IT law [14:27:32] any management who knowingly violates it needs to be given the boot [14:27:35] Betacommand: I kind of endorse that, but my project that was simple and worked, was shut down forcefully... because it wasn't proper ™ [14:28:03] some people here prefer complicated solutions [14:28:14] petan: proper is relative, if it doesn't work, its not proper [14:28:39] in theory it's proper. that's what I mean [14:28:49] petan: If they do they have no business in IT [14:50:31] petan: email sent [15:09:53] Betacommand: omgf i spent 3 days debugging why i couldnt access tools.wmflabs.org internally [15:09:57] omfg* [15:10:10] legoktm: you could just ask here :P [15:10:17] i did! [15:10:21] o.O [15:10:26] nobody replied? [15:10:43] it just got lost. after 3 days someone responded and we figured it out [15:13:16] legoktm: see my email rant [15:13:26] i did [15:13:30] :) [15:13:38] what did you think? [15:16:44] it was pretty accurate [16:13:15] petan: I am sorry for not highlighting your good work. See my followup [17:28:34] Betacommand: BTW, toolserver uptime of four months? I call bovine poop. :-) [17:32:44] Coren: one sec [17:33:09] betacommand@willow:~$ uptime [17:33:11] 17:33pm up 123 days 21:25, 12 users, load average: 1.54, 1.83, 2.05 [17:33:26] [bz] (8ASSIGNED - created by: 2kolossos, priority: 4Unprioritized - 6normal) [Bug 48896] Please install Postgresql on Tool-labs - https://bugzilla.wikimedia.org/show_bug.cgi?id=48896 [17:33:37] Coren: please eat your words [17:37:21] coren both nightshade and wolfsbane have been up 60+ days [17:38:01] That they have not been booted does not mean that they were /usable/; but even then, 60 days is half of four months. [17:38:22] 123 is, though. [17:39:07] Coren: I only cited willow in my emai; [17:39:09] But having the box up while LDAP was down, or the webserver was down, or replication had stalled... hardly qualifies as "working for four months: [17:39:40] hell, I can keep up a bastion for years [17:39:42] Coren: other than a few hickups with cron toolserver has been stable [17:39:47] (Mind you, tool labs has been far from perfect either -- the NFS hardware issues has bit hard lately) [17:40:02] Betacommand: their storage was unavailable for over a week [17:40:05] at one point [17:40:15] their webservers were throwing 500s for weeks [17:40:39] there were numerous threads on village pump about toolserver being down [17:40:50] My point isn't that toolserver is shit and tool labs is the l33t; but that saying that "toolserver was always up and reliable" is hogwash. [17:40:57] indeed [17:41:10] we've had stability issues, but they are far less than TS [17:41:37] Coren: functionality of tools vs ts Is is about 80 while labs is 50+60 [17:41:42] !search dynamicproxy [17:41:42] http://bots.wmflabs.org/~wm-bot/searchlog/index.php?action=search&channel=%23wikimedia-labs [17:41:53] Betacommand: what does that mean? [17:42:20] !project dynamicproxy [17:42:20] There are multiple keys, refine your input: project-access, project-discuss, projects, [17:42:46] !projects dynamic [17:42:46] https://labsconsole.wikimedia.org/w/index.php?title=Special:Ask&q=[[Resource+Type%3A%3Aproject]]&p=format%3Dbroadtable%2Fheaders%3Dshow%2Flink%3Dall%2Fsearchlabel%3D%E2%80%A6-20further-20results%2Fclass%3Dsortable-20wikitable-20smwtable&po=%3FMember%0A%3FDescription%0A&limit=500&eq=no [17:43:15] Ryan_Lane: story didnt finish my thought its out of 100 [17:43:34] !help [17:43:34] !documentation for labs !wm-bot for bot [17:43:55] !wm-bot [17:43:56] http://meta.wikimedia.org/wiki/WM-Bot [17:44:02] on meta.. ah [17:44:47] Betacommand: oh, good we're 110% [17:44:54] I'll take that [17:45:20] Ryan_Lane: typo should have been 50-60 [17:45:31] what do you base this claim on? [17:45:38] @labs-project-instances dynamicproxy [17:45:38] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 32 seconds ago [17:46:23] Betacommand: That's completely off. Since we "opened", we had ~90% uptime or so, and most of the bigger outages in the first month. [17:46:36] In fact, last night's fuck was the only unplanned outage in the past two months or so. [17:46:59] And the only reason it lasted that long was a communication problem I made sure wouldn't happen again. [17:47:20] Ryan_Lane: several issues, repeated large outages of NFS, issues with DNS, and screwy web servers, and other issues [17:48:17] Betacommand: there have been no issues with DNS [17:48:23] not for over 2 years [17:49:05] ah, you mean the NAT issues [17:49:23] (that's not DNS and there's ways to workaround that issue currently) [17:49:58] you can connect to the instance's private DNS or IP and send a host header for tools.wmflabs.org [17:50:41] oh, it seems you don't even need to use a host header right now, but it's good practice [17:57:34] When will deleted edits be on the edit counter? [17:58:00] @seen Cyberpower678 [17:58:01] T13|needsCoffee: Last time I saw Cyberpower678 they were quitting the network with reason: Ping timeout: 276 seconds N/A at 9/11/2013 4:04:57 PM (1h53m3s ago) [18:00:20] YuviPanda|away: this is why i couldnt find "dynamicproxy" as Jan pointed out :) https://gerrit.wikimedia.org/r/#/c/83127/ [18:00:49] @labs-project-instances labsproxy [18:00:50] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 43 seconds ago [18:01:04] err :p [18:04:24] Ryan_Lane: I should be able to use the external URL and have it work [18:05:27] Betacommand: I don't disagree [18:10:13] hey folks! we're doing some network maintenance in tampa - i've taken precautions and am reasonably certain no network disruption should happen, but please let me know (and accept my apologies) if it does [18:12:19] LeslieCarr: Way to jinx yourself. :-) [18:12:33] :) [18:12:47] well since your'e in SF today, you can come over and throw things at me in person if it does [18:19:17] if(preg_match('/cr[0-9]/',$log_message) echo "weee. she's on the core router" [18:19:55] mutante: hmm? [18:19:59] mutante: ahh [18:20:03] mutante: yeah, unmerged yes [18:20:07] YuviPanda: i can't find your project name [18:20:13] mutante: proxy-project [18:20:17] mutante: or project-proxy, not sure. [18:20:20] arr, i thought labsproxy [18:20:23] mutante: labsproxy is just the name of the module [18:20:28] ah:p [18:20:34] mutante: this is the same project that was running instanceproxy before [18:20:46] @labs-project-instances proxy-project [18:20:47] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 43 seconds ago [18:20:55] @labs-project-instances project-proxy [18:20:55] Following instances are in this project: instance-proxy, pmtpa-proxy, proxy-pure-2, proxy-project-proxy, labsproxy, proxy-dammit, [18:21:03] got it:) [18:21:11] @labs-project-info proxy-dammit [18:21:11] I don't know this project, sorry, try browsing the list by hand, but I can guarantee there is no such project matching this name unless it has been created less than 7 seconds ago [18:21:26] @labs-info proxy-dammit [18:21:26] [Name proxy-dammit doesn't exist but resolves to I-000008a8] I-000008a8 is Nova Instance with name: proxy-dammit, host: virt7, IP: 10.4.0.214 of type: m1.small, with number of CPUs: 1, RAM of this size: 2048M, member of project: project-proxy, size of storage: 35 and with image ID: ubuntu-12.04-precise [18:21:31] jzerebecki: ^ [18:22:17] @labs-info proxy-project-proxy [18:22:17] [Name proxy-project-proxy doesn't exist but resolves to I-000008a3] I-000008a3 is Nova Instance with name: proxy-project-proxy, host: virt11, IP: 10.4.0.60 of type: m1.small, with number of CPUs: 1, RAM of this size: 2048M, member of project: project-proxy, size of storage: 35 and with image ID: ubuntu-12.04-precise [18:22:39] hmm, should those resolve? [18:22:53] mutante: the box name is proxy-dammit [18:23:10] proxy-project-proxy is my test box that i use for testing puppet patches [18:23:34] yep, i remembered "dammit" :) [18:23:56] YuviPanda: are you setting up individual host names for each of these proxied wikis? [18:24:12] Ryan_Lane: I'm dyanmically setting Host: headers [18:24:25] how are the host names resolving? [18:24:36] Ryan_Lane: I put a 'resolver' clause there. probably the problem? [18:24:38] hardcoded IP [18:24:41] to a DNS server [18:25:02] resolver 10.4.0.1; [18:25:18] mutante: Ryan_Lane ^ [18:32:54] YuviPanda: Ryan_Lane oh, it looks like it may already be done [18:32:59] 81 define install_certificate( $group="ssl-cert", $ca="", $privatekey="true" ) { [18:33:06] if ( $privatekey == "false" ) { [18:33:23] $key_loc = "puppet:///files/ssl/${name}" } else { $key_loc = "puppet:///private/ssl/${name}" [18:43:38] new function: [18:43:40] @replag [18:43:41] Replication lag is approximately 00:00:00.9606510 [18:43:55] however I would be happy if there was a reliable way to check it [18:44:07] this thing just select latest RC timestamp [18:44:15] and compare it with DateTime.Now [18:45:44] @ping [18:45:44] Pinging all local filesystems, hold on [18:45:45] Written and deleted 4 bytes on /tmp in 00:00:00.0007520 [18:45:46] Written and deleted 4 bytes on /data/project in 00:00:00.0106500 [18:51:08] YuviPanda: would you like me to merge https://gerrit.wikimedia.org/r/#/c/83127/2 ? [18:53:11] andrewbogott:hey! [18:53:15] andrewbogott: not yet, haven't tested it yet [18:53:21] andrewbogott: i mean, tested it by hard coding [18:53:24] 'k [18:53:35] andrewbogott: but not on puppet [18:53:37] can merge in a bit but [18:55:18] https://gerrit.wikimedia.org/r/#/c/83843/1 [19:04:29] mutante, can you tell me more abou "the files/ssl part doesn't exist anymore" [19:27:28] petan, did labs just restart? [19:27:41] no [19:27:50] wm-bot didn't even notice anything [19:28:01] So what caused every script on my node to restart? [19:28:05] @replag [19:28:05] Replication lag is approximately 00:00:00.2917910 [19:28:49] Cyberpower678: no idea... [19:29:01] Cyberpower678: you need to implement some logs [19:29:18] I have logs. But I can't access them now. [19:29:31] that aren't useful logs then [19:29:47] Well I can't really SSH into labs right now. [19:30:04] * Cyberpower678 really needs to develop a web log tool. [19:31:08] I see on Ganglia that memory usage on Cyberbot has dropped off and the scripts are now recovering from the crash. [19:31:18] hmm [19:31:31] andrewbogott: ping? [19:32:07] petan, as in 1 GB of memory usage just vanished. [19:39:32] [bz] (8NEW - created by: 2Nemo, priority: 4Unprioritized - 6major) [Bug 53987] sulinfo is unusable (takes tens of seconds) - https://bugzilla.wikimedia.org/show_bug.cgi?id=53987 [19:41:56] YuviPanda: what's up? [19:42:20] andrewbogott: hey! [19:42:22] andrewbogott: so, https://gerrit.wikimedia.org/r/#/c/83773/ works [19:42:36] andrewbogott: but, i need to enter my passpharse by hand whenever restarting nginx [19:42:39] is that expected? [19:43:39] It is if your key requires a passphrase... [19:44:04] hmm, the openssl commands i used didn't let me create a key without a passphrase [19:44:06] but I guess that's okay [19:44:25] Oh, that surprises me… maybe a good Ryan question (when he returns) [19:44:28] yeah [19:44:45] andrewbogott: also, I'm trying to remove the hostname blue-dragon.wmflabs.org from proxy-dammit instance [19:44:52] andrewbogott: and unable to do so - says 'no such hostname' [19:45:16] remove via wikitech you mean? [19:45:17] andrewbogott: perhaps because I added it to another instance (proxy-project-proxy) before removing it. [19:45:18] andrewbogott: yeah [19:45:26] * andrewbogott looks [19:46:08] hm so I see [19:47:03] YuviPanda: What openssl commands did you use? [19:47:08] I just added faketesthostname and it can't be removed either, same failure [19:47:25] So it thinks that proxy-dammit doesn't exist :/ [19:47:35] anomie: http://wiki.nginx.org/HttpSslModule#Generate_Certificates [19:47:39] andrewbogott: ow :( [19:47:41] that's bad [19:48:05] I cannot explain [19:48:08] YuviPanda: Did you use the version or openssl genrsa with -des or without? [19:48:12] anomie: I was being lame and just copy pasting those without reading what is happening. [19:48:31] anomie: ow, I see it now [19:48:33] I'm an idiot [19:48:43] anomie: got it now, thanks ;) [19:49:25] anomie: works now, thanks! [19:49:27] woohoo! [19:49:53] andrewbogott: can you poke me when Ryan_Lane or mutante come back? [19:50:01] yep [19:50:16] andrewbogott: can things be removed from other hosts? [19:51:13] Nope. Looks 100% broken. I will investigate. [20:04:47] hey mutante [20:05:13] andrewbogott: Ryan told me in RL(tm) [20:05:26] ok, fair enough :) [20:05:45] anomie: so my puppet patch for ssl works. Just needs to get a certificate, and then do the install stuff. [20:06:04] andrewbogott: the reason for doing it is so that yuvi can use install_certificate for his proxy [20:08:35] Don't suppose there's any chance of tools getting a rabbit node quickly? [20:08:59] why? [20:09:09] you need that much reliability for message delivery? [20:09:51] Well no, but I do want to be able to get messages sent while I'm not connected (which you can't do with pub/sub in redis) [20:10:00] Damianz: well, don't use pub/sub [20:10:10] Damianz: use lpush and rbpop [20:10:20] gets you messages sent when not connected [20:10:27] that's what grrrit-wm does [20:10:31] hmmm [20:10:35]