[00:00:06] iSCSI taget framework, centos6. [00:00:27] ah [00:00:37] Though I'm kinda leaning towards just using ubuntu server and getting a cluster up for internal stuff over trying to test the rpms in epel-testing as the ubuntu ones seem kinda stable and supported. [00:00:37] I'm not using iscsi [00:00:46] why not just use gluster? [00:00:56] already have a reliable iscsi server? [00:01:24] I probably would have used iscsi if our netapp didn't want to charge us insane amounts of money to use it [00:01:35] I probably will use gluster as we're kinda thinking about dtiching the sans. We use local storage for vms with off site iscsi mounts for some people currently. [00:01:41] * Ryan_Lane nods [00:01:45] it'll be slower, likely [00:01:55] cheaper, though [00:02:00] Yeah we have some nice dell sans that cost about 60k and are a beast to admin :( [00:02:20] eww. dell ones? [00:02:27] how's the failover for those? [00:02:51] I don't think they are setup for failover currently as well... when they where the failover failed. [00:03:03] * Ryan_Lane twitches [00:03:06] For sans I'm more of a DRDB+pacemaker fan. [00:03:10] than gluster may actually be more reliable [00:03:13] But I'd rather not tough iscsi with a broom. [00:03:59] I'm just worried about losing more than two gluster nodes and having a partial outage on whichever instances only exist on those two nodes [00:04:09] Currently playing with gluster for some storage, not 100% sure on vms but I'd rather not have a big probablySPOF in regards to a san. [00:04:18] though, if two nodes go down, then we lost 60+ instances right there anyway [00:04:31] and with the new hardware 200+ instances per node [00:04:32] iSCSI mounts do weird things when they loose network access or the network lags out =/ [00:04:37] yep [00:05:09] we're splitting our IO [00:05:43] one for space, another to try to divide the IO into OS and application specific traffic [00:05:43] The only thing I have about gluster is if a node is down then comes back up, re-stating the files to sync it is quite extensive and there is no simple way to check the sync status (I don't believe) so I couldn't say ah yeah if we loose 2 nodes right now the remainings ones are only 70% in sync. [00:06:02] it's easy to make it re-sync [00:06:09] with the newer version of cluster [00:06:13] same to make it rebalance [00:06:20] it's a single command [00:06:33] Oooh really [00:06:36] yeah [00:06:45] I added a node and rebalanced it in a couple hours [00:06:49] I've not properly tried gluster in a while but it use to be a case of find /data and :( [00:06:55] well, the command took a couple hours to run [00:07:07] But then it was also a huge manual re-specifying replicaes on new nodes. [00:07:10] I think the find is still necessary [00:07:33] but that usually takes no time at all [00:07:45] rebalances take a while, but that makes sense, since it needs to transfer data [00:07:48] Does the cluster use server side replication or rely on the clients to do it? [00:08:01] rebalances are server side [00:08:07] the find requires a client [00:08:16] so, you mount the volumes on the server and do a find [00:08:52] that said, when a node comes back up, if a client tries to access a file, it'll be sync'd then [00:09:02] Yeah [00:09:03] which means requests for those files will be slow [00:09:24] easiest to just do it when the node comes up [00:09:27] you can script it [00:09:30] My concern is when you get into the issues on loosing 2 nodes and suddenly you have your redundant part out of sync. Then you loose something and bam weird data loss issues. [00:09:47] and even with hundreds of volumes, a find shouldn't take ages, since it just needs to scan metadata [00:09:49] But it can be worked around pretty easily. [00:10:03] well, you'd have to fully lose those two nodes [00:10:14] I'm using raid underneath gluster [00:10:29] two raid 6 arrays LVM'd together, then shared via gluster [00:10:36] it's lossy, but safer [00:11:08] on the instance storage, every node has a raid 10, shared via gluster [00:11:09] Any reason for raid6 over say raid 10 and just bang a big zfs or xfs fs on there? Though zfs still ins't support in linux I don't think which is a shame as its snapshotting is pretty cool. [00:11:23] raid 6 for volume storage, and raid 10 for instance storage [00:11:39] raid 6 because raid 10 costs too much for volume storage [00:11:53] it eats *way* too many disks [00:12:10] raid6 is raid5 with an extra spare, right? [00:12:13] yeah [00:12:23] I worry about a 12 disk raid 5 [00:12:31] Yeah... [00:12:47] raid 6 is slower, but it makes me sleep better at night :) [00:13:12] One of our raid6 sans blew a disk the other day, raid5 would have caused arghness. [00:13:22] yeah [00:13:39] I honestly don't trust raid 5 anymore, with these much larger disks [00:14:06] the chance of getting a silent write error is basically guaranteed with raid 5 [00:16:16] Raid 5 takes long enough to rebuild with like a 300gb disks, wouldn't even try it with a 2tb. Thats like building something over 2tb on ext3 then hitting a 2month fsck xD [00:44:27] ssmollett: your change went through everywhere and is working properly :) [01:03:04] !log deployment-prep reconfiguring the web server instances to remove puppet classes that no longer exist [01:03:07] Logged the message, Master [01:33:23] * Damianz gives ssmollett a cookie [01:33:30] Also I FUCKING hate dc people [02:24:03] Damianz: hahaha. why's that? [02:27:21] 02/22/2012 - 02:27:21 - Creating a project directory for orgcharts [02:37:25] 02/22/2012 - 02:37:24 - Creating a home directory for marktraceur at /export/home/orgcharts/marktraceur [02:38:11] 02/22/2012 - 02:38:10 - Creating a home directory for marktraceur at /export/home/bastion/marktraceur [02:38:23] 02/22/2012 - 02:38:22 - Updating keys for marktraceur [02:39:10] 02/22/2012 - 02:39:10 - Updating keys for marktraceur [09:44:33] I'd like to add a jdk and tomcat appserver to the puppet config menu how do I proceed [09:47:46] Hey, gadgets aren't working at hi-wp deployment [09:47:56] anyone know why? [09:48:39] Change on 12mediawiki a page Wikimedia Labs was modified, changed by OrenBochman link https://www.mediawiki.org/w/index.php?diff=502555 edit summary: /* Proposals */ [09:48:49] *mean at http://hi.wikipedia.beta.wmflabs.org ofcourse [09:58:44] anyone? [10:06:21] anyone here? [10:07:45] anyone? [13:45:43] Why aren't gadgets working at the test site? [13:46:07] ? [13:46:20] Sid-G: could you be more specific? which test site? [13:46:46] sumanah: I'm not getting gadgets at hi-wp labs [13:46:54] at http://hi.wikipedia.beta.wmflabs.org [13:47:34] sumanah:No gadgets tab in the prefs anymore (it was there yesterday) [13:47:43] That is strange! [13:48:24] sumanah: Yeah, and the gadgets I had enabled dont show up anymore either [13:48:47] The script doesn't load (checked with the chrome console) [13:48:54] Sid-G: let's talk about this in #wikimedia-dev as I think some folks will be there who care about this.... [13:49:44] ok [14:41:10] 02/22/2012 - 14:41:10 - Updating keys for marktraceur [14:41:22] 02/22/2012 - 14:41:21 - Updating keys for marktraceur [15:05:08] 02/22/2012 - 15:05:07 - Updating keys for hydriz [15:05:17] 02/22/2012 - 15:05:17 - Updating keys for hydriz [18:13:45] hi Ryan_Lane do you have a couple minutes for a labs question? I'm trying to ssh to my labs instance from bastion and failing. [18:13:53] sure [18:13:57] are you forwarding your key? [18:14:01] which instance? [18:14:07] *forwarding your agent? [18:14:46] from bastion can you ssh to bastion1? [18:15:00] (it's the same system, but lets me know if forwarding is working correctly. [18:15:12] s/\./)/ heh [18:15:48] I have an instance named "firstinstance". I ssh to bastion just fine. according to https://labsconsole.wikimedia.org/wiki/Access, it seems I should be able to ssh from bastion to "firstinstance" but I get "Permission denied (publickey)." [18:16:33] actually bastion1, I'm "cmcmahon@bastion1" at the shell prompt [18:18:08] <^demon|away> Ryan_Lane: I think my gitweb redirect was wrong. https://gerrit.wikimedia.org/gitweb/operations/puppet.git gives a 404, and just /gitweb/ exposes the perl script :p [18:20:05] yeah [18:20:25] chrismcmahon: yeah, but try sshing to bastion1 from bastion [18:20:37] it lets me know your agent is forwarding properly [18:21:21] Ryan_Lane: I do "ssh -A cmcmahon@bastion.wmflabs.org" and end up with a shell prompt on bastion1 [18:21:28] ..... [18:21:38] now ssh to bastion1, from bastion1 [18:21:57] if your agent is forwarded properly it'll work, otherwise it wont [18:22:22] which project is this instance in? [18:22:31] weird. "cmcmahon@bastion1:~$ ssh bastion1" "Permission denied (publickey)" [18:22:57] type: ssh-add -l [18:23:15] does it show a key? [18:23:16] my "firstinstance" instance is in project "quality-assurance" [18:23:27] ok. gimme a sec, I need to add myself to it [18:23:39] $ ssh-add -l [18:23:39] The agent has no identities. [18:23:44] actually, no I don't [18:23:51] you're instance works properly [18:24:00] hi Ryan_Lane [18:24:04] you need to get agent forwarding working properly [18:24:15] chrismcmahon: https://labsconsole.wikimedia.org/wiki/Help:Access#Using_agent_forwarding [18:24:22] koolhead12: howdy [18:24:24] * Damianz stretches out [18:24:36] Ryan_Lane, am good. 1 more to go. :) [18:24:52] PHP + file permissions = sad person [18:24:53] chrismcmahon: this is a common problem ;) [18:24:59] Damianz: heh [18:25:03] koolhead12: 1 more to go? [18:25:34] Ryan_Lane, multi node automation of diablo :) [18:25:39] ahhhh ok [18:27:17] although the isolating the nova-network from API is still not working for me. have to dig deeper inside :) [18:27:36] yeah, I had to put the api on the network node [18:28:04] Ryan_Lane: seems to me that "ssh -A" should just dtrt. not sure where the breakdown is. [18:28:59] otherwise there's a need for some awkward networking rules [18:29:12] I think in essex they are moving the metadata server to the compute service [18:29:24] then the API can be separate [18:29:37] chrismcmahon: on your local system: ssh-add -l [18:29:49] you want to ensure your agent has it locally, then also has it on bastion [18:29:57] Isn't essex due for release soon? [18:29:58] you don't want to forward for agent past bastion, in general [18:30:06] Damianz: yeah, in weeks, likely [18:30:11] Damianz, yes in few months!! :) [18:30:14] I need to defer a bug of mine :( [18:30:45] I'm wondering how smooth the update path is going to be. [18:30:56] no clue. I had a terrible, terrible time for diablo [18:31:06] but I also kind of fucked myself with facebook mysql libraries [18:31:22] lol [18:31:28] I had an outage because of that [18:31:30] it wasn't pleasant [18:31:34] Damianz, if your using keystone u will be safe [18:31:38] I really hope they don't get stuck in a 'you have to update all nodes at once' place [18:31:42] else you have to take same route [18:31:55] apparently openstack dislikes changing the default charset [18:32:01] or, well, sqlalchemy does [18:32:05] ha. got it. Ryan_Lane I didn't realize "eval 'ssh-agent'" wasn't a permanent condition. I'm on my instance now. [18:32:27] chrismcmahon: ah, yeah, whenever you open a new shell your environment won't have the agent [18:32:38] you can re-connect to your agent in other shells, though [18:32:49] btw Damianz hellos :) [18:32:59] I have scripts that write the environment variables out to a file, then I source the file in other shells [18:33:01] * Damianz waves at koolhead12 [18:33:11] Ryan_Lane: been some time since I've been deep in ssh, it's coming back to me, thanks [18:33:13] I run two agents, one for production and another for labs [18:33:15] Damianz, are you also a stacker :) [18:33:24] good to see openstack people in the channel :) [18:33:24] * koolhead12 is still clueless of swift [18:33:32] * Ryan_Lane is also mostly clueless of swift [18:33:41] I need to learn it soon enough, though [18:33:43] I'm more confused about switft... [18:33:47] hahaha [18:33:53] Damianz: thanks for the cookie; glad the ldap library change worked. [18:34:04] ben will be out of town for the openstack conference. I may need to give the swift talk for us. [18:35:15] my next week target is to start swimming inside it --> swift [18:35:30] Ryan_Lane, and how will one loadbalance nova-api? [18:35:38] by keeping multiple zones [18:35:44] ssmollett: a few instances still need puppet fixed and run [18:35:55] I've been doing that. it's only about 5-6 more right now [18:36:04] this is a good way of ensuring systems are running puppet properly :) [18:37:09] is it worth having monitoring tell us if puppet hasn't run recently or has run but failed? [18:37:18] yes, definitely [18:37:28] we do that in production using an snmptrap [18:37:36] Bah stupid home internets [18:37:40] if you could add that to the nagios instance, that would rock [18:37:53] it probably just needs to be slightly modified in puppet to work [18:38:29] o.0 [18:38:31] * Damianz doesn't trust snmp traps to be reliable enough [18:38:53] Damianz: well, if the snmptrap isn't working, then all nodes show puppet as not running [18:39:05] then we know the snmptrap is broken [18:39:27] Ah, so you have the thingy option on the pasive check that if it isn't updated in xmin nagios moans? [18:39:41] I believe that's how it works [18:39:51] I didn't set up the trap [18:41:54] Hmm I really should fix my nagios box, was kinda leaning towards having puppet manage it though over my lovely mysql+perl solution *shudder* [18:42:17] New patchset: Sara; "First iteration of adding ganglia for labs." [operations/puppet] (test) - https://gerrit.wikimedia.org/r/2157 [18:42:36] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/2157 [18:44:00] Ryan_Lane: those ganglia changes are all minor. i still need to work out the gmond stuff. maybe we can sit down and figure that out friday? [18:44:16] Damianz: it's going to suck. puppet managament of nagios is really, really painful [18:44:22] ssmollett: sure. [18:44:48] Hmm [18:46:17] ssmollett: basically, you'll want to change the cluster name to the project, the udp_send_channel to use unicast, rather than multicast, where the address is that of the ganglia instance [18:46:29] and then set the port by the gid [18:46:52] I think alll of the "accept" stuff isn't needed, since none of the instances will be acting as aggregators [18:47:04] accept/recv [18:47:41] ... [18:47:48] I can put ubuntu on a driod o.0 [18:47:56] but do you *want* to? :D [18:48:09] Nope, I have an iPhone :P [18:48:21] heh [18:48:22] Though I do want an andriod. [18:48:48] I may eventually get one. I'm not buying a new phone for a while, though [18:49:02] but I want a non-skinned android, that isn't fucking massive [18:49:16] all of these tablet sized phones can die in a fire [18:49:35] I mainly use my phone for music and email so kinda don't want to pay for a new one. [18:50:39] hm. how am I going to sanely manage the project storage till the volume driver is ready? [18:51:05] this is gonna need caching of come variety. heh [18:51:08] Free for all! [18:51:34] well… the problem is that the number of directories is based on the number of users and projects [18:51:51] though, I guess it's based on users in a project, so it's not as bad as I'm making it out [18:52:07] I guess I'll just modify my home directory script for now [18:52:33] it creates an exports file right now. I need to change it to manage gluster volumes instead [18:53:15] I also need to work out a directory structure for volumes [18:53:35] andrewbogott_afk: I should likely coordinate this with you, so that our structures match [18:54:18] It will get more complex when you take into account projects having blocks of storage that arn't related to users. [18:54:34] nah, that'll be quota'd by project [18:54:50] right now it's volume/project [18:55:08] so, home/bastion/laner, home/bastion/damianz, etc. [18:55:49] but, the gluster volume driver would manage home/bastion, home/mail, home/feeds, etc. and a script will add the home directories underneath that volume [18:56:05] even the home directories per project will have a project specific quota, not per-user [18:56:08] it makes things easier [18:56:45] I'll make the script add a default quota, and I can manually adjust them for now [18:56:57] Makes sense [18:57:11] though I could always do project/volume, too [18:57:21] not sure which is easier [18:57:41] I think likely volume/project, since it's what I'm already doing :) [18:57:44] Puppet modules that come with install instructions past 'put it in the modules dir' confuse me :( [18:57:49] heh [18:58:03] well, some properly require server side things [18:58:09] we have a few puppet things that do [18:58:47] For some reason this module actually comes with instructions of how to install puppet... [18:59:02] hahaha [18:59:54] * koolhead12 is scared of Ruby [19:00:21] I wish cfengine was more highlevelish [19:00:42] It's just a more reliable way of what I do now and writing bash scripts to validate stuff other bash scripts did just ends in pain [19:00:43] ruby is easy enough, it's just managed poorly [19:00:47] Damianz, don`t we have a configuration management system in python [19:00:56] unfortunately, no [19:00:58] they are all ruby [19:01:03] That would be nice [19:01:07] cfengine sucks [19:01:15] well, it's outdated, really [19:01:24] Ryan_Lane, it was pain for me to install it on my ubuntu box :P [19:01:33] Ubuntu is easy... [19:01:38] Damianz, i still prefer old bash for my deployments [19:01:40] You should try making puppetmaster work on c5 [19:01:57] Damianz, yeah. puppet is easily configured in Ubuntu [19:02:03] bash is evil [19:02:04] Random bash scripts don't work when you have 500 servers that might be up/down/lagged out etc that you'd rather be in one state. [19:02:20] I rarely write bash scripts anymore [19:02:39] Ryan_Lane, so using puppet most of your work? [19:02:47] $(perl -e "") :D [19:02:49] almost exclusively [19:02:55] and python scripts for a lot too [19:02:57] Damianz, your scaring me ;P [19:03:01] and adding support to openstack for other things [19:03:39] I really want openstack to be fully extendable. it's not right now :( [19:03:42] I'm stuck with perl for a bunch of stuff due to the python version on centos sucking but python is nice for lager stuff that requires mroe than like 2k lines in one file. [19:03:50] Damianz: are you going to the openstack conference? [19:03:53] koolhead12: or you? [19:04:01] Ryan_Lane, no. am in india :( [19:04:04] ah [19:04:27] Hmm maybe, dunno [19:04:36] i am learning python, still learning :D [19:04:39] have you submitted code to openstack at all? [19:04:45] invites right now are limited to contributors [19:04:48] I'm in the UK so it's a trek but... I also have half my team in the us so I could find an excuse for the flight. [19:04:53] heh [19:04:54] Ryan_Lane: been looking around and not finding: is there some canonical procedure to install Mediawiki on a labs instance? [19:05:05] And nope, I keep meaning to but been busy. [19:05:12] chrismcmahon, juju :P [19:05:19] i can help if needed :) [19:05:21] chrismcmahon: unfortunately it's from scratch until someone makes an all-in-one puppet manifest for it [19:05:24] no juju! [19:05:27] heh [19:05:29] Ryan_Lane, :D [19:05:36] puppet only, please ;) [19:05:40] looks like a lot of yaks from here :) [19:05:51] Ryan_Lane, gosh!! you really want me to learn puppet. ok sir [19:05:52] :) [19:06:00] chrismcmahon, hehe [19:06:58] koolhead12: we have no plans to use anything other than puppet or openstack right now [19:07:03] koolhead12: if you could get me started down the road to install MW on an instance, I am sure it would save me time exploring dead ends. [19:07:25] chrismcmahon: documentation on mediawiki.org for installing on ubuntu should be appropriate [19:07:34] don't install from the mediawiki package, though [19:07:35] chrismcmahon, i can do that. right away if needed [19:07:37] You need docs? [19:07:51] Don't you just like download it and add a mysql db, run the maint script and done? [19:07:56] basically, yes [19:07:59] Damianz, yes [19:08:10] you need to install apache, mysql, php, apc, memcache, and a few other things [19:08:13] memcache isn't necessary [19:08:24] neither is apc, but you'd be crazy to not install apc [19:08:30] and create upload directory with right permission [19:08:30] memcache makes it faster though... not that mw is ever going to be fast. [19:09:04] often we install in /srv for one-off installs, but /mnt may be more appropriate on labs [19:10:13] heh, oren made an instance called dumpster1? :D [19:10:15] I really hate people that have like 25 quotes in email replies... also top posting is confusing [19:10:37] lol [19:10:51] people name instances the strangest things [19:11:07] at least we haven't been hitting naming conflicts yet :) [19:11:32] these docs? http://www.mediawiki.org/wiki/Manual:Installation_requirements http://www.mediawiki.org/wiki/Manual:Installing_MediaWiki [19:11:34] Is that possible? I thought all the references where to the stupid openstack generated hostname. [19:11:52] there's two names [19:11:58] instance name, and instance id [19:12:03] both need to be unique [19:12:12] and mediawiki enforces it [19:12:48] so people can't name two things the same, but they can try [19:12:55] and no one has complained yet. heh [19:13:09] i mean you guys might find it funny but am 4 experimenting with internal infra, i used Bash and 5 nodes, cobbler :P [19:13:28] cobbler is kinda annoying [19:14:14] cobbler is fine for a regular build server [19:14:21] Damianz, indeed. changes from version to version, espacially the whole preseed rule gets screwed [19:14:41] Ryan_Lane, that post_install script is cool. it does most automation [19:14:55] we just use puppet for that too [19:15:01] like injecting public key and stuff [19:15:05] Ryan_Lane, got it :P [19:15:40] Damianz, you also a openstack dev? [19:15:52] I used to do a lot of stuff via post-inst and simple config files, when I used red hat satellite server [19:15:55] nope [19:15:57] puppet is more powerful [19:16:51] hmm i will dig into it soon. [19:17:21] i installed it and passed simple stuff like changing file permission and installing apache :) [19:18:00] I have puppet forcing ntpd on my test servers atm, currently figuring out making it manage openstack though D: Though there is an openstack module in puppetlabs. [19:19:02] our repo has puppet manifests for openstack [19:19:08] I don't think I recommend mine, though ;) [19:19:18] they aren't modules, for one [19:19:42] I'm in 2 minds about modules [19:19:55] I can see why they are good but I'm also taking others peoples ideas of how my configs should look. [19:20:12] Though I guess I could just change the templates and then screw pulling updates from upstream. [19:20:44] yeah [19:20:53] we basically write everything from scratch [19:21:22] Ryan_Lane, any wiki/doc you would suggest for n00b like me who knows no ruby :) [19:21:28] for puppet heads up [19:21:42] I was reading about facter or w/e it's called the other day which looks interesting. Hope I can pull the data out of that to use in manifests and set values in templates based on things like ram in a box. [19:23:17] koolhead12: well, it's a puppet dsl, not ruby [19:23:30] koolhead12: read the tutorial and intro documents for puppet [19:23:33] they are a good start [19:23:44] Damianz: yeah, we use facter quite a bit [19:24:01] Ryan_Lane, factor uses python i suppose :P [19:24:08] Damianz: I'd *really* love to pull the metadata information for instances, then have it available as facter info [19:24:11] does it? [19:24:14] I thought it was also ruby [19:24:34] Facter is ruby I'm pretty sure [19:24:50] it's scripts are defintitely ruby [19:24:58] its* [19:25:00] Instance data would be cool. [19:25:14] You could do like location based puppet things based on which zone it's in. [19:25:29] we do that now, by populating ldap with that info [19:25:50] but we want to eventually move away from using ldap for node info [19:27:44] Hmm [19:28:46] I'd prefer to keep it in the metadata [19:29:12] hmm, no gcc on instances it seems [19:29:24] Is openstacks metadata distributed or on the controller? [19:29:30] apt-get install gcc? [19:29:36] Damianz, Ryan_Lane yes ruby :P [19:29:39] Why do you need gcc though [19:29:47] Compiling stuff = PITA the maintain. [19:29:48] Damianz: installing Apache [19:29:54] apt-get install apache2? [19:29:55] Damianz: it's on the API server right now [19:30:08] Damianz: I think in essex it's on every compute node [19:30:22] the data is stored in the database, thouguh [19:30:23] *though [19:30:44] chrismcmahon: no. installs are base, on purpose :) [19:30:54] there are puppet classes for build systems, though [19:31:05] you can add that by reconfiguring the instance, then force running puppet [19:31:16] some puppet classes may cause puppet to stop running, though... [19:31:17] no sudo, so apt-get install fails [19:31:24] you have sudo [19:31:26] You should have sudo. [19:31:28] it's your wiki password [19:31:34] you don't have it on bastion [19:32:11] aha. wrong password :) [19:32:17] I'm in 2 minds about the openstack structure, like a single controller is pointless spreading the databases but then after that you still have to keep all the dbs in replication etc. Tempting to just mirror the single controller than have to build a cluster to manage the cluster =/ [19:33:22] Bah I just remembered I need to fix my ldap server, hate corrupt slapd dbs :( [19:33:35] heh [19:33:46] another reason I hate openldap [19:33:54] its database gets corrupted far too easily [19:34:10] somehow opendj uses a bdb, but its database *never* gets corrupted [19:34:27] sun directory server also never corrupted itself [19:34:34] opendj is java isn't it? [19:34:59] And yeah, it's like the 3rd time though this last time I blame a certain person for powercycling the host in a not very nice way ... [19:35:32] yeah, java [19:35:50] and the bdb implementation is java too [19:35:58] Hmm probably easier to update than openldap then, I assume it's just a jar you run? [19:36:08] well.... [19:36:16] it's not that easy, unfortunately [19:36:23]