[00:09:05] <Ryan_Lane>	 so, network-node per compute-node
[00:09:16] <Ryan_Lane>	 replace nfs home dirs with gluster home dirs
[00:09:24] <Ryan_Lane>	 (we already have shares configured for that)
[00:09:41] <Ryan_Lane>	 hm. we're leaving
[00:09:46] <Ryan_Lane>	 to ben's going away
[00:09:49] <paravoid>	 oh, Ben's party
[00:09:50] <paravoid>	 right
[00:09:55] <Ryan_Lane>	 so, I'll talk with you later about upcoming things
[00:10:00] <paravoid>	 probably tomorrow
[00:10:17] <paravoid>	 I *hope* tomorrow, it's 3am already :)
[00:10:22] <Ryan_Lane>	 heh
[00:10:41] <Damianz>	 Only 1am meh
[01:11:59] <Damianz>	 !log nagios Free ram plugin seems to be working, 27 hosts it's not working on - probably puppetmaster::self/broken puppet
[01:12:01] <labs-morebots>	 Logged the message, Master
[03:34:21] <Damianz>	 http://twitter.github.com/ that is pretty damn sexy
[03:34:44] * Damianz  thinks labsconsole so needs an overview page like that

[08:03:51] <ori-l>	 blast from the past: no nova credentials?
[08:03:58] <ori-l>	 are any "nova administrators" about?
[08:04:12] <Damianz>	 logout and back in
[08:04:28] <Damianz>	 I swear the wiki isn't keeping my session anymore now, wonder if the keystone ttl is looowwww
[08:04:50] <ori-l>	 magic! works
[08:05:01] <ori-l>	 (did i miss an email?)
[08:05:15] <hashar>	 !beta deployment-integration : update puppet git repo to latest master
[08:05:15] <wm-bot>	 !log deployment-prep deployment-integration : update puppet git repo to latest master
[08:05:18] <labs-morebots>	 Logged the message, Master
[08:05:22] <Damianz>	 The one that said ryan force logged everyone out wasn't 100% right it seems
[08:07:28] <Damianz>	 Hmm, labs allows any software under an opensource license right? so wtfpl counts
[08:10:21] <Nikerabbit>	 I get "There were no Nova credentials found for your user account. Please ask a Nova administrator to create credentials for you." when I go to "Manage instances"
[08:10:47] <Nikerabbit>	 fixed by logout,login
[08:13:06] <ori-l>	 yeah, same thing i experienced
[08:13:55] <Nikerabbit>	 is it in bugzilla?
[08:15:02] <hashar>	 Ryan deleted all credentials to force relogin
[08:15:16] <hashar>	 if it still happen then I guess we can fill in a bug for him
[08:15:24] <hashar>	 I did not face the issue myself
[08:17:32] <Damianz>	 what is this 'fill a bug' I just yell at him
[08:17:57] <Damianz>	 He ignores my bugs anyway (looks at ldap patch that's been there 6+months)
[08:50:19] <hashar>	 Damianz: which patch ? :-D
[08:50:30] <Damianz>	 ?
[08:50:57] <hashar>	 [Ryan] ignores my bugs anyway (looks at ldap patch that's been there 6+months)
[08:53:42] <Damianz>	 bz 34787, no one ever looks at patches in bz though
[09:00:45] <Damianz>	 Anythoughts on https://github.com/DamianZaremba/labsnagiosbuilder/blob/master/labsnagiosbuilder/build.py vs http://svn.wikimedia.org/svnroot/mediawiki/trunk/tools/nparser/fetch.sh + http://svn.wikimedia.org/svnroot/mediawiki/trunk/tools/nparser/nparser.cpp?
[09:07:51] <hashar>	 Damianz: yup we have some troubles managing patches in bugzilla :(
[09:10:02] <Damianz>	 gerrit makes that all better... for viewing them anyway
[09:14:03] <Damianz>	 o.0 linking like actually works
[09:14:18] <hashar>	 I will mail ryan lane
[09:14:28] <Damianz>	 also yay hashar make tidy, I suck at doing bz reports heh
[09:17:01] <hashar>	 Damianz: mail sent :)
[09:17:18] <hashar>	 also tweaked https://gerrit.wikimedia.org/r/#/c/21309/ commit message and fixed some whitespaces
[09:17:19] <hashar>	 ;)
[09:19:44] <Damianz>	 Yeah... there seems to be lots of tab usage, makes my pep8ified bain and editor settings go crazy
[09:20:42] <hashar>	 we use tab by default in mediawiki 
[09:20:59] <hashar>	 maybe your editor is able to change tab / space preferences based on file type ?
[09:21:08] <hashar>	 aka python : 4 spaces,  php: tabs
[09:21:41] <Damianz>	 I could probably do that, it's suppose to autodetect but sucks at doing so.
[09:30:35] <Damianz>	 Btw did we ever get around to removing the production logos from beta?
[09:56:30] <hashar>	 never did
[09:56:33] <hashar>	 err
[09:56:36] <hashar>	 never looked at doing so
[09:57:15] <hashar>	 we could write an extension that would inject a css to overlay a big BETA on the logo
[09:57:31] <hashar>	 (or make the body pink :-))
[09:58:26] <Damianz>	 lol
[09:58:34] <Damianz>	 Make the whole thing ping
[15:20:39] <Damianz>	 Hmm no ryan :(
[15:20:58] <^demon>	 It's only 8:20 in SF.
[15:21:17] <Damianz>	 Ah true, he's not usually around for another hour
[15:21:31] <Damianz>	 Feel like annoying him about labsconsole which is being fail ux wise
[15:22:26] <^demon>	 andrewbogott_afk: So, turns out our bug with java in precise is <https://bugs.launchpad.net/ubuntu/+source/openjdk-6/+bug/888100>
[15:22:32] <^demon>	 Not actually our fault :)
[15:22:39] <^demon>	 Production's still on lucid.
[15:23:10] <Damianz>	 Oh meh, I really should remember to login to wikis
[15:30:03] <andrewbogott>	 ^demon:  Can we use the workaround described in that bug?  (I don't entirely understand why doing that would make a difference.)
[15:31:44] <Damianz>	 I don't suppose anyone happens to have a deep love for python's unittest module?
[15:34:46] <andrewbogott>	 Damianz:   I don't know that I love it, but I use it quite a bit.  What's up?
[15:36:39] <Damianz>	 Wondering if there's a better way of doing https://github.com/DamianZaremba/labsnagiosbuilder/blob/master/labsnagiosbuilder/tests/TestLdap.py#L49 without looping over each key, check if the key and value match returning false it not or dropping out the loop with true at the end. Granted those tests hardly cover any of what the script does but I don't usually write tests for stuff.
[15:36:59] <Damianz>	 Basically if I don't sorted()'d the dicts, they error eventhough the keys/values are identical but running them though sorted seems so hacky.
[15:37:27] <^demon>	 andrewbogott: Possibly. I'm trying a fresh instance to see.
[15:38:56] <andrewbogott>	 Damianz:  Does assertDictEqual maybe do what you want?
[15:39:19] <andrewbogott>	 Oh, docs say that assertEqual calls assertDictEqual anyway.
[15:39:31] <andrewbogott>	 I sorta can't imagine that it's ordering sensitive :(
[15:40:01] <andrewbogott>	 Have you run an experiment to verify that order is for sure the problem?
[15:41:09] <Damianz>	 Hmm I didn't think so but https://secure.travis-ci.org/#!/DamianZaremba/labsnagiosbuilder/jobs/2270803 suggested otherwise (as soon as sorted was added it pssed... after a bit of trying to fudge it with a diff of 2 sets)
[15:41:52] <Damianz>	 It works for now... probably will break at some point heh. Oh well, I'll work on it and get it reviewed in gerrit when ^demon makes me a repo and then I can annoy suggest actually using it =\
[15:42:35] <andrewbogott>	 What if you do assertDictContainsSubset(A,B) and then assertDictContainsSubset(B,A)?
[15:42:47] <andrewbogott>	 That's just as stupid as what you suggest, but uses fewer lines of code :)
[15:43:34] <Damianz>	 That could work... maybe. I'm thinking I might just re-write this to have 1 big dict and non-duplicated checks if I end up doing that.
[15:43:40] <andrewbogott>	 Oh, wait, I guess what you have is shorted.  Nevermind :)  I'd just add a comment apoligizing for the sort.
[15:45:00] <Damianz>	 heh
[15:45:09] <Damianz>	 # Warning: your sanity may be lost reading furthur
[15:46:43] <andrewbogott>	 # If this line bothers you, complain to <addressofunittestdeveloper>
[15:49:30] <Damianz>	 This is probably why there's so many plugins for nose heh
[15:52:46] <Damianz>	 Hi hashy
[15:59:17] <^demon>	 andrewbogott: Purging and reinstalling doesn't seem to have worked.
[15:59:28] <andrewbogott>	 dang
[16:00:58] <^demon>	 So for now, I'm just going to stick with lucid for labs. I don't feel like dealing with a stupid package that put itself in a new directory.
[16:08:29] <andrewbogott>	 ^demon:  Did you see 'This bug was fixed in the package openjdk-6 - 6b24-1.11-0ubuntu1'?
[16:08:40] <andrewbogott>	 Can we just get that version into our apt repo?
[16:09:16] <^demon>	 That might be possible.
[16:10:06] <andrewbogott>	 I'm not ready to abandon precise.  Lemme see if I can get the version into our repo.
[16:10:20] <andrewbogott>	 Um… as soon as I finish this other thing...
[16:43:42] <andrewbogott>	 ^demon, can you try an update of openjdk-6-jre-headless and see if you get openjdk-6-jre-headless_6b24-1.11.3-1ubuntu0.12.04.1_amd64?
[16:43:53] <andrewbogott>	 (Presuming you still have a precise test system up)
[16:45:08] <andrewbogott>	 oops
[16:55:49] <andrewbogott>	 ^demon:  Nevermind, the version we're using is already newer than the one that claims to fix the bug.  So, I'm baffled.
[18:37:09] <Ryan_Lane>	 !log testlabs deleting p-b. It's corrupted and un-used.
[18:37:10] <labs-morebots>	 Logged the message, Master
[19:17:08] <kaldari>	 I seem to not be able to ssh to ee-prototype.pmtpa.wmflabs from bastion today. Gives me permission denied error.
[19:17:59] <kaldari>	 sshing to beta works fine though
[19:18:12] <kaldari>	 Ryan_Lane: ^
[19:18:24] <Ryan_Lane>	 sec
[19:19:10] <Ryan_Lane>	 kaldari: try it for me
[19:19:31] <Ryan_Lane>	 ah
[19:19:33] <Ryan_Lane>	  / is full
[19:19:43] <kaldari>	 oops :)
[19:19:58] <Ryan_Lane>	 the gluster log filled it
[19:20:01] <Ryan_Lane>	 not sure wh¥
[19:20:11] <Ryan_Lane>	 I guess I should have checked out the log before I deleted it
[19:20:20] <Ryan_Lane>	 try now
[19:20:37] <Ryan_Lane>	 I think I was testing gluster out on this node. it was probably my fault
[19:21:01] <Ryan_Lane>	 hm. gluster mount isn't working on this node
[19:21:17] <kaldari>	 same error
[19:21:32] <Ryan_Lane>	 hm. autofs isn't workng
[19:22:18] <kaldari>	 I'm going to grab some lunch while you're working on it
[19:23:04] <Ryan_Lane>	 ok
[19:23:29] <Ryan_Lane>	 I'm going to reboot your instance
[19:26:07] <Ryan_Lane>	 kaldari: ok. it's working now
[20:39:20] <^demon>	 Ryan_Lane: Can the tiny option go away?
[20:41:29] <Ryan_Lane>	 heh
[20:41:36] <Ryan_Lane>	 I should atleast make the default xsmall
[20:41:48] <Ryan_Lane>	 I don't think I can delete it if any tiny instances still exist
[20:42:10] <Ryan_Lane>	 I'm adding some other features right now
[20:42:16] <Ryan_Lane>	 let me add that while I'm at it
[20:42:58] <Ryan_Lane>	 hey, at least the interface is usable again, right? :)
[20:46:43] <Ryan_Lane>	 ^demon: has 1.20wmf10 been deployed anywhere yet?
[20:46:52] <^demon>	 It's deployed evyerwhere.
[20:46:55] <Ryan_Lane>	 ok
[20:46:56] <Ryan_Lane>	 good
[20:47:02] <Ryan_Lane>	 I'm going to upgrade labsconsole soon
[20:47:11] <Ryan_Lane>	 I'm going to test it on my test instance of course, right
[20:47:32] <Kozuch>	 will someone add me (kozuch) to "bastion" project?
[20:48:37] <Ryan_Lane>	 Kozuch: are you already in another project?
[20:48:55] <Kozuch>	 I am purely new at Labs, I guess not
[20:50:17] <Ryan_Lane>	 Kozuch: what would you like to use labs for?
[20:50:25] <Ryan_Lane>	 bastion access alone doesn't really give you much
[20:51:01] <Kozuch>	 I guess Labs allows to set up a MediaWiki instance for "hacking"/developing?
[20:51:16] <Ryan_Lane>	 yeah
[20:51:27] <Ryan_Lane>	 but you'll need to be added to a project for it, or have your own
[20:51:41] <Ryan_Lane>	 was there a specific extension you wanted to work on?
[20:51:47] <Ryan_Lane>	 we try to name projects after their use
[20:53:18] <Kozuch>	 i am interested in couple of things, cant you name my instance with my username? if not, i am interested in wikidata... also in developing JavaScript charting/graphing reature
[20:53:40] <Kozuch>	 and trying to fix some easy bugs
[20:54:26] <Ryan_Lane>	 hm
[20:54:28] <Ryan_Lane>	 heh
[20:54:43] <Ryan_Lane>	 we don't necessarily have a generic dev project set up yet
[20:55:04] <Ryan_Lane>	 you create your own instances in labs
[20:55:12] <Ryan_Lane>	 maybe I should make a generic mediawiki project now
[20:55:39] <Kozuch>	 well I guess I am unable to create any instance...
[20:55:42] <^demon>	 Ryan_Lane: PS39 added ldap client utils to formey & manganese (in site.pp, so to not mess with labs).
[20:55:53] <Ryan_Lane>	 Kozuch: currently, yes, because you aren't in any projects
[20:56:08] <^demon>	 Lemme try it on a fresh instance and then maybe we can merge this behemoth.
[20:56:10] <Ryan_Lane>	 ^demon: ok. cool
[20:56:45] <Ryan_Lane>	 Kozuch: I'm trying to think of how to handle generic mediawiki work
[20:57:03] <Ryan_Lane>	 people so far have come with project ideas, rather than bugfixes
[20:57:12] <Ryan_Lane>	 ^demon: naming things is hard. any opinions?
[20:57:20] <^demon>	 What're you naming?
[20:57:23] <Ryan_Lane>	 labs project for generic mediawiki work
[20:57:30] <Ryan_Lane>	 like bugfixes and such
[20:57:56] <Ryan_Lane>	 robla: ^^
[20:58:00] <Ryan_Lane>	 also, workflow ideas
[20:58:10] <^demon>	 mediawiki-bugfix?
[20:58:15] <Ryan_Lane>	 ideally I wanted a clustered environment where the wikis were setup already
[20:58:21] <^demon>	 wikibugs?
[20:58:33] <^demon>	 meh, wikibugs is a bot
[20:58:35] <Ryan_Lane>	 heh
[20:58:36] <Ryan_Lane>	 yeah
[20:58:40] <Ryan_Lane>	 mediawiki-bugfix would work
[21:00:00] <Ryan_Lane>	 my test wiki's upgrade to wmf10 went well
[21:01:03] <Kozuch>	 Ryan_Lane: so would you get me going somehow please?
[21:01:08] <Ryan_Lane>	 yeah
[21:01:27] <Ryan_Lane>	 gimme a min
[21:01:32] <Kozuch>	 ok
[21:01:54] <Kozuch>	 whats the problem in me joining bastion?
[21:03:37] <Ryan_Lane>	 it doesn't give you anything if you aren't also in another project
[21:05:40] <Kozuch>	 the help page says: In order to access instances (see Help:Terminology) you will need to be a member of the 'bastion' project.
[21:05:50] <Ryan_Lane>	 yes
[21:06:01] <Ryan_Lane>	 but that alone won't let you access instances
[21:06:13] <Ryan_Lane>	 it'll let you access instances in projects you are a member of
[21:06:46] <Kozuch>	 ok
[21:06:48] <Kozuch>	 i see
[21:06:52] <Ryan_Lane>	 I'm making a project for you, though
[21:06:55] <Ryan_Lane>	 so, gimme a min
[21:08:45] <Ryan_Lane>	 labs-home-wm: stupid bot, why don't you start on boot?
[21:10:55] <labs-home-wm>	 08/29/2012 - 21:10:55 - User laner may have been modified in LDAP or locally, updating key in project(s): mediawiki-bugfix
[21:10:55] <labs-home-wm>	 08/29/2012 - 21:10:55 - User kozuch may have been modified in LDAP or locally, updating key in project(s): mediawiki-bugfix
[21:16:00] <labs-home-wm>	 08/29/2012 - 21:16:00 - User kozuch may have been modified in LDAP or locally, updating key in project(s): bastion
[21:16:19] <Ryan_Lane>	 Kozuch: so, you can create instances in the mediawiki-bugfix project
[21:16:23] <^demon>	 Ryan_Lane: So the only caveat is--on a fresh install it looks very broken (install tries to proceed before gerrit.config exists, we can fix this). If you just run init, then subsequent runs will work just fine.
[21:16:23] <Ryan_Lane>	 !instances
[21:16:23] <wm-bot>	 need help? -> https://labsconsole.wikimedia.org/wiki/Help:Instances  want to manage? -> https://labsconsole.wikimedia.org/wiki/Special:NovaInstance  want resources? use !resource <instance id>
[21:16:26] <Ryan_Lane>	 !security
[21:16:26] <wm-bot>	 https://labsconsole.wikimedia.org/wiki/Help:Security_Groups
[21:16:33] <^demon>	 We need to fix the behavior of when to run init, but other than that we're golden.
[21:16:35] <^demon>	 One last minor fix.
[21:16:36] <Ryan_Lane>	 Kozuch: please read that documentation
[21:16:55] <Ryan_Lane>	 ^demon: ok
[21:17:00] <Kozuch>	 Ryan_Lane: ok thanks!
[21:17:04] <Ryan_Lane>	 yw
[21:17:54] <^demon>	 PS40 was the magic # :p
[21:18:18] <Ryan_Lane>	 is jvm_path defined anywehre?
[21:21:16] <^demon>	 Nope, that's what PS40 removed.
[21:21:37] <Ryan_Lane>	 ah
[21:21:42] <Ryan_Lane>	 read that wron g
[21:22:23] <^demon>	 Still got some apache config cleanup to do for labs, but nothing there changed so production's still fine.
[21:22:32] <Ryan_Lane>	 let's see if this breaks production, now :)
[21:24:34] <^demon>	 andrewbogott: I'm cleaning up the gerrit project now. Do you still want your instances?
[21:24:44] <andrewbogott>	 ^demon:  Nope.
[21:25:08] <^demon>	 Released to the cloud :)
[21:25:10] <andrewbogott>	 ^demon, did you figure something out about the java path?  Last time I tried I got an error message about not having defined it.
[21:25:13] <Ryan_Lane>	 seems to still be working
[21:25:25] <Ryan_Lane>	 wow. people cleaning up after themselves?
[21:25:27] <Ryan_Lane>	 amazing
[21:25:39] <^demon>	 andrewbogott: Yeah, I'll cross that bridge when we upgrade to precise. At least I know what I'm dealing with now.
[21:25:46] <^demon>	 For now we're just going to leave the lucid path
[21:44:55] <paravoid>	 Ryan_Lane: heya
[21:45:28] <paravoid>	 want to continue that chat from yesterday?
[22:01:33] <^demon>	 http://gerrit-dev.wmflabs.org/r/#/ - totally puppetized \o/
[22:26:27] <^demon>	 paravoid: I've got one small problem with LDAP right now.
[22:26:39] <paravoid>	 what's that?
[22:26:47] <^demon>	 gerrit-dev is now configured to use LDAP for auth (to be more like production), and I've got the hostnames and password & such.
[22:27:02] <^demon>	 But it doesn't seem to be working. Does the host/ip have to be added to the LDAP server or something?
[22:28:00] <^demon>	 Oh, I think the username is bogus.
[22:28:01] <paravoid>	 I don't think so, the machine uses LDAP for unix logins anyway
[22:29:00] <^demon>	 Hmm, not sure if this is right.
[22:29:35] <^demon>	 I'll compare to production :)
[22:29:54] <paravoid>	 what settings do you use?
[22:32:30] <^demon>	 paravoid: http://p.defau.lt/?Uq_KUzJ0wvEtyFAydmlY6w
[22:32:43] <^demon>	 I know [redacted] is correct.
[22:33:23] <paravoid>	 oh, so gerrit has a bind dn to do search first
[22:35:58] <paravoid>	 are these settings copied from production?
[22:36:10] <paravoid>	 I have no idea how gerrit interprets "server" for example
[23:49:28] <wm-bot>	 Change on 12mediawiki a page Developer access was modified, changed by Zen4us link https://www.mediawiki.org/w/index.php?diff=577558 edit summary: 
[23:52:51] <mutante>	 let's fix "/usr/local/bin/decom_servers.sh"?
[23:53:17] <mutante>	 (cronspam)
[23:55:16] <mutante>	 undefined/operations/puppet/manifests/decommissioning.pp: No such file or directory
[23:55:36] <mutante>	 looking for the "undefined" part in there obviously.. hmm