[00:04:46] LeslieCarr: he's doing things a lot differently than us [00:04:55] since we don't have nagios fully puppetized [00:05:35] he's doing a SMW query in labsconsole for all instances [00:05:56] then he steps through them, adding them to nagios, with monitoring based on puppet classes and variables [00:06:11] it's actually a pretty clever way of doing it, and it's faster than puppet [00:07:09] but, there aren't as many checks defined either [00:08:01] though. I guess we could probably define checks in labsconsole based on puppet classes, then he could do an additional search for checks and apply them [00:09:40] gotcha [00:10:22] does he want to puppetize nagios and do ti for prod? ;) [00:10:43] he'd need access to spence [00:11:07] that's one of the reasons he did it the way he did. he didn't have access to see how it's done in production [00:11:33] though, honestly, I'm glad it's being done this way, because we don't depend on external resources (which won't work without a central puppetmaster) [00:13:18] Wouldn't it make more sense to pull the puppet classes out of ldap directly than go though the wiki? [00:13:36] actually, no [00:13:52] we plan on moving away from using ldap for the puppet info [00:13:57] at some point [00:14:23] Directly into nova? [00:14:35] not totally sure [00:14:40] likely nova metadata [00:14:50] we're still investigating it [00:15:01] but, puppet's ldap implementation is limited [00:15:10] and as far as I can tell, it's not maintained [00:17:38] Hmm, not looking into ldap for puppet that much tbh. Kinda tempted to move away from that external meta script that it does and just write the classes and push them into git directly as static files. [00:18:11] yep. that could work too [01:03:32] * jeremyb waves from amtrak [01:04:27] Jeff_Green: did you make your first project yet? [01:04:38] * jeremyb wants another one too, forgot earlier [01:06:06] 11 21:07:00 < jeremyb> can I get a project made for upload wizard? [01:06:50] kaldari says upload wizard doesn't really have an owner atm and we found some bugs in the commons version this weekend [01:20:21] Ryan_Lane or lcarr? can i have a project? (^^) [01:21:17] sure [01:21:22] you already have a labs account, right? [01:21:34] i does [01:21:40] wiki username? [01:21:43] user=nick [01:21:48] ok [01:22:11] done [01:22:19] 02/15/2012 - 01:22:19 - Creating a project directory for upload-wizard [01:22:19] 02/15/2012 - 01:22:19 - Creating a home directory for jeremyb at /export/home/upload-wizard/jeremyb [01:23:18] * Damianz stretched out [01:23:18] idk who else is interested in it to add to the project [01:23:20] 02/15/2012 - 01:23:20 - Updating keys for jeremyb [01:23:25] Ryan_Lane: danke [01:23:35] yw [01:26:49] Ryan_Lane: so, we're still getting IP edits... [01:26:58] yeah. no clue why it does that [01:27:24] especially since it uses the stupid wgUser's username in the page content [01:27:30] and it's correct there [01:27:42] I must be doing something wrong [01:28:54] * jeremyb stabs the amtrak internets [01:29:27] Ryan_Lane: it only happens when it's doing a fake edit or also when you click edit, type, click save? [01:29:35] just fake edits [01:51:02] hah, there's a project called leslie! [01:51:08] can she be puppetized? [01:51:17] We have some people in leslie too. [02:14:49] * jeremyb stabs no credentials for your account [02:42:06] RECOVERY Free ram is now: OK on bots-sql3 bots-sql3 output: OK: 20% free memory [02:42:56] PROBLEM Free ram is now: WARNING on puppet-lucid puppet-lucid output: Warning: 12% free memory [02:50:06] PROBLEM Free ram is now: WARNING on bots-sql3 bots-sql3 output: Warning: 18% free memory [03:07:56] PROBLEM Free ram is now: CRITICAL on puppet-lucid puppet-lucid output: Critical: 3% free memory [03:27:56] RECOVERY Free ram is now: OK on puppet-lucid puppet-lucid output: OK: 20% free memory [03:28:09] 02/15/2012 - 03:28:09 - Creating a home directory for otto at /export/home/reportcard/otto [03:29:08] 02/15/2012 - 03:29:08 - Updating keys for otto [06:36:01] PROBLEM Free ram is now: WARNING on puppet-lucid puppet-lucid output: Warning: 19% free memory [06:41:01] RECOVERY Free ram is now: OK on puppet-lucid puppet-lucid output: OK: 20% free memory [09:35:40] !log deployment-prep add-apt-repository ppa:j/timedmediahandler && apt-get install ffmpeg php-pear on deployment-webs1 and deployment-web2 [09:37:51] so what caused all thumbnails to go on http://commons.wikimedia.beta.wmflabs.org/wiki/File:Sheep.ogv this was working yesterday [10:11:33] j^: we switched to wmf1 branch and that has broke a lot of stuff [10:11:43] so probably [10:11:45] that's it [10:12:06] maybe [10:14:08] petan|wk: ok will let the switch settle and look if its related to TMH code later [10:14:11] hyperon: hi [10:14:38] I don't really if it's broken in branched version or somewhere on test cluster [10:14:44] I don't know [10:15:16] hyperon: can you move more bot to bots-1 [14:38:16] * jeremyb waves petan|wk [16:56:04] surrur [16:56:32] I now have a cool instance but I don't know what I can with it [17:32:38] Nikerabbit: anything you'd like to do with it (that doesn't involve misuse ;) ) [17:38:12] Ryan_Lane: but do I have to do everything from scratch or can I easily install some wikies, setup a shared database and play with them? :) [17:38:23] unfortunately from scratch [17:38:30] most of it is installable via packages [17:38:45] we're working on a puppet class to do an all-in-one installer [17:39:05] eventually we'll also have an environment where everything is already there, and you run a script to get a new wiki [17:40:02] hmm oki [17:40:23] sounds like I should find someone to explain me how it is done now in the cluster and then try to replicate that [17:40:59] well, deployment-prep is already set up like the cluster [17:41:08] for now I'd probably avoid doing that [17:41:13] it takes a ton of instances [17:41:18] and a lot of configuration [17:41:21] Ryan_Lane: Petr was saying he needs the instance limit raised for that project, he can't create any more instances now [17:41:30] (maxed at 10) [17:41:37] Ryan_Lane: well, I only need a subset of that [17:42:05] RoanKattouw_away: which project? [17:42:08] deployment-prep? [17:42:12] I raised it yesterday [17:42:25] Oh, KO [17:42:28] Then it's all good [17:42:31] Nikerabbit: you probably mostly need apache, mysql, memcache, apc, and php [20:39:27] hi roan,ryan; would it be possible to configure gerrit in such a way that review can be turned on / off for a specific branch of a repository? [20:39:40] yes [20:39:46] I believe so [20:40:22] how? [20:42:40] hm [20:43:41] maybe ask in the #gerrit channel? [20:43:43] Ryan_Lane: Allow Push for refs/heads/ , I think [20:43:46] I don't know off the top of my head [20:44:01] I don't think that necessarily lets you bypass review [20:44:42] Well, you had this set up for operations at some point, didn't you? [20:47:24] i found it [20:47:26] $ git config remote.noreview.pushurl ssh://:29418/.git [20:47:26] $ git config remote.noreview.push refs/heads/* [20:47:27] $ git push noreview // this will push your current branch up, bypassing review [20:50:01] hi [20:50:38] howdy [20:50:45] hi all [20:50:46] danakim: so, you wanted to help out with puppet configs, right? [20:50:56] !account-questions | danakim [20:50:56] danakim: I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [20:51:12] hashar: hm. what user runs jenkins on the jenkins server? [20:51:27] no idea, probably jenkins [20:51:41] I'm wondering if I should just name the user jenkins in LDAP [20:51:45] that might cause issues, though [20:51:57] name it jenkinsbot ? [20:51:58] since it's also a system user in labs [20:52:00] yep, I would really like to help with that [20:52:00] Isn't there already one in there? [20:52:04] jenkinsbot is likely better [20:52:07] RoanKattouw: is there? [20:52:18] I ran into an existing user at the SF hackathon, that was either gerrit or jenkins [20:52:24] hm [20:52:24] and I do a a lot of puppet work at my current job so that would fit [20:52:27] let's see [20:52:35] ok, let me answer the questions [20:52:38] danakim: sounds good to me :) [20:52:43] Ryan_Lane: go ahead, the username can be changed in jenkins [20:52:50] I meant the user that connect to gerrit [20:53:07] or we can use some kind of tracking number like application_user_0001 [20:53:08] ;) [20:53:09] yeah. better to name it something else [20:53:13] heh. nah [20:53:21] named by purpose is better [20:53:44] RoanKattouw: there's a gerrit2 user [20:54:04] that's the one you had issues with [20:54:14] Ah, right [20:54:15] and in hindsight, I wish I would have named it gerritbot ;) [20:54:29] my previous comp used that. So you had application D5B with module D5B_MOD_00 running batch D5B_MOD_00/BATCH_0001 under username D5B_ADM [20:54:30] Ryan_Lane: 1. danakim 2. dachim 3. akim.dan@gmail.com [20:54:31] * Ryan_Lane enters an rt ticket for that [20:54:45] I'm going to rename that stupid user :) [20:54:59] hashar: oh. usernames under usernames? [20:55:07] not really just made it up [20:55:10] ah [20:55:24] thanks Ryan [20:55:25] it's possible to have them under each other in LDAP ;) [20:55:49] though that would be heresy, and I'd stab anyone who did it ;) [20:56:11] Ryan_Lane: I have generated a ssh key pair for jenkinsbot. I am wondering were the private key should be put. Is that in LDAP? [20:56:21] err [20:56:29] hi al [20:56:31] l [20:56:41] hello koolhead12 [20:56:49] hi hashar [20:57:13] Ryan_Lane: I am guessing I will get the details in an email. how about instructions where and how to connect and so on? [20:57:26] hashar: no. the private key should only be on the jenkins server [20:57:39] hashar: we can add it to the private repo, as well [20:57:49] but only ops can add it to the private repo [20:58:25] danakim: I'll walk you through it when I make the account. gimme a min :) [20:58:36] Ryan_Lane, what are we working on currently [20:58:52] it is on gallium:/var/lib/jenkins/.ssh/id_rsa [20:59:08] !projects | koolhead12 [20:59:08] koolhead12: https://labsconsole.wikimedia.org/wiki/Special:Ask/-5B-5BResource-20Type::project-5D-5D/-3F/-3FMember/-3FDescription/mainlabel%3D-2D [21:00:15] Ryan_Lane, i can work with juju :) [21:00:25] Ryan_Lane: sure! just curious :) [21:00:51] koolhead12: the juju people were going to reimplement our infrastructure using juju [21:01:01] koolhead12: but, they haven't come back in a while [21:01:07] you may want to talk to them about it [21:01:13] mark mims was the guy working on it [21:01:25] Ryan_Lane, am one of the charmers !! :) [21:01:31] heh [21:01:38] we don't really have many plans to use juju right now [21:01:43] s/many/any/ [21:01:54] mark is really busy TBH [21:01:59] okey [21:02:19] either way, they may still be interested in doing that project, so I'd say work with them on it [21:03:02] hashar: ok. I think we can get away with the user just being an LDAP user with a key. no password, no labs account [21:03:12] if it needs to have more integration later, we can extend the account [21:03:26] hashar: where's the public key? [21:03:38] Ryan_Lane: gallium:/var/lib/jenkins/.ssh/id_rsa.pub [21:04:35] !initial-login | danakim [21:04:35] danakim: https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [21:04:52] danakim: it may also be good to subscribe to labs-l, which is currently very low volume [21:05:00] https://lists.wikimedia.org/mailman/listinfo/labs-l [21:06:06] hashar: I'm going to use jenkins-bot [21:06:53] hm. it needs to log into gerrit at least once... [21:07:02] so I guess I need to give it a password, then remove it [21:07:30] \o/ [21:08:14] Ryan_Lane:thanks. subscribed to the list, reading now the wiki link... [21:08:15] also the key is protected with a passphrase, I am not sure where you want the passphrase to be stored [21:08:55] hashar: well, jenkins needs to use it right? [21:09:04] I don't see why you'd have a passphrase... [21:09:12] just remove it [21:09:30] a bot has to access it, so it needs to have the passphrase, which means you need to store it plaintext [21:09:34] so, there's not much point [21:09:36] cause anyone could grab that key and use it to connect to gerrit ? [21:09:47] would be the same either way [21:10:01] what kind of gerrit permissions with the bot have? +1,-1? [21:10:07] anything else? [21:10:57] I have no idea yet :/ For now the JenkinsBot group has just the -2, 0 rights [21:11:21] It shouldn't have -2 / +2 [21:11:27] I'd imagine you'd want +1,-1 [21:11:28] Jenkins should touch "Verified" but never "Reviewed" [21:11:37] it *may* also want submit.... [21:11:45] if you want jenkins to automatically merge [21:11:51] <^demon|away> No, that's scary [21:11:56] then only +1,-1 [21:12:07] which means the bot has basically no rights. [21:12:11] <^demon|away> Just because you can write code that passes tests doesn't mean it should automatically go in ;-) [21:12:13] so, if it gets stolen, what's the damage? [21:12:30] well probably not much [21:12:32] we reset the key and go on with our lives :) [21:12:49] you are way more pragmatic than I am [21:12:49] <^demon|away> Ryan_Lane: People might fraudulently +1 a revision!!!1! [21:12:54] heh [21:13:02] yeah. not too worried about it [21:13:09] I guess that is why I never started a career in security, I ask myself too many questions and I am too paranoid [21:13:17] heh. I'm super paranoid too [21:13:19] ^demon|away: Allowing Jenkins to merge would only happen if someone else has already +2ed [21:13:24] (OpenStack model) [21:13:46] <^demon|away> Well presumably if you're going to +2 it you're going to go ahead and merge. [21:13:52] but I think about the vulnerability surface and limit my paranoia to that [21:14:08] ^demon|away: indeed. that's why openstack's model lets jenkins automatically merge [21:14:20] see, that would make me a little more paranoid about the key getting stolen [21:14:49] but, we'd notice merges going through and just revert the bad ones, then change the key [21:15:08] The reason OS lets Jenkins merge is because their workflow is 1) human reviews and +2s, 2) Jenkins attempts the merge locally, tests merged state, 3) Jenkins tells Gerrit to merge [21:15:47] They don't want to test before review has happened, because their tests are a bit security-sensitive. I mean their product is a freakin' virtualization suite after all [21:15:55] indeed [21:16:04] we'll likely have some workflows like this for infrastructure testing [21:16:13] 02/15/2012 - 21:16:13 - Creating a home directory for danakim at /export/home/bastion/danakim [21:16:59] <^demon|away> Yeah, ours is going to be 1) push, 2) jenkins cherry picks and runs the tests, reports results, 3) human reviews, +2s and merges [21:17:12] 02/15/2012 - 21:17:11 - Updating keys for danakim [21:17:12] 4) deploy [21:17:21] <^demon|away> 5) ??? [21:17:24] <^demon|away> 6) Profit!!! [21:17:27] By default, #2 takes the commit as submitted [21:17:40] Ryan_Lane: have you created the jenkins-bot ? [21:17:51] yes. I just logged into gerrit with it, as well [21:17:53] I have some code from OpenStack that can make #2 work against the state-as-it-would-be-if-we-merged-this-into-master-right-now [21:17:58] gimme a sec [21:18:11] Which makes a bit more sense for their workflow but still makes some sense for ours [21:18:19] Like, it'll reject anything that doesn't merge cleanly [21:18:40] Success [21:18:42] which email should we use for this user account? [21:18:51] note: it doesn't require one [21:19:03] want to just leave blank? [21:19:27] ok. lemme reset its password to something random [21:19:39] or, even better, remove the password attribute [21:20:11] ok. you guys are good to go [21:20:17] thanks a ton ryan! [21:20:22] yw [21:20:26] * hashar sends asher to hugs you [21:20:30] hahaha [21:22:45] danakim: so, you can modify our puppet config in gerrit, and test it in your project [21:22:56] we don't yet have per-project branches, though :( [21:23:21] we'd really love help turning our spaghetti code repo into modules [21:24:31] Ryan_Lane: just got keys uploaded and accounts created [21:24:51] I did clone you git tree yesterday and noticed you don't use modules :) [21:25:02] yeah. it makes things harder in some ways [21:25:12] we started using puppet before modules were available [21:25:20] and never switched how we were doing things [21:25:30] true, but in the long run it does help a lot [21:25:33] yeah [21:25:34] agreed [21:25:48] it is hard to beat inertia [21:25:58] so I have to create a new project? [21:26:06] also, a number of the server types we have don't run properly on initial run [21:26:28] and then create instances in that project, right? [21:26:31] that's another thing we'd like to fix, because we can't do automated infrastructure testing without that [21:26:39] danakim: yeah. I'll need to create a project for you, though [21:26:42] puppet-cleanup? :) [21:26:53] please do! :) sounds like a good name [21:27:02] cool. [21:27:16] agreed, but honestly puppet is a pita in that respect [21:27:21] yep [21:27:32] but our end-goal is complete infrastructure tests [21:27:40] so, we have to make it work ;) [21:27:42] especially with the random catalog compilation [21:27:47] yeah [21:27:51] heh, can be tricked into [21:27:57] btw, what version of puppet do you use? [21:28:21] so we use puppet here [21:28:36] 2.7.6 [21:29:18] 02/15/2012 - 21:29:17 - Creating a project directory for puppet-cleanup [21:29:18] 02/15/2012 - 21:29:18 - Creating a home directory for danakim at /export/home/puppet-cleanup/danakim [21:30:19] 02/15/2012 - 21:30:19 - Updating keys for danakim [21:30:42] nice,thanks. and good to know you are up to date. I am afraid of 2.8 though, lots of big changes [21:31:10] so those will have to think about those too [21:32:05] Ryan_Lane: I see "updating keys for danakim". should I wait a bit before being able to log in the bastion host? currently I am unable to [21:32:29] danakim: yeah. we've already started making changes for that [21:32:36] hm. you should be able to get there now [21:32:56] danakim@bastion.wmflabs.org, right? [21:33:55] danakim: you were using dachim ;) [21:34:10] shit [21:34:17] I gave you the wrong shellaccountname [21:34:22] * Ryan_Lane goes to fix that [21:34:24] ah! damn it! [21:34:30] that's alright [21:34:39] if it is too much trouble leave it like that [21:34:39] it's ok. I can fix it [21:34:43] cool then [21:35:12] 02/15/2012 - 21:35:11 - Creating a home directory for danakim at /export/home/bastion/danakim [21:35:21] 02/15/2012 - 21:35:21 - Updating keys for dachim [21:35:42] the scripts *should* handle this properly [21:36:32] it seems that they actually have ;) [21:36:35] thanks! [21:37:54] yw [21:38:00] I wrote these scripts ages ago [21:38:06] :) [21:38:10] glad to see they still work [21:38:20] 02/15/2012 - 21:38:20 - Creating a project directory for lvs-labs [21:38:21] 02/15/2012 - 21:38:20 - Creating a home directory for laner at /export/home/lvs-labs/laner [21:38:21] 02/15/2012 - 21:38:20 - Creating a home directory for lcarr at /export/home/lvs-labs/lcarr [21:38:51] heh [21:38:57] it didn't rename your home directory [21:39:02] it is a good feeling to not have to go through old code again, trying to remember what you wanted to do there [21:39:02] it re-created it [21:39:05] that's technically a bug [21:39:21] 02/15/2012 - 21:39:21 - Updating keys for lcarr [21:39:22] 02/15/2012 - 21:39:21 - Updating keys for laner [21:39:22] 02/15/2012 - 21:39:21 - Creating a home directory for mark at /export/home/lvs-labs/mark [21:39:26] you want me to file a bug report? :P [21:39:32] heh. nah [21:39:44] we do renames so infrequently that it doesn't matter [21:40:20] 02/15/2012 - 21:40:20 - Updating keys for mark [21:40:43] ok. food [21:40:46] alright, I also see the project page and that I can create an instance [21:41:01] danakim: yep. I'll be back in a little bit. let me know if you have any questions [21:41:08] I will zone out for the night [21:41:18] 02/15/2012 - 21:41:18 - Creating a home directory for hashar at /export/home/jenkins/hashar [21:41:19] will start reading and going through code asap [21:41:45] and will ask if I have question! thanks for the help [21:41:52] talk to you all later [21:42:17] 02/15/2012 - 21:42:17 - Updating keys for hashar [21:53:04] !accountreq [21:53:05] in case you want to have an account on labs, please contact someone who is in charge of doing that: Ryan.Lane, m.utante or ssmolle.tt [21:53:58] ssmollett Ryan_Lane|away: Hallo. Can haz labs? [21:55:21] RoanKattouw: sorry I can not allocate an IP, I got a "Failed to allocate new public IP address" error [21:55:30] RoanKattouw: I did release the one I used 208.80.153.212 [21:56:19] Hmm, maybe it's per-project [21:56:50] IIRC it is not implemented yet [21:58:43] Then you'd have to bother Ryan I guess [21:59:23] at least I have released it :) [22:12:17] <^demon|away> Oh Ryan's away :( [22:20:42] He isn't food? [22:20:51] That's too bad I was kinda hungry [22:21:34] petan|wk: by more bot you mean the channel log bot? [22:21:57] sure i can in about an hour... [22:23:24] actually [22:24:10] i can make it a lot easier in myself but i have to wait until Ryan_Lane|away gets back so i can ask him something [22:58:59] Any chance anybody else can set me up with a labs account? [22:59:09] And/or is there anyone in the SF office I could pester about this? [23:00:24] You'll very probably have to wait for Ryan [23:00:28] Isn't he in the SF office? [23:03:29] !account-questions | dschoon [23:03:29] dschoon: I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [23:04:42] declerambaul: (1) wiki/git/shell name: dsc (2) svn account: dsc (3) email: dschoonover@wikimedia.org [23:06:42] so. many. pings. [23:07:29] dschoon: sorry i didn't notice to 'from you'. You should dm Ryan_Lane with that info (ping) [23:09:50] Ryan_Lane|away: Want some pongs to go with your pings? [23:22:57] !initial-login | dschoon [23:22:57] dschoon: https://labsconsole.wikimedia.org/wiki/Access#Initial_log_in [23:23:27] ty! [23:31:32] Ryan_Lane: I think the answer is "no", but is it possible to use Labs such that I could in some env make changes to LocalSettings.php, restart apache at will, things like that? [23:32:01] well, in labs, right now, there's no pre-made mediawiki [23:32:20] so, you get a project, or are added to a project, and in that project you can create instances [23:32:25] in those instances you have root [23:32:39] so you can install mediawiki, and develop away [23:33:57] Ryan_Lane: ah. I didn't get that from http://labs.wikimedia.beta.wmflabs.org/wiki/Main_Page. so an "instance" is like a bare Linux VM? [23:34:11] well, beta is not labs ;) [23:34:18] it's something that lives in labs [23:34:34] Ryan_Lane: I are a n00b :-) [23:34:46] yes, an instance is a virtual machine [23:34:49] !terminology [23:34:49] https://labsconsole.wikimedia.org/wiki/Terminology [23:34:58] reading... [23:35:03] so, this is basically like a private ec2 [23:37:17] Ryan_Lane: can snapshots be made of an instance? [23:37:26] not currently [23:37:36] but, you can puppetize the instance [23:37:55] then if you wanted to re-make an instance, it would automatically re-create itself [23:38:20] our puppet repository is open to the public, and anyone with a labs account can push into it, for review [23:38:29] * Damianz wonders if installing all the controller stuff being on a single box is a hugly bad idea. [23:38:35] Damianz: I did [23:39:01] if you think about it, it's less SPOFs ;) [23:39:21] Yeah [23:39:25] load isn't enough to matter [23:39:31] Like I could just drdbify the box for a cold standby [23:39:36] unless you are going to have a *lot* of people using it [23:39:43] Ryan_Lane: I think I could use a Labs account if that's possible. [23:39:45] Don't fancy all the db replicas being maintainced seperatly... [23:39:48] !account-questions | chrismcmahon [23:39:48] chrismcmahon: I need the following info from you: 1. Your preferred wiki user name. This will also be your git username, so if you'd prefer this to be your real name, then provide your real name. 2. Your SVN account name, or your preferred shell account name, if you do not have SVN access. 3. Your preferred email address. [23:40:01] chrismcmahon: sure, what do you plan on working on? [23:40:15] Ryan_Lane: I'm the new QA person for WMF. [23:40:19] ahhh. ok [23:40:24] yeah, you should have a labs account, for sure [23:40:33] MWF have QA people? [23:40:41] we do now :) [23:40:48] I'm looking for place(s) where I can do perverse things to a Mediawiki install. [23:40:57] that would be this [23:40:59] Damianz: I'm the only one, and I'm new :) [23:41:08] lol [23:41:43] Mediawiki did perverse things to my extensions theother day... hate updates. [23:42:47] Damianz: I'm really excited about it. I get to invent what "QA" means for WMF. It's a long-term project for sure. :) [23:43:11] chrismcmahon: Awww it doesn't mean we get to string you up and shoot you when stuff breaks? [23:44:34]