[00:18:58] 07/17/2012 - 00:18:58 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [01:20:59] 07/17/2012 - 01:20:58 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [01:41:46] petan: some beta wikis have a robots.txt, others don't [01:42:02] is there any way we can make all of them exist, and have them all disallow everything? [01:42:17] there's no reason beta should be indexed [01:42:28] this would make the spam problem less of an issue [02:22:58] 07/17/2012 - 02:22:58 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [02:49:59] 07/17/2012 - 02:49:59 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [03:50:59] 07/17/2012 - 03:50:59 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [04:52:58] 07/17/2012 - 04:52:58 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [05:54:58] 07/17/2012 - 05:54:58 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [06:56:59] 07/17/2012 - 06:56:58 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [07:58:57] 07/17/2012 - 07:58:57 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [08:02:56] !log deployment prep hopefully enabled SORBS based autoblocking https://gerrit.wikimedia.org/r/#/c/15768/ [08:02:56] deployment is not a valid project. [09:20:17] !log deployment-prep sync upload6 dirs again. root@deployment-nfs-memc:$ rsync -a --progress --inplace /mnt/export/upload6 /data/project/upload6 [09:20:19] Logged the message, Master [09:25:46] your previous log didn't work either [09:25:49] see above :P [09:26:23] !log deployment-prep Adding class role::applicationserver::jobrunner [09:26:24] Logged the message, Master [09:27:58] hashar: ^ [09:28:13] paravoid: labs-morebots told me it is logged!! [09:28:40] 11:02 < hashar> !log deployment prep hopefully enabled SORBS based autoblocking https://gerrit.wikimedia.org/r/#/c/15768/ [09:28:43] 11:02 <+labs-morebots> deployment is not a valid project. [09:28:48] ohh h that one :) [09:29:02] !log-deployment prep hopefully enabled SORBS based autoblocking https://gerrit.wikimedia.org/r/#/c/15768/ [09:29:09] grmblbl [09:29:13] !log deployment-prep hopefully enabled SORBS based autoblocking https://gerrit.wikimedia.org/r/#/c/15768/ [09:33:07] !deployment-prep [09:33:08] deployment-prep is a project to test mediawiki at beta.wmflabs.org before putting it to prod [09:33:15] !deployment [09:33:16] deployment-prep is a project to test mediawiki at beta.wmflabs.org before putting it to prod [09:33:19] yay [09:38:39] !wikiversity-sandbox [09:40:22] Jan_Luca you need to define it [09:40:30] !key is blah [09:40:30] Key was added [09:40:32] !key [09:40:32] blah [09:40:35] !key del [09:40:36] Successfully removed key [09:40:42] thank you [09:41:16] !wikiversity-sandbox is a project to test and develop new features for Wikiversity [09:41:16] You are not authorized to perform this, sorry [09:41:24] you don't have a cloak? [09:41:33] !wikiversity-sandbox is a project to test and develop new features for Wikiversity [09:41:33] Key was added [09:41:43] I have a cloak [09:41:52] aah [09:41:53] @trusted [09:41:54] I trust: petan!.*@wikimedia/Petrb (2admin), .*@wikimedia/.* (2trusted), .*@mediawiki/.* (2trusted), .*@wikimedia/Ryan-lane (2admin), .*@wikipedia/.* (2trusted), .*@nightshade.toolserver.org (2trusted), .*@wikimedia/Krinkle (2admin), .*@wikimedia/Thehelpfulone (2admin), [09:42:20] @trustadd .*@[Ww]ikimedia/.* trusted [09:42:20] Successfuly added .*@[Ww]ikimedia/.* [09:42:31] you have capital W in cloak [09:42:34] that was why [09:42:41] Thank you [09:42:49] !test is test [09:42:49] This key already exist - remove it, if you want to change it [09:42:55] !test2 is test [09:42:55] Key was added [09:43:00] !test2 del [09:43:00] Successfully removed test2 [12:08:29] somehow back, eating a snack :) [12:34:47] !test [12:34:47] $1 [13:25:47] I am fed up with that cluster [13:26:08] This request could not be forwarded to the origin server or to any parent caches. The most likely cause for this error is that: [13:26:11] yeahhhh [13:28:14] !log deployment-prep All apaches are dead :/ [13:28:16] Logged the message, Master [13:29:42] !log deployment-prep err: /Stage[main]/Mediawiki::Sync/Exec[mw-sync]: Failed to call refresh: Command exceeded timeout at /etc/puppet/manifests/mediawiki.pp:24 [13:29:44] Logged the message, Master [13:37:38] grazeaze [13:44:45] !log deployment-prep Imported all.conf apache conf from production [13:44:47] Logged the message, Master [13:45:00] !log deployment-prep Manually restarted apaches [13:45:01] Logged the message, Master [13:58:35] !g 13304 [13:58:35] https://gerrit.wikimedia.org/r/#q,13304,n,z [14:17:28] paravoid: back around ? :-] [14:17:35] yes [14:18:04] do you have time today to do the beta migration to /data/project ? [14:18:16] relevant change is https://gerrit.wikimedia.org/r/#/c/15545/ [14:40:29] squid is confusing me :) [14:40:31] 2012/07/17 14:40:12| The request GET http://sq.wikipedia.beta.wmflabs.org:80/wiki/FireWire is DENIED, because it matched 'blacklistedagents' [14:40:32] 2012/07/17 14:40:12| The reply for GET http://sq.wikipedia.beta.wmflabs.org:80/wiki/FireWire is ALLOWED, because it matched 'blacklistedagents' [14:45:21] !log deployment-prep Blacklisted user agents matchin /.*Googlebot.*/ [14:45:22] Logged the message, Master [17:19:04] I swear Ryan just appeared with an awesome looking cat... g+ says that's not his picture though #sadface [17:19:43] o.O [17:20:35] Hello kitty cats are awesome :D [17:23:40] ahhhhh [17:23:41] that [17:23:42] heh [17:23:47] it's my chat photo [17:23:50] and my facebook one [17:24:41] paravoid: morning [17:24:56] well, night, there technically :) [18:10:04] things transfer so much faster when gluster isn't being used. it's like night and day [18:12:07] paravoid: so, for the migrations I'm going to shut the instance down, rsync it across, modify database directly, then bring the instance up on the new hardware [18:21:35] !log deployment-prep hashar: rebooting squid glusterfs gone wild apparently [18:21:36] Logged the message, Master [18:36:35] so squid is dead [18:36:45] !log deployment-prep Squid is bugged as hell : 2012/07/17 18:36:13| Store rebuilding is -0.1% complete and looping [18:36:47] Logged the message, Master [18:38:39] !log deployment-prep removing swap files in /data/project/squid1 [18:38:40] Logged the message, Master [18:39:15] !log deployment-prep installed `tree` on deployment-squid [18:39:16] Logged the message, Master [18:39:25] <^demon> If the squid are dead can we fry them up? [18:39:30] <^demon> Maybe with a spicy mustard sauce? [18:39:31] I wish :-] [18:39:38] I guess I will just migrate to use varnish [18:39:44] if I figure out how to set it up as a load balancer [18:40:05] !log deployment-prep -squid bah doing rm -fR /data/project/squid1/* [18:40:06] Logged the message, Master [18:40:09] I am fed up with beta [18:43:12] I am wondering if I could deleted a directory by just mv LARGEDIR /dev/null [18:45:01] hm [18:45:04] gluster fucked up, eh? [18:45:11] we really need to get off of gluster [18:45:20] in this case it is squid which is f***d [18:45:22] * Damianz stabs google [18:45:31] well, is it squid that's broken because of gluster? [18:45:36] if you reload the configuration while it is rebuilding something, that something get corrupted [18:45:38] or the instance was broken during a migration? [18:45:42] hashar: Setup multiple backends and a director :) [18:45:47] I did some changes then "service squid reload" [18:45:49] several time [18:45:52] that screwed it up [18:45:52] that's very likely a problem with gluster [18:45:53] for sure [18:46:14] Damianz: if you have any idea how to configure that using our puppet class, I am all listening to you :-] [18:46:19] andrewbogott: made any progress on ceph? :) [18:46:34] hashar: There's a varnish puppet class? [18:46:50] to replace project storage we'll need to be able to have share permissions like gluster [18:46:53] Damianz: we have some yes in operations/puppet.git look for manifests/varnish.pp IIRC [18:46:56] I'm not totally sure that ceph has that [18:47:00] or if it's easy to configure [18:47:02] Damianz: we have a parameterized class varnish::instance [18:47:17] Ryan_Lane: I got you a change in OpenStackManager https://gerrit.wikimedia.org/r/#/c/15775/ :D [18:47:17] Hmm I wonder how hard it is to pupetize directors... not that big on puppet [18:47:40] Damianz: I am pretty sure we handle directors. Just have to read the conf and understand it I guess :) [18:47:50] I'll take a look [18:47:54] Ryan_Lane: Only teeny, tiny bits of progress, but partly because I keep pushing it to the end of my list. [18:48:00] Just breaking java [18:48:02] * Ryan_Lane nods [18:48:34] andrewbogott: at some point in time soon we'll be using that hardware for upgrade testing of openstack [18:48:47] thinking of that, I really need to update openstack manager for keystone [18:49:16] hashar: heh, it's not used anywhere, though :D [18:49:31] hahaha. reedy made the same comment [18:49:32] Hmm maybe I can create a box to figure our puppet out on [18:49:51] Ryan_Lane: it need to be applied to a ton of place, I did not feel like I wanted to do that this afternoon [18:50:08] hashar: you could've done one of them for proof ;) [18:50:10] * Ryan_Lane nods [18:50:24] indeed [18:50:27] the proof of concept was https://gerrit.wikimedia.org/r/#/c/7083/ [18:50:40] I thought we could use an easy method to build that [18:50:53] damn I have too many projects in parralel [18:51:08] me too [18:51:09] I got an experimental eval.php with command completion [18:51:18] I'm making the new deployment system [18:51:20] and an apache configuration parser :D [18:51:25] but I really need to update openstackmanager [18:51:40] AJAXify it !!! ;-D [18:51:48] that would be a great project for an intern [18:51:48] that will come later [18:51:56] I very much want to ajaxify it [18:52:13] I need to update for keystone and the openstack api first, though [18:52:21] * hashar sends Reedy to APIize OpenStackManager [18:54:37] I really wish someone wrote a php library for nova :( [18:54:51] !log deployment-prep squid resumed. The swap files got corrupted somehow, needed to delete them entirely to start again. Squud storing again. [18:54:53] Logged the message, Master [18:55:07] Ryan_Lane: isn't it what you have wrote ? [18:55:15] no. I'm using the ec2 api [18:55:21] with amazon's client library [18:55:40] are they sharing the same api or something ? [18:55:44] I am a cloud noob [18:55:45] now I'm going to need to write the client library for nova [18:55:50] they have a native api [18:55:56] and they also implement amazon's ec2 api [18:56:00] Hmm how long does it take to build an instance? [18:56:08] Damianz: generally about 5 minutes [18:56:21] ok, I'll make one when I get home then :D [18:56:23] don't they have libraries already? [18:56:27] not for php [18:56:38] because no one is writing a web interface in php [18:56:46] oh, wait. right. damn. ;) [18:57:00] someone from analytics (can't remember who) told me about Facebook thrift which seems to be a good way to write REST service [18:57:03] Write it in python? :D [18:57:10] heh [18:57:12] Thrift is a complicated beast [18:57:13] you basically describe your API and it generates all the libs for you (php / c / java ...) [18:57:26] rest is easy, thankfully [18:57:49] I just need to make my client implementation abstract and subclass for ec2/nova [18:59:03] stupid squid [19:00:15] fuck it. I'll work on the nova api stuff today [19:00:19] I need that for keystone anyway [19:00:24] yeahhhh [19:00:43] http://www.apache.org/dyn/closer.cgi?path=/thrift/0.8.0/thrift-0.8.0.tar.gz [19:00:53] I can't migrate instances till I finish some rsyncs anyway [19:01:07] I'm not going to use a library for simple rest calls ;) [19:01:27] deployment-prep has lots of nrpe issues still :( [19:01:28] I'll just add a rest call method, then implement the client calls in each function [19:02:54] tip about robots.txt , specify a user-agent :-) [19:03:28] hashar: btw, can you make all beta sites disallow all robots? [19:03:35] right now not all sites have a robots.txt [19:04:28] !log deployment-prep updated robots.txt to specify a user-agent. Will definitely prevents Google from killing beta :) [19:04:30] Logged the message, Master [19:04:39] hmm [19:07:50] Ryan_Lane: In your puppet groups thing - are variables global to every class or do you have to assin it to the class somewhere, it doesn't seem to have the later option but the former makes little sense [19:08:08] I'm not sure what you mean [19:08:12] Ooooh... or do I have to make a group for each set? [19:08:14] which puppet groups thing? [19:08:20] yeah [19:08:34] oh [19:08:35] Like is group == class not group == something you extend on the node [19:08:43] ohhhhhhhh [19:08:45] a group is just a grouping in the interface [19:08:50] it has nothing to do with puppet [19:08:50] robots.txt is an alias to /w/robots.php !! [19:09:01] hmm [19:09:11] <^demon> hashar: Magic :) [19:09:18] Damianz: the variables and classes are defined in the node [19:09:25] ^demon: really? :-D [19:09:27] <^demon> Cuz it combines site-wide robots.txt with site-specific mediawiki:robots.txt [19:09:30] <^demon> wmf fun hack :) [19:09:33] indeed [19:09:36] it's a great hack [19:09:45] at least now I know where it is :-] [19:09:50] and can rewrite it for 'beta' [19:09:55] So... if I make a group, with the vars I need, then configure the node that will end up in the node in puppet... well via some enc magic anyway [19:10:14] It's bad that I understand the puppet file more than your interface to it ;) [19:10:15] Damianz: the classes and variables go into ldap [19:10:27] Damianz: puppet uses ldap as an external node [19:10:33] external node classifier [19:10:53] Yeah [19:11:59] Say class x has variables z, t & e - if I make class x , variables z, t & e then want to add class u with no options do I create u in a seperate group? If not the interface seems to add the vars to the second class [19:12:06] If that makes sense at all [19:15:21] <^demon> Ryan_Lane: Did you get to the gerrit manifest thing last week? If not that's totally cool. [19:15:30] nope [19:15:43] I'll get to it this week [19:16:01] I'm still kind of in a mental fog from travel [19:16:04] Ryan_Lane: about cluster, could some of issue be fixed by the gluster team ? [19:16:07] <^demon> Oh sweet. There's no impending rush since 2.5 is still under "maybe we should branch now" discussion :) [19:16:10] they could get some instances on our labs :-] [19:16:17] hashar: well, we need to upgrade gluster [19:16:30] <^demon> Ryan_Lane: I hear that. I got back Sunday with my voice lost, a twisted knee, and both feet horribly swollen from running around all week. [19:16:38] rough [19:17:14] ^demon: That sounds like a strang saturday night [19:17:21] strange* [19:17:25] <^demon> Saturday was fun :p [19:17:31] I'm trying to abstract the nova/ec2 apis and my mind simply isn't working [19:17:46] <^demon> Friday is what nearly killed me ;-) [19:17:50] lol [19:18:09] hmm seems like home time, might as well shuffle to the other room [19:18:25] <^demon> Ryan_Lane: I got through e-mails, trying to be more productive but it's hard. [19:20:47] !log deployment-prep updated AFTv5: f97811f..d3bd97f [19:20:48] Logged the message, Master [19:22:51] Ryan_Lane: Any idea why I would be able to run a remote ssh command on the command line without a password prompt, but when a ceph script does the same, it prompts me? [19:23:02] Ryan_Lane: Specifically, http://pastebin.com/FT2unqJi [19:23:19] I think this is not a ceph thing so much as ssh doing some clever security thing. [19:23:20] you likely have a forwarded agent [19:23:26] and it's likely launching a shell [19:23:32] which kills off your agent [19:23:48] OK, yes, I have a forwarded agent. [19:24:03] So if I want to do passphrase-free ssh between virt1001, virt1002 and virt1003, what do you recommend? [19:24:31] does mkcephfs do anything other than install config files? [19:25:01] it may be easier to just implement this in puppet [19:25:17] People in #ceph are strongly discouraging me from trying to just run the individual commands on each system. [19:25:32] I'd kinda like to see this work at least once 'by hand' before trying to automate things. [19:25:58] At the moment the issue isn't with mkcephfs (which works fine.) [19:26:07] The issue is getting the agents to talk to each other, subsequently. [19:26:23] I think. Right now I'm just trying to start them. [19:26:30] !log deployment-prep upgrading MediaWiki core 80fbb70..ef3132f [19:26:31] Logged the message, Master [19:26:52] ahhh [19:26:58] I also need to get wikibugs tested / updated [19:26:59] grmblblb [19:27:14] Ryan_Lane: I'm pretty impressed that you immediately appeared in #ceph when I mentioned it even though it's on a different server [19:27:25] heh [19:27:29] I was already in the channel ;) [19:27:30] Ah, guess you were there already [19:32:01] !log deployment-prep running git bisect of core 80fbb70..ef3132f [19:32:03] Logged the message, Master [19:33:21] 07/17/2012 - 19:33:21 - Creating a home directory for jaideraf at /export/keys/jaideraf [19:34:22] 07/17/2012 - 19:34:22 - Updating keys for jaideraf at /export/keys/jaideraf [19:38:59] !log deployment-prep beta broken by PAGEID magic word introduced with 0a7cf03 / I11d42ca7 {{gerrit|9858}} [19:39:01] Logged the message, Master [19:45:07] !log deployment-prep rebuilding l10n cache: mwscript rebuildLocalisationCache.php --wiki=aawiki --threads=2 [19:45:08] Logged the message, Master [19:51:20] !log deployment-prep 369 languages rebuilt out of 369 [19:51:21] Logged the message, Master [19:55:24] andrewbogott: based on the info in #ceph, it may be good to take the storage hit and split out two disks into a raid 1 [19:55:31] with the rest in a raid 10 [19:57:10] Ryan_Lane: Oh, the issue is writing data and journal to the same physical disk, rather than to the same volume? [19:57:18] yep [19:57:21] 07/17/2012 - 19:57:21 - Creating a home directory for avicennasis at /export/keys/avicennasis [19:57:24] makes sense then. [19:58:20] 07/17/2012 - 19:58:20 - Updating keys for avicennasis at /export/keys/avicennasis [20:06:06] Ryan_Lane: OK, I now have (at last!) a working cephfs mount. Anything you want to try, or want me to look for, before I melt this down and switch to a more reasonable drive config? [20:06:26] And/or: is there a disk performance benchmark you recommend? Or should I write my own? [20:09:04] andrewbogott: hm. performance tests aren't really my expertise [20:09:13] andrewbogott: we need to test especially for random access [20:09:14] iops [20:09:32] since that's the general io profile for virtualization [20:09:38] That's asher's bag, right? [20:09:48] yep [20:09:59] I know he and notpeter wanted to do some ceph testing [20:10:05] so maybe they'd be willing to help out [20:10:44] Isn't Asher's handle 'Asher'? [20:11:17] <^demon> binasher [20:12:15] ah! Hm… he isn't here either. [20:13:31] yeah, not online :( [20:13:39] Ryan_Lane: The other question is, what are we comparing our benchmarks to? Should I set up an equivalent gluster cluster to compare, or is NFS a candidate still, or... [20:13:45] I was hoping he'd be here to do the gerrit db move [20:13:57] gluster would be a good comparison [20:14:20] also vs local disks [20:14:47] ok. [20:14:50] I don't consider NFS a viable option [20:15:04] ok. I gotta get lunch [20:16:40] always the waiting with sveta. [20:16:43] heh [20:31:13] j^: helllooo :-] [20:32:53] j^: so you get to work on TimedMediaHandler ? :) [20:40:10] deployment-prep is meant to be a replication of the production cluster (somehow) [20:40:28] we thought about deploying new wikis with specific configuration, but nothing done so far [20:40:50] oh the transcoding [20:41:25] not at all [20:41:33] too complicated to set up for my little brain :/ [20:41:50] I tried to do that while I was in SF back in May and eventually failes [20:41:52] failed [20:42:09] * hashar IRC connection is lagged [20:45:00] I think we have a swift cluster in labs [20:45:07] probably don't want to use it though [20:45:14] so would need to set one up in deployment-prep [20:45:29] http://wikitech.wikimedia.org/view/Swift/Setup_New_Swift_Cluster_%28labs%29 [20:45:37] ah here is ben :) [20:45:46] have at it! [20:45:58] though Ithink it's ok to use the existing one if you prefer. [20:46:06] entirely up to you. [20:46:21] something j^ and you should discuss I guess [20:46:36] it would probably be better to use the existing one indeed [20:46:45] that will give maplebed some real traffic :-] [20:46:53] and let us forget about swift :-] [20:47:26] the job runner I would need to have https://gerrit.wikimedia.org/r/#/c/11610/ deployed [20:47:47] * maplebed runs away again [20:48:28] lets open bugzilla / rt request [20:48:49] j^: we have ton of bugs tracked in bugzilla. The main entry is https://bugzilla.wikimedia.org/showdependencytree.cgi?id=37079&hide_resolved=1 [20:48:58] feel free to add a tracking bug for TMH on beta [20:49:19] wgJobTypesExcludedFromDefaultQueue should exclude the transcoding job indeed [20:49:29] it had a few bugs I fixed few weeks ago [20:49:47] I think I got it deployed in our mediawiki config ( operations/mediawiki-config.git ) [20:50:04] CommonSettings.php [20:50:04] 2402:$wgJobTypesExcludedFromDefaultQueue[] = 'webVideoTranscode'; [20:50:06] indeed :) [20:50:38] great! [20:51:13] MediaWiki core had a bug that caused some infinite loop [20:51:14] https://bugzilla.wikimedia.org/show_bug.cgi?id=37072 [20:51:17] that is fixed though [20:54:23] oh really? [20:54:25] hmm [20:54:35] I know the video thumbnails are made by the main apaches [20:54:40] (which are basically doing anything [20:54:56] if transcoding happens, I have no idea which box / process do them :( [20:57:48] yup that one got corrupted two week ago [20:57:53] I am setting up to use varnish [20:58:16] j^ hashar transcoding was happening, then we broke it when we made beta commons env not a hack [20:58:22] something I have worked friday and today. Will hopefully have upload working again this week [20:58:27] mark is helping me on that :) [20:58:41] (well to be correct: mark is reviewing / giving instructions :D ) [20:59:15] chrismcmahon: maybe I should have started an entirely new cluster instead of hacking the existing :( [20:59:34] https://bugzilla.wikimedia.org/show_bug.cgi?id=38455 - wikimedia-job-runner should be able to run a specific job type [20:59:50] hashar: it wouldn't have helped :-) this one is getting really good [21:00:08] well it is getting closer and closer to the production cluster!! [21:02:45] conf call again [21:08:04] hashar are you taking !b 38449 or I should [21:08:20] !b 38449 [21:08:20] https://bugzilla.wikimedia.org/38449 [21:08:39] petan: go ahead! :-) [21:08:46] make sure to make the change in CommonSettings-wmflabs.php [21:08:54] right [21:09:01] and have it through Gerrit :-] [21:09:18] I have updated AFTv5 extension an hour ago [21:09:21] as well as Core [21:09:23] (see SAL) [21:09:24] ok [21:09:44] so it is about figuring out the $wgArticleFeedback* settings that are needed to enabled it on all page [21:10:19] Reedy / Aaron will be able to merge your change to operations/mediawiki-config.git [21:10:25] then just git pull and hopefully it will work [21:10:39] note that we still have ton of local hacks in /home/wikipedia/common :-( [21:10:44] like the .dblist files [21:10:52] or the extension-something file [21:10:53] ;:( [21:11:03] are you able to login to deployment-bastion? [21:11:08] nop [21:11:11] never managed to [21:11:15] ok I will delete it then [21:11:16] I use -dbdump [21:11:19] yup [21:12:38] conf call for real then I will head to bed. see you later [21:12:50] !log deployment-prep petrb: updating ArticleFeedbackv5 extension [21:12:52] Logged the message, Master [21:13:22] hm... upto date [21:13:51] yup did it an hour or so ago [21:13:56] as well as core :-] [21:14:09] rest of extension are not updated though [21:14:31] ok [21:20:22] 07/17/2012 - 21:20:22 - Creating a home directory for spage at /export/keys/spage [21:21:21] 07/17/2012 - 21:21:21 - Updating keys for spage at /export/keys/spage [21:26:25] why does production have /mnt/upload6 ? [21:26:26] Is it equivalent to labs /mnt/upload ? [21:26:38] hashar [21:26:51] that extension is already installed, wondering what is needed to do [21:26:55] !log deployment-prep installed python-imaging and wamerican on deployment-dbdump [21:26:56] /mnt/upload on labs is a symlink [21:26:57] Logged the message, Master [21:26:59]