[00:21:51] <labs-logs-bottie>	 !log bots gifti: Installed fcron on bots-4
[00:21:53] <labs-morebots>	 Logged the message, Master
[09:10:51] <hashar>	 Platonides: are you around by any chance? The captcha are broken on labs and always fail. I suspect it is because we do not have a wgCaptchaSecret variable set and you generated the captcha using a key
[09:21:49] <hashar>	 !log deployment-prep On dbdump, regenerating captcha using: python php-master/extensions/ConfirmEdit/captcha.py --wordlist=/usr/share/dict/words --font=/usr/share/fonts/truetype/ttf-dejavu/DejaVuSansMono.ttf --key=********* --output=/mnt/upload/private/captcha --count=1000
[09:21:51] <labs-morebots>	 Logged the message, Master
[09:22:31] <hashar>	 !log deployment-prep Set $wmgCaptchaSecret in the local file common/wmf-config/PrivateSettings.php and used that value in captcha.py
[09:22:32] <labs-morebots>	 Logged the message, Master
[09:24:55] <hashar>	 !log deployment-prep forgot to set a directory level. python php-master/extensions/ConfirmEdit/captcha.py --wordlist=/usr/share/dict/words --font=/usr/share/fonts/truetype/ttf-dejavu/DejaVuSansMono.ttf --key=******* --output=/mnt/upload/private/captcha --count=1000 --dirs=3
[09:24:56] <labs-morebots>	 Logged the message, Master
[09:26:04] <hashar>	 !log deployment-pre Deleted all captchas in /mnt/upload6/private/captcha and regenerate them using 1000 and dirs=3
[09:26:04] <labs-morebots>	 deployment-pre is not a valid project.
[09:26:52] <hashar>	 !log deployment-prep Deleted all captchas in /mnt/upload6/private/captcha and regenerate them using 1000 and dirs=3
[09:26:54] <labs-morebots>	 Logged the message, Master
[09:35:52] <hashar>	 !log deployment-prep Regenerating captcha with the new shared key fixed {{bug|38699}}
[09:35:54] <labs-morebots>	 Logged the message, Master
[09:41:11] <giftpflanze>	 sigh
[09:42:04] <hashar>	 Platonides: I have regenerated the captcha, seems to work now :-]
[09:42:11] <hashar>	 giftpflanze: what is happening?
[09:42:43] <giftpflanze>	 i've again messed up
[10:35:49] <Nikerabbit>	 I'm trunning puppetd manually but I get an error:
[10:35:51] <Nikerabbit>	 err: /Stage[main]/Certificates::Wmf_ca/Exec[/bin/ln -s /etc/ssl/certs/wmf-ca.pem /etc/ssl/certs/$(/usr/bin/openssl x509 -hash -noout -in /etc/ssl/certs/wmf-ca.pem).0]/returns: change from notrun to 0 failed: /bin/ln -s /etc/ssl/certs/wmf-ca.pem /etc/ssl/certs/$(/usr/bin/openssl x509 -hash -noout -in /etc/ssl/certs/wmf-ca.pem).0 returned 1 instead of one of [0] at /etc/puppet/manifests/certs.pp:198
[10:35:58] <Nikerabbit>	 any ides?
[10:38:14] <Damianz>	 Someone else had that issue yesterday hmmm
[10:39:23] <Damianz>	 require => File["/etc/ssl/certs/wmf-ca.pem"]; is out of source => "puppet:///files/ssl/wmf-ca.pem",
[10:40:29] <Damianz>	 files/ssl/wmf-ca.pem totally exists though
[10:41:27] <hashar>	 Nikerabbit: I have opened a bug about that one
[10:41:36] <hashar>	 Nikerabbit: that happens on all Precise instances
[10:41:57] <Damianz>	 Yeah it breaks on my desktop that's running precise :(
[10:42:19] <hashar>	 Nikerabbit: oh no it is just in the ops-l mailing list.
[10:43:01] <hashar>	 Nikerabbit: anyway the reason is that we have hardcoded the openssl hash in puppet to find out which file is generated  (something like /etc/ssl/certs/12345ABC ).0
[10:43:22] <hashar>	 Nikerabbit: the algorithm was changed with OpenSSL 1.0 which is provided by Precise so Puppet think that the file is missing
[10:43:29] <hashar>	 Nikerabbit: long story short, you can safely ignore that message :)
[10:43:51] <hashar>	 Nikerabbit: I forwarded you my email to ops-l
[10:53:43] <Nikerabbit>	 hashar: thanks
[13:10:35] <hashar>	 !log deployment-prep Manually running puppet for {{gerrit|15545}} which should fix  {{bug|38084}} uses /data/project instead of NFS instance
[13:10:37] <labs-morebots>	 Logged the message, Master
[13:14:21] <hashar>	 !log deployment-prep stopping apache backends
[13:14:22] <labs-morebots>	 Logged the message, Master
[13:14:40] * Damianz  hugs his backend

[13:15:15] <hashar>	 !log depoyment-prep root@deployment-nfs-memc:~# rsync -a --progress --delete --inplace /mnt/export/upload6 /data/project
[13:15:15] <labs-morebots>	 depoyment-prep is not a valid project.
[13:15:49] <hashar>	 !log deployment-prep manually umount /mnt/upload6 on apaches 
[13:15:50] <labs-morebots>	 Logged the message, Master
[13:19:56] <hashar>	 paravoid: so that is slowly syncing the upload files from the old instance to /data/project
[13:20:06] <hashar>	 I am fixing the /mnt/upload6 on the apaches right now
[13:20:10] <paravoid>	 okay
[13:20:18] <paravoid>	 I guess I can merge it into prod now
[13:20:22] <paravoid>	 and see if it'll have any effect
[13:21:02] <hashar>	 most probably
[13:22:40] <hashar>	 !log deployment-prep applying nfs::upload::labs  to -dbdump so it uses /data/project/upload6 at  /mnt/upload6
[13:22:41] <labs-morebots>	 Logged the message, Master
[13:23:23] <hashar>	 grrmblbl need to migrate the apache confs too :-D
[13:24:01] <Nikerabbit>	 I've set EDITOR=nano but git commit still opens vi, what could be the issue?
[13:24:48] <hashar>	 are you sure git uses EDITOR ?  maybe it is a git config ?   Are you sure EDITOR is set to nano ?  (echo $EDITOR )
[13:25:01] <Nikerabbit>	 http://digitaldreamer.net/blog/2010/4/7/setting-default-editor-git-commit-messages/
[13:25:03] <Jens_WMDE>	 Nikerabbit: try git config --global core.editor "nano"
[13:25:20] <Nikerabbit>	 hah found it
[13:25:27] <Nikerabbit>	 i had EDITOR=nano in my bashrc
[13:25:51] <Nikerabbit>	 for some reason it doesn't work unless it is prefixed with export 
[13:26:00] <Jens_WMDE>	 of course you could always learn the way of vi, but...
[13:26:11] <hashar>	 yup without export it is kind of local to the current line
[13:26:19] <hashar>	 export write it to the general env I think
[13:26:33] <hashar>	 Jens_WMDE: or use emacs, it comes with the evim editor
[13:26:55] <Jens_WMDE>	 'git config --global core.editor "nano"' is really the way to set the editor for git...
[13:27:22] <Nikerabbit>	 I prefer not to configure it for each program separately
[13:28:32] <Jens_WMDE>	 okay okay
[13:29:40] <hashar>	 !log deployment-prep root@deployment-nfs-memc:/mnt/export# rsync -a --progress --delete --inplace /mnt/export/apache /data/project
[13:29:41] <labs-morebots>	 Logged the message, Master
[13:29:54] <hashar>	 paravoid: so that is rsyncing, the change seems right to me :)
[13:30:51] <paravoid>	 great
[13:32:08] <hashar>	 !log deployment-prep on -dbdump, unmounted /mnt/upload (from nfs-memc). Please use the /mnt/upload6 -> /data/project/upload6
[13:32:10] <labs-morebots>	 Logged the message, Master
[13:32:29] <hashar>	 paravoid: there is also my hack to prevents dbdump to have the remote rsyslog thing
[13:32:39] <hashar>	 paravoid: need it so I can set up syslog-ng there :)
[13:34:23] <hashar>	 !log deployment rsync from nfs-memc:/mnt/export/upload6 to /data/project/upload6 completed. YEAHHH
[13:34:23] <labs-morebots>	 deployment is not a valid project.
[13:35:37] <hashar>	 !log deployment-prep rsync from nfs-memc:/mnt/export/upload6 to /data/project/upload6 completed. YEAHHH
[13:35:39] <labs-morebots>	 Logged the message, Master
[13:46:04] <hashar>	 !log deployment-prep rsync finished for both apache and upload6. Remounting and restarting apaches
[13:46:06] <labs-morebots>	 Logged the message, Master
[13:46:16] <hashar>	 !log seems to work out of /data/project now!!!
[13:46:16] <labs-morebots>	 seems is not a valid project.
[13:46:50] <hashar>	 !log deployment-prep seems to work fine with /data/project now. 
[13:46:52] <labs-morebots>	 Logged the message, Master
[13:46:53] <hashar>	 paravoid: http://en.wikipedia.beta.wmflabs.org/?IhateCache  !!!
[13:46:58] <hashar>	 served by /data/project yeahhhh
[13:47:10] <hashar>	 I will actually enjoy my week-end :-D  thanks for that!
[13:48:47] <paravoid>	 I don't hate cache
[13:48:48] <paravoid>	 :P
[13:50:52] <hashar>	 :-D
[13:56:08] <hashar>	 !log deployment-prep archiving -nfs-memc:/mnt/export : root@deployment-nfs-memc:/mnt# mv export /data/project/deployment-nfs-memc_mnt-export_backup 
[13:56:09] <labs-morebots>	 Logged the message, Master
[13:56:25] <hashar>	 I think I will be able to get ride of that NFS instance!
[14:57:22] <labs-home-wm>	 07/26/2012 - 14:57:21 - Creating a home directory for gbruin at /export/keys/gbruin
[14:58:22] <labs-home-wm>	 07/26/2012 - 14:58:22 - Updating keys for gbruin at /export/keys/gbruin
[16:12:34] <hashar>	 !log deployment-prep -dbdump umounted deployment-nfs-memc:/mnt/export/apache on /usr/local/apache
[16:12:35] <labs-morebots>	 Logged the message, Master
[16:15:10] <hashar>	 !log deployment-prep applying nfs::apache::labs on -dbdump to get /usr/local/apache from /data/project
[16:15:12] <labs-morebots>	 Logged the message, Master
[16:22:06] <alchimista>	 can anyone help me with some git stuff? I've cloned a repo with git clone, then updated my local repo with git pull, and now that changes are made, i just have to make git review, am i correct?
[16:57:33] <jarry1250>	 Okay, so I can ssh to bastion, from there to my project, but then if I try to clone from gerrit I get a simple permission denied error...
[16:57:36] <jarry1250>	 Anything obvious?
[16:57:50] <jarry1250>	 I'm using -A, obviously
[17:01:58] <Platonides>	 alchimista, correct
[17:05:43] <jarry1250>	 nvm, copied my keys to my project /home/~ssh
[17:13:44] <alchimista>	 Platonides: i got this error: Errors running git rebase -i remotes/gerrit/production
[17:13:57] <alchimista>	 Cannot rebase: You have unstaged changes
[17:14:08] <alchimista>	 Please commit or stash them
[17:29:40] <Ryan_Lane>	 sec
[17:31:20] <Ryan_Lane>	 The disk drive for /usr/local/apache is not ready yet or not present.^M^M                                             
[17:31:20] <Ryan_Lane>	 Continue to wait, or Press S to skip mounting or M for manual recovery^M^M                                            
[17:31:26] <Ryan_Lane>	 is that an nfs mount?
[17:31:34] <Ryan_Lane>	 this is why I hate nfs fstab mounts
[17:31:45] <Ryan_Lane>	 on which instance?
[17:31:46] <Ryan_Lane>	 it's not working
[17:31:55] <Damianz>	 Ryan_Lane: 
[17:31:57] <Ryan_Lane>	 if it is now, then you'll want to reboot this instance again
[17:32:03] <Damianz>	 Err, ignore that I was just seeing if you where here 
[17:32:05] <Ryan_Lane>	 heh
[17:32:12] <Ryan_Lane>	 I had just pm'd you ;)
[17:32:33] <Damianz>	 Yeah I was figuring out if that was from today or yesterday, totally been ignoring irc lately :P
[17:33:47] <Ryan_Lane>	 j^: showmount -e deployment-nfs-memc
[17:42:19] <Ryan_Lane>	 that would be a problem
[17:42:22] <Ryan_Lane>	 make the directory
[17:42:27] <Ryan_Lane>	 then get the system to boot
[17:42:29] <Ryan_Lane>	 then fix the problem
[17:42:39] <Ryan_Lane>	 whether its the mount or the server
[17:51:50] <Damianz>	 don't think so
[17:52:27] <Damianz>	 Is there any reason why we can't just kill nfs on beta?
[17:52:42] <Damianz>	 I assume it's mostly hard coded paths to stuff and we use a set automount path for gluster
[17:58:50] <Ryan_Lane>	 !resource deployment-prep
[17:58:50] <wm-bot>	 https://labsconsole.wikimedia.org/wiki/Nova_Resource:deployment-prep
[17:59:14] <Ryan_Lane>	 looks like hashar made some changes
[17:59:29] <Ryan_Lane>	 hooray for logs :D
[18:01:02] <MaxSem>	 does mediawiki::sync work on Labs?
[18:04:30] <adminxor>	 Can anyone help me to select an instance type which includes X (with gnome/kde) by default? 
[18:04:34] <adminxor>	 I just need to fire up a browser from that instance. Nothing more than that.
[18:05:28] <MaxSem>	 I don't think it's a good idea
[18:07:05] <MaxSem>	 adminxor, see https://labsconsole.wikimedia.org/wiki/Help:Access#Accessing_services_using_port_forwarding
[18:08:43] <adminxor>	 MaxSem: Thank you! :D
[18:17:09] <Ryan_Lane>	 adminxor: yeah, use port forwarding or socks proxy
[18:35:47] <MaxSem>	 does mediawiki::sync work on Labs?
[18:42:21] <labs-logs-bottie>	 !log deployment-prep j: new instance deployment-video02 with videoscaler with access to gluster instead of nfs
[18:42:23] <labs-morebots>	 Logged the message, Master
[19:27:31] <labs-logs-bottie>	 !log deployment-prep j: delete broken deployment-video01 instance
[19:27:32] <labs-morebots>	 Logged the message, Master
[19:59:23] <labs-home-wm>	 07/26/2012 - 19:59:23 - Updating keys for gbruin at /export/keys/gbruin
[20:01:23] <labs-home-wm>	 07/26/2012 - 20:01:23 - Updating keys for gbruin at /export/keys/gbruin
[20:02:31] <andrewbogott>	 Ryan_Lane:  I'm behind, and also lost my IRC backscroll.  You declared victory last night, but I'm not sure what you were victorious over… working keystone?
[20:02:37] <andrewbogott>	 working essex?  Both?
[20:11:44] <Ryan_Lane>	 andrewbogott: both
[20:11:47] <andrewbogott>	 Ryan_Lane:  Also, does your keystone patch mean you're refitting keystone to handle our existing ldap schema?
[20:11:54] <Ryan_Lane>	 nope
[20:11:59] <andrewbogott>	 Ryan_Lane:  \o/
[20:12:00] <Ryan_Lane>	 we're going to migrate schema
[20:12:11] <andrewbogott>	 ok.  I have still not looked at that at all :(
[20:12:18] <Ryan_Lane>	 I already did it :)
[20:12:22] <andrewbogott>	 better yet!
[20:12:33] <Ryan_Lane>	 but, there were bugs in keystone
[20:12:35] * andrewbogott  stands back, nods approvingly

[20:12:41] <Ryan_Lane>	 ayoung and I fixed them all just now :)
[20:12:44] <paravoid>	 hi
[20:12:48] <Ryan_Lane>	 paravoid: howdy
[20:13:07] <paravoid>	 it's late and I shouldn't be working, but this sounds interesting :)
[20:14:34] <Ryan_Lane>	 heh
[20:14:45] <Ryan_Lane>	 oh, well, I just have openstack essex working on nova-precise1
[20:14:50] <Ryan_Lane>	 with keystone using the ldap backend
[20:15:13] <Ryan_Lane>	 it required a few code changes, which have just been pushed into origin/stable/essex branch
[20:15:20] <Ryan_Lane>	 so, hopefully ubuntu will pick them up and backport
[20:15:29] <Ryan_Lane>	 well, not backport, but add to their packages
[20:16:08] <Ryan_Lane>	 seems the hardcoded policy for api actions is now in a policy configuration file, too
[20:16:34] <Ryan_Lane>	 which unfortunately was completely unimplemented. so, I've added roles to functions I think are appropriate
[20:16:53] <Ryan_Lane>	 paravoid: did you get a chance to put in a nomination? :)
[20:17:19] <andrewbogott>	 I thought there was a default policy file in the essex tree… maybe I'm thinking of something else.
[20:17:35] <Ryan_Lane>	 andrewbogott: there is one, but it lets project members basically do everything
[20:17:47] <Ryan_Lane>	 roles are totally free-form in keystone, though
[20:17:56] <Ryan_Lane>	 so I guess it makes sense that they made it that way
[20:18:15] <paravoid>	 I haven't yet, no
[20:18:19] <andrewbogott>	 Yeah, I thought you meant you had to write an all-new file from scratch.
[20:18:20] <Ryan_Lane>	 ok
[20:18:36] <Ryan_Lane>	 I'm poking because nominations are required by Aug 6th
[20:18:47] <andrewbogott>	 There's talk on the list about changing the nomination process, although there are no specifics.
[20:18:51] <Ryan_Lane>	 I think I'll likely get the required 10 today, though
[20:19:01] <Ryan_Lane>	 yeah
[20:19:05] <andrewbogott>	 I wouldn't think they'd de-nominate anyone though.
[20:19:08] <Ryan_Lane>	 I don't think they'll change by Aug 6
[20:19:14] <paravoid>	 change it how?
[20:19:23] <Ryan_Lane>	 make it more transparent
[20:19:27] <Ryan_Lane>	 right now it's done by email
[20:19:34] <Ryan_Lane>	 because it's kind of a rush job
[20:19:41] <paravoid>	 I was about to say this exact thing
[20:19:56] <paravoid>	 that it must suck to have to mail people "please send an email there"
[20:20:02] <Ryan_Lane>	 yep
[20:20:11] <Ryan_Lane>	 they'll have an actual system next time
[20:20:33] <paravoid>	 actually, I'm not sure if see the point of having supporters for nominations
[20:20:41] <Ryan_Lane>	 they are posting the nominees: http://www.openstack.org/community/openstack-foundation-board-2012-election-candidates/
[20:21:01] <paravoid>	 why block a nomination?
[20:21:16] <Ryan_Lane>	 they don't want 1000 people on the ballot
[20:23:11] <Ryan_Lane>	 29 people already on the nominations list
[20:23:31] <paravoid>	 you're there too
[20:23:43] <Ryan_Lane>	 I got 4 nominations last night at dinner. heh
[20:24:41] <Ryan_Lane>	 that's everyone who is nominated, not necessarily who will be on the ballot
[20:32:00] <Damianz>	 Ryan_Lane: Ooh shiny
[23:00:21] <Ryan_Lane>	 well shit
[23:00:30] <Ryan_Lane>	 I've somehow broken libvirt on virt2
[23:00:31] <Danny_B|backup>	 that sounds sreious
[23:00:44] <Ryan_Lane>	 ah
[23:00:45] <Ryan_Lane>	 nevermind
[23:00:51] <Ryan_Lane>	 restarting libvirt fixed it :)
[23:07:23] <labs-home-wm>	 07/26/2012 - 23:07:22 - Updating keys for mwalker at /export/keys/mwalker
[23:38:57] <wm-bot>	 Change on 12mediawiki a page OAuth was modified, changed by Dantman link https://www.mediawiki.org/w/index.php?diff=565792 edit summary: /* See also */ 
[23:40:14] <wm-bot>	 Change on 12mediawiki a page OAuth was modified, changed by Dantman link https://www.mediawiki.org/w/index.php?diff=565793 edit summary: /* Previous discussions */ 
[23:46:04] <sumanah>	 Ryan_Lane: I saw a hint that you're speaking at an OpenStack meetup sometime in the next few weeks?
[23:46:12] <Ryan_Lane>	 I am?
[23:46:23] <Ryan_Lane>	 I'll probably speak at the upcoming conference
[23:50:48] <sumanah>	 Ryan_Lane: I thought I saw something in a shared calendar - maybe you're attending something or hosting something, not speaking?
[23:51:03] <Ryan_Lane>	 can you please point me to what you're talking about?
[23:51:25] <sumanah>	 sure, just a moment
[23:52:47] <Ryan_Lane>	 oh. do you mean a possible gerrit meetup in the office?
[23:53:29] <sumanah>	 ARGH, not OpenStack.  Actually I was thinking of 8 August, CodeChix
[23:53:38] <NotDemonSsshhh>	 Ryan_Lane: Speaking of Gerrit meetup, I pinged Shawn today to try and get a firm answer. No response :(
[23:53:41] <Ryan_Lane>	 am I scheduled to talk then?
[23:53:45] <Ryan_Lane>	 heh
[23:53:59] <Ryan_Lane>	 ah. it's tentative
[23:54:06] <Ryan_Lane>	 I'm not totally sure why that's there
[23:54:14] <sumanah>	 it says tentative
[23:54:36] <sumanah>	 so, http://www.codechix.org/ says nothing about it
[23:54:39] <Ryan_Lane>	 yeah
[23:54:44] <Ryan_Lane>	 I think that's not accurate
[23:54:52] <Ryan_Lane>	 I think we discussed maybe doing it on this date
[23:54:52] <sumanah>	 given that it's < 2 weeks away, I figure you're not actually doing it if it's not settled by now
[23:54:54] <sumanah>	 ok
[23:55:14] <Ryan_Lane>	 heh. I was worried for a second :)
[23:55:17] <sumanah>	 and I was conflating it with the OpenStack stuff in October!
[23:55:25] * Ryan_Lane  nods

[23:56:56] <sumanah>	 I was writing a note to people who attended the SF hackathon to tell them about other upcoming events and I was trying to trace this one. Will leave it out
[23:58:36] <Ryan_Lane>	 ok. cool
[23:59:10] <sumanah>	 unless there's anything else you want them to know about, Ryan_Lane