[00:24:22] <labs-home-wm>	 07/18/2012 - 00:24:22 - Updating keys for helderwiki at /export/keys/helderwiki
[03:36:57] <labs-home-wm>	 07/18/2012 - 03:36:57 - Created a home directory for rocsteady in project(s): bastion,puppet-cleanup
[03:37:19] <labs-home-wm>	 07/18/2012 - 03:37:19 - Creating a home directory for rocsteady at /export/keys/rocsteady
[03:37:59] <labs-home-wm>	 07/18/2012 - 03:37:59 - User rocsteady may have been modified in LDAP or locally, updating key in project(s): bastion,puppet-cleanup
[03:38:21] <labs-home-wm>	 07/18/2012 - 03:38:20 - Updating keys for rocsteady at /export/keys/rocsteady
[03:54:48] <r0csteady>	 Roan: I am at the puppet-labs party! 
[06:27:32] <r0csteady>	 Guys: I am curious, why are we using gerrit?
[06:31:24] <r0csteady>	 I just want to understand..
[06:31:29] <r0csteady>	 <<<<<<newbie
[07:45:03] <hashar>	 hello
[08:02:19] <labs-home-wm>	 07/18/2012 - 08:02:19 - Updating keys for shantanoo at /export/keys/shantanoo
[08:26:09] <hashar>	 !log deployment-prep Created deployment-integration to be used as a puppetmaster::self host
[08:26:10] <labs-morebots>	 Logged the message, Master
[08:59:37] <hashar>	 !log deployment-prep Applying {{gerrit|15545}} to deployment-integration
[08:59:38] <labs-morebots>	 Logged the message, Master
[09:02:52] <hashar>	 !log deployment-prep adding nfs::apache::labs and nfs::upload::labs to deployment-integration
[09:02:53] <labs-morebots>	 Logged the message, Master
[09:12:21] <hashar>	 err: /Stage[main]/Nfs::Apache::Labs/Mount[/usr/local/apache]: Could not evaluate: Execution of '/bin/mount -o defaults,_netdev=eth0,log-level=WARNING,log-file=/var/log/gluster.log /usr/local/apache' returned 1: Mount failed. Please check the log file for more details.
[09:12:23] <hashar>	 grmblblblblbl
[09:12:28] <hashar>	 while debug: Puppet::Type::Mount::ProviderParsed: Executing '/bin/mount -o defaults,_netdev=eth0,log-level=WARNING,log-file=/var/log/gluster.log /usr/local/apache'
[09:13:06] <hashar>	 failed to fetch volume file (key:/{deployment-prep}/upload6)
[09:13:07] <hashar>	 grmblblbl
[10:22:36] <hashar>	 !log deployment-prep copying apache dir to /data/project . Run as root@deployment-nfs-memc in a screen  session
[10:22:38] <labs-morebots>	 Logged the message, Master
[10:22:41] <hashar>	 out for lunch
[11:03:46] <petan|wk>	 !gerrit
[11:03:46] <wm-bot>	 https://gerrit.wikimedia.org/
[11:54:32] <hashar>	 re
[12:02:53] <petan|wk>	 !ping
[12:02:54] <wm-bot>	 pong
[12:06:02] <hashar>	 pong
[12:11:25] <hashar>	 paravoid: I got the change using symlink :-]
[12:11:26] <hashar>	 https://gerrit.wikimedia.org/r/#/c/15545/
[13:08:33] <hashar>	 !log deployment-prep deployment-cache-upload01 : running apt-get upgrade / dist-upgrade and rebooting
[13:08:35] <labs-morebots>	 Logged the message, Master
[13:24:40] <hashar>	 !log deployment-cache-upload01 seems screwed : waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id
[13:24:40] <labs-morebots>	 deployment-cache-upload01 is not a valid project.
[13:24:40] <hashar>	   failed dhcp acquisition ? => rebooting.
[13:44:24] <hashar>	 !log deployment-prep deployment-cache-upload01 seems screwed : waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id . Failed DHCP acquisition ? => rebooting
[13:44:26] <labs-morebots>	 Logged the message, Master
[13:46:23] <hashar>	 !log deployment-prep creating deployment-cache-upload03 to replace upload01
[13:46:25] <labs-morebots>	 Logged the message, Master
[13:46:34] <hashar>	 !log deployment-prep deleting upload01 (screwed somehow)
[13:46:36] <labs-morebots>	 Logged the message, Master
[14:38:36] <hashar>	 !log deployment-prep New / rebooted instances are no more accessible : {{bug|38473}} - instances can not boot / reboot anymore
[14:38:38] <labs-morebots>	 Logged the message, Master
[14:45:30] <petan|wk>	 hashar all instances or new only?
[14:45:39] <petan|wk>	 I changed the name of bug a bit
[14:45:47] <hashar>	 petan|wk: I can't reboot instances any more
[14:45:55] <petan|wk>	 all of them?
[14:46:01] <petan|wk>	 or only new
[14:46:14] <hashar>	 that message appear on boot
[14:46:20] <hashar>	 so that is any instance starting
[14:46:27] <petan|wk>	 I mean sometimes it crash when you make ne
[14:46:32] <hashar>	 (either a created one or rebooting an existing one)
[14:46:35] <hashar>	 trust me
[14:46:35] <petan|wk>	 you just need to delete it and create again
[14:46:36] <hashar>	 :)
[14:46:44] <petan|wk>	 aha
[14:47:20] <hashar>	 changed it back with a comment :-D
[14:47:47] <petan|wk>	 ok
[14:47:49] <petan|wk>	 :)
[14:58:01] <hashar>	 that might be related to virt1 apparently flacking
[14:58:20] <hashar>	 or that is just nagios :-D
[14:58:38] <hashar>	 petan|wk: also our nagios is broken : provideCheckTitleEncodingData
[14:58:44] <hashar>	 http://nagios.wmflabs.org/cgi-bin/nagios3/status.cgi?hostgroup=deployment-prep&style=detail
[14:58:54] <petan|wk>	 yes it is
[14:59:06] <petan|wk>	 I will try to get mutante and sort it out
[18:17:25] <labs-home-wm>	 07/18/2012 - 18:17:25 - Updating keys for svemir at /export/keys/svemir
[19:07:45] <hashar>	 Ryan_Lane: I did some changes to OpenStackManager extension at https://gerrit.wikimedia.org/r/#/q/status:open+project:mediawiki/extensions/OpenStackManager,n,z
[19:07:55] <hashar>	 all 3 are dependents
[19:08:08] <hashar>	 mostly doing some cleanup / factoring out in Special:NovaInstance
[19:08:53] <Damianz>	 Hmm
[19:09:11] <Damianz>	 I wonder if the ldap extension is in gerrit now so I can apply my diff from bz and moan about it again
[19:09:21] <Ryan_Lane>	 yes
[19:09:22] <Ryan_Lane>	 please do
[19:10:44] <hashar>	 well I am out for now, heading bed :-D
[19:10:49] <hashar>	 have a good afternoon!
[19:39:09] <labs-home-wm>	 07/18/2012 - 19:39:09 - Created a home directory for j in project(s): swift
[19:39:30] <hashar>	 j^: need a varnish instance
[19:39:43] <hashar>	 j^: I got  deployment-cache-upload something for it
[19:39:50] <hashar>	 then after lunch the labs died :-D
[19:40:06] <labs-home-wm>	 07/18/2012 - 19:40:06 - User j may have been modified in LDAP or locally, updating key in project(s): swift
[19:40:08] <hashar>	 so I have just been reading  manifests/varnish.pp and figuring out how the varnish upload cache works in production
[19:40:24] <hashar>	 j^: will have to setup some class for our upload whenever labs is up again (should not last long)
[19:40:26] <Ryan_Lane>	 I'm still tracking down problems in labs
[19:40:37] <Ryan_Lane>	 seems there's some inconsistency in the database
[19:40:48] <Ryan_Lane>	 I fixed that
[19:40:55] <Ryan_Lane>	 but the api service is still taking years to respond
[19:42:41] <hashar>	 j^: I will probably end up writing a very simple upload instance just for beta instead of replicating the production setup (it uses two layers of cache as I understand it)
[19:43:41] <hashar>	 j^: the puppet class should support load balancing anyway 
[22:11:59] <labs-home-wm>	 07/18/2012 - 22:11:58 - Created a home directory for spage in project(s): editor-engagement
[22:12:59] <labs-home-wm>	 07/18/2012 - 22:12:59 - User spage may have been modified in LDAP or locally, updating key in project(s): editor-engagement
[22:58:33] <ori-l>	 How do I requisition a public IP for an instance? labsconsole fails with "Failed to allocate new public IP address."
[23:01:34] <RoanKattouw>	 ori-l: You need to ask Ryan_Lane to assign a public IP to your project
[23:01:46] <Ryan_Lane>	 which project, and for which purpose? :)
[23:01:47] <RoanKattouw>	 Then once the project has a non-empty pool of IP addresses, you can assign stuff
[23:14:32] <ori-l>	 RoanKattouw: ah, thanks
[23:14:52] <ori-l>	 Ryan_Lane: e3, for testing ui/ux changes
[23:15:10] <Ryan_Lane>	 do non-labs people need to access it?
[23:15:15] <ori-l>	 Ryan_Lane: yeah.
[23:15:17] <Ryan_Lane>	 ok
[23:15:23] <ori-l>	 Ryan_Lane: all the product people on the team
[23:15:49] <Ryan_Lane>	 ok. I upped your quota
[23:15:52] <ori-l>	 thanks
[23:36:50] <wm-bot>	 Change on 12mediawiki a page Wikimedia Labs was modified, changed by Skierpage link https://www.mediawiki.org/w/index.php?diff=563174 edit summary: /* Communications */ use {{irc}} template