[00:05:05] <gerrit-wm>	 New patchset: Ryan Lane; "More followup to Ida74297b" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6356
[00:05:18] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6356
[00:05:47] <gerrit-wm>	 New patchset: Ryan Lane; "Adding proxyagent pass for keystone" [labs/private] (master) - https://gerrit.wikimedia.org/r/6357
[00:06:08] <gerrit-wm>	 New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/6356
[00:06:11] <gerrit-wm>	 Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6356
[00:06:26] <gerrit-wm>	 New review: Ryan Lane; "(no comment)" [labs/private] (master); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/6357
[00:06:28] <gerrit-wm>	 Change merged: Ryan Lane; [labs/private] (master) - https://gerrit.wikimedia.org/r/6357
[00:10:56] <gerrit-wm>	 New patchset: Ryan Lane; "The variable definitions were flipped and slightly redundant" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6358
[00:11:01] <Ryan_Lane>	 spammy spammy spam
[00:11:09] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6358
[00:11:10] <Ryan_Lane>	 I'll never be able to fucking merge all these changes into production
[00:11:25] <gerrit-wm>	 New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/6358
[00:11:28] <gerrit-wm>	 Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6358
[00:20:54] <gerrit-wm>	 New patchset: Ryan Lane; "Use the right variables and remove needless subscribes" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6359
[00:20:59] <Ryan_Lane>	 will *this* be the magical merge!?
[00:21:07] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6359
[00:21:18] <gerrit-wm>	 New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/6359
[00:21:21] <gerrit-wm>	 Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6359
[00:54:09] <gerrit-wm>	 New patchset: Ryan Lane; "Change around nova config some and fix keystone database usernames" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6363
[00:54:22] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6363
[00:54:37] <gerrit-wm>	 New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/6363
[00:54:40] <gerrit-wm>	 Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6363
[06:56:11] <petan|wk>	 Damianz hey
[06:56:23] <petan|wk>	 can you open a ticket for that
[07:01:01] <petan|wk>	 !log nagios rebooting
[07:01:02] <labs-morebots>	 Logged the message, Master
[07:06:43] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on opengrok-web i-000001e1 output: OK - load average: 0.00, 0.00, 0.00
[07:06:44] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on pediapress-ocg2 i-00000234 output: OK - load average: 0.05, 0.03, 0.05
[07:06:44] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on precise-test i-00000231 output: OK: 94% free memory
[07:07:23] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on pediapress-ocg2 i-00000234 output: USERS OK - 0 users currently logged in
[07:07:28] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on swift-be2 i-000001c8 output: OK - load average: 0.09, 0.03, 0.01
[07:07:28] <labs-nagios-wm>	 RECOVERY Total Processes is now: OK on test2 i-0000013c output: PROCS OK: 86 processes
[07:07:53] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 96% free memory
[07:07:53] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on pediapress-ocg2 i-00000234 output: DISK OK
[07:08:03] <labs-nagios-wm>	 RECOVERY Total Processes is now: OK on utils-abogott i-00000131 output: PROCS OK: 75 processes
[07:08:33] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on opengrok-web i-000001e1 output: OK: 89% free memory
[07:08:33] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on pediapress-ocg2 i-00000234 output: OK: 91% free memory
[07:08:53] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on deployment-feed i-00000118 output: DISK OK
[07:09:03] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on labs-lvs1 i-00000057 output: All packages OK
[07:09:13] <labs-nagios-wm>	 RECOVERY SSH is now: OK on opengrok-web i-000001e1 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0)
[07:09:23] <labs-nagios-wm>	 PROBLEM Total Processes is now: CRITICAL on deployment-dbdump i-000000d2 output: Connection refused by host
[07:09:28] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on bots-apache1 i-000000b0 output: OK - load average: 0.30, 0.09, 0.02
[07:09:28] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 0.18, 0.18, 0.17
[07:09:33] <labs-nagios-wm>	 RECOVERY Total Processes is now: OK on dumps-2 i-00000174 output: PROCS OK: 93 processes
[07:09:43] <labs-nagios-wm>	 RECOVERY Total Processes is now: OK on opengrok-web i-000001e1 output: PROCS OK: 85 processes
[07:09:53] <labs-nagios-wm>	 RECOVERY Total Processes is now: OK on pediapress-ocg2 i-00000234 output: PROCS OK: 78 processes
[07:10:03] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on deployment-dbdump i-000000d2 output: Connection refused by host
[07:10:23] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on opengrok-web i-000001e1 output: All packages OK
[07:10:23] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on pediapress-ocg2 i-00000234 output: All packages OK
[07:10:23] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[07:10:23] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on precise-test i-00000231 output: USERS OK - 0 users currently logged in
[07:10:33] <labs-nagios-wm>	 RECOVERY Total Processes is now: OK on swift-be2 i-000001c8 output: PROCS OK: 90 processes
[07:11:03] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on precise-test i-00000231 output: DISK OK
[07:11:03] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[07:11:13] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on utils-abogott i-00000131 output: DISK OK
[07:11:13] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on deployment-dbdump i-000000d2 output: Connection refused by host
[07:11:33] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[07:11:52] <petan|wk>	 meh
[07:11:53] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on deployment-dbdump i-000000d2 output: Connection refused by host
[07:12:43] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on deployment-dbdump i-000000d2 output: CHECK_NRPE: Socket timeout after 10 seconds.
[07:13:03] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on deployment-dbdump i-000000d2 output: Connection refused by host
[07:13:13] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on dumps-2 i-00000174 output: DPKG CRITICAL dpkg reports broken packages
[07:14:23] <labs-nagios-wm>	 RECOVERY Total Processes is now: OK on deployment-dbdump i-000000d2 output: PROCS OK: 100 processes
[07:15:03] <labs-nagios-wm>	 RECOVERY dpkg-check is now: OK on deployment-dbdump i-000000d2 output: All packages OK
[07:16:13] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on deployment-dbdump i-000000d2 output: OK - load average: 0.17, 0.51, 0.25
[07:16:53] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on deployment-dbdump i-000000d2 output: USERS OK - 0 users currently logged in
[07:17:33] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on deployment-dbdump i-000000d2 output: DISK OK
[07:18:03] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on deployment-dbdump i-000000d2 output: OK: 94% free memory
[07:36:51] <petan|wk>	 hi Ryan_Lane
[07:36:56] <petan|wk>	 what is etherpad project for?
[07:36:58] <petan|wk>	 in labs
[07:37:07] <Ryan_Lane>	 フォrえてぇrぱd
[07:37:12] <Ryan_Lane>	 err
[07:37:13] <Ryan_Lane>	 for etherpad
[07:37:46] <Ryan_Lane>	 specifically, for installing and testing etherpad, for moving changes to production
[07:40:43] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[07:41:50] <hashar>	 hello
[07:42:03] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[07:42:23] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[08:10:43] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[08:12:13] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[08:12:23] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[08:34:53] <petan|wk>	 hashar: we have a lot of troubles on deployment
[08:35:05] <petan|wk>	 there used to be a redirect from beta to deployment
[08:35:12] <petan|wk>	 it doesn't work for some reason
[08:40:43] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[08:42:13] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[08:42:23] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[08:54:25] <hashar>	 petan|wk: back, I was deploying a change
[08:54:36] <hashar>	 petan|wk: so that sounds like just one issue isn't it ? :-]
[08:57:09] <hashar>	 also http://beta.wmflabs.org/  redirects me to  http://deployment.wikimedia.beta.wmflabs.org/
[09:07:56] <petan|wk>	 that's right
[09:08:03] <petan|wk>	 http://labs.wikimedia.beta.wmflabs.org/wiki/Special:RecentChanges
[09:08:08] <petan|wk>	 this isn't
[09:08:12] <petan|wk>	 hashar: ^
[09:08:14] <petan|wk>	 fix it
[09:08:15] <petan|wk>	 :P
[09:08:18] <petan|wk>	 log it
[09:08:32] <hashar>	 ohhh
[09:10:43] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[09:12:13] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[09:12:23] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[09:27:56] <hashar>	 petan|wk: I think I will end up proposing a new name 
[09:28:05] <hashar>	 deployment-preparation is too long :-∏
[09:28:10] <hashar>	 beta not specific enough
[09:28:18] <hashar>	 I need to figure out a nice acronym :-]
[09:31:41] <mutante>	 hashar: BZ is now 4.0.6
[09:32:16] <mutante>	 hashar: nevermind, wrong nick :P. but hi :)
[09:33:01] <hashar>	 well done mutante :-]
[09:33:43] <petan|wk>	 hashar: whatever just so that it's short
[09:33:50] <petan|wk>	 like beta
[09:33:56] <petan|wk>	 and correspond to logo
[09:33:59] <petan|wk>	 so only "beta"
[09:34:04] <petan|wk>	 there is no other choice :P
[09:34:34] <hashar>	 we can change the logo
[09:37:48] <petan|wk>	 ok, let's make the logo first then I can change the dns
[09:38:12] <hashar>	 https://bugzilla.wikimedia.org/36414
[09:38:16] <hashar>	 to find a better name :°
[09:38:25] <hashar>	 if you have any idea 
[09:40:43] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[09:41:39] <mutante>	 Mid-air collision detected!
[09:41:44] <mutante>	 :)
[09:42:13] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[09:42:23] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[09:43:47] <hashar>	 petan|wk: finding a better name does not block the redirection issue
[09:43:57] <hashar>	 but I feel we need a real project name :-]]
[09:44:02] <petan|wk>	 but you flagged it as blocker
[09:44:18] <hashar>	 it is a blocker to create the git repositories in Gerrit yes
[09:44:24] <petan|wk>	 eh: Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
[09:44:25] <hashar>	 cause we will want to use the 'new' name
[09:44:30] <petan|wk>	 this is another bug
[09:44:34] <petan|wk>	 2 bugs in one
[09:44:38] <hashar>	 ohh let me fix that
[09:47:28] <hashar>	 mutante: I like staging :-]
[09:48:19] <hashar>	 I thought about "preprod" for "pre production"
[09:48:48] <mutante>	 i think it fits pretty good in http://en.wikipedia.org/wiki/Staging_%28websites%29
[09:48:57] <mutante>	 "The staging server will resemble the production environment where the clients can do the user acceptance testing"
[09:49:31] <hashar>	 that would be it yes
[09:49:44] <hashar>	 usually you do something like:  dev -> stag -> preproduction -> production
[09:49:54] <hashar>	 I guess preproduction would be test.wikipedia.org :-]
[09:50:29] <mutante>	 "get it up on the stage"
[09:53:27] <hashar>	 area52 :-]
[09:54:34] <mutante>	 hah, unstable.wm
[09:54:45] <labs-logs-bottie>	 !log deployment-prep hashar: /usr/local/apache/conf is now an independant git repository
[09:54:46] <labs-morebots>	 Logged the message, Master
[09:55:02] <hashar>	 that would be the debian way of doing things :-]
[09:55:30] <mutante>	 yea, Debian fan
[09:58:28] <mutante>	 so "sid.wm" would always be the latest 
[09:59:22] <mutante>	 but we need to replace toy story naming scheme with wm related stuff
[10:04:27] <hashar>	 lets name it Roan so :-]
[10:05:22] <mutante>	 hashar: i just imagined a link on a BZ ticket that automagically adds/integrates a voting template somewhere on meta 
[10:05:54] <mutante>	 or is BZ itself good for actually voting 
[10:06:56] <mutante>	 haha nice, i guess he will agree
[10:10:43] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[10:12:13] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[10:12:23] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[10:40:43] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[10:42:13] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[10:42:23] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[10:43:07] <hashar>	 hmm
[10:43:16] <hashar>	 somehow there is nothing set up for http://labs.wikimedia.beta.wmflabs.org/
[10:43:24] <hashar>	 so it most probably ends up in a default virtual host
[10:57:13] <hashar>	 petan|wk: do you have any script to reload the Apaches?
[11:05:00] <gerrit-wm>	 New patchset: Hashar; "classes for deployment preparation project (beta)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5790
[11:05:14] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/5790
[11:05:21] <gerrit-wm>	 New review: Hashar; "Adds `dsh` package" [operations/puppet] (test); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/5790
[11:05:53] <hashar>	 mutante: I have added you as a review for https://gerrit.wikimedia.org/r/5790
[11:06:17] <hashar>	 which is an utility class for the deployment-pre (beta) / staging / R.O.A.N project
[11:08:22] <labs-logs-bottie>	 !log deployment-prep hashar: install dsh package on deployment-dbdump
[11:08:23] <labs-morebots>	 Logged the message, Master
[11:10:43] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[11:12:13] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[11:12:23] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[11:16:16] <gerrit-wm>	 New review: Dzahn; "wait for the naming bug you opened before you add the class name now?" [operations/puppet] (test); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/5790
[11:17:08] <gerrit-wm>	 New review: Hashar; "Pending better group name : https://bugzilla.wikimedia.org/36414" [operations/puppet] (test); V: 0 C: -1;  - https://gerrit.wikimedia.org/r/5790
[11:20:42] <hashar>	 ahhhh
[11:20:54] <hashar>	 we need net groups :-D
[11:30:26] <hashar>	 broke the site :-D
[11:30:28] <hashar>	 (again)
[11:35:06] <petan|wk>	 hashar: there is no script so far because of weird sudo config
[11:35:22] <hashar>	 we need a bug report so :-)
[11:35:24] <hashar>	 well two
[11:35:28] <petan|wk>	 if sudo didn't need a password every time it would be great
[11:35:44] <petan|wk>	 but now it's a problem
[11:38:10] <petan|wk>	 hashar: so stage?
[11:38:20] <petan|wk>	 mutante: can you register dns
[11:38:32] <petan|wk>	 stage.wmflabs.org
[11:39:53] <petan|wk>	 btw hashar I deployed udp logs
[11:39:58] <petan|wk>	 it works fine
[11:40:43] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[11:41:45] <wm-bot>	 Change on 12mediawiki a page Wikimedia Labs was modified, changed by OrenBochman link https://www.mediawiki.org/w/index.php?diff=531502 edit summary: /* Proposals */ 
[11:42:13] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[11:42:15] <hashar>	 petan|wk: I have noticed that
[11:42:20] <petan|wk>	 ok
[11:42:21] <hashar>	 :-)
[11:42:23] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[11:42:59] <hashar>	 sudo: no tty present and no askpass program specified
[11:43:02] <hashar>	 ohhh
[11:43:08] <petan|wk>	 	DISK CRITICAL - free space: / 0 MB (0% inode=53%): 
[11:43:12] <petan|wk>	 meh
[11:43:24] <petan|wk>	 hashar: where
[11:43:42] <hashar>	 on bastion1 while doing :   ssh deployment-web3 sudo service apache restart
[11:45:59] <hashar>	 we will need to find out a way to remotely connect using a passwordless key
[11:46:02] <hashar>	 or something like that
[11:46:51] <labs-logs-bottie>	 !log deployment-prep petrb: purged stuff on transcoding and freed some 119468kb
[11:46:52] <labs-morebots>	 Logged the message, Master
[11:47:33] <petan|wk>	 oh yes I think we need a ticket for that
[11:49:23] <hashar>	 petan|wk:  https://bugzilla.wikimedia.org/36422  "easily reload all apaches"
[11:49:32] <petan|wk>	 I meant sudo
[11:49:42] <petan|wk>	 it's labs issue not project related
[11:50:42] <hashar>	 man sudoers ,  see 'visiblepw'
[11:50:43] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on deployment-transcoding i-00000105 output: DISK OK
[11:50:50] <hashar>	 looks like it is intended
[11:51:05] <petan|wk>	 it should work without pw
[11:51:20] <hashar>	 need to set 'visiblepw'  in /etc/sudoers so :-]
[11:51:32] <hashar>	 anyway that does not scale
[11:51:39] <hashar>	 we need ssh keys so we can load them in an agent
[11:52:40] <labs-logs-bottie>	 !log deployment-prep j: add transcoding settings to CommonSettings.php again
[11:52:41] <labs-morebots>	 Logged the message, Master
[11:52:57] <labs-logs-bottie>	 !log deployment-prep petrb: changed sudo policies on web to test if puppet override it
[11:52:58] <labs-morebots>	 Logged the message, Master
[11:53:21] <petan|wk>	 hashar: it doesn't require pw on -web now
[11:53:31] <petan|wk>	 I manualy changed policies
[11:53:37] <petan|wk>	 but I think puppet will override it
[11:54:43] <hashar>	 run puppet you will see :-]
[11:54:46] <hashar>	 sudo puppetd -tv
[11:54:49] <petan|wk>	 ok
[11:55:22] <petan|wk>	 yes
[11:55:23] <petan|wk>	 it did
[11:55:36] <petan|wk>	 -%sudo ALL=(ALL) NOPASSWD: ALL
[11:55:36] <petan|wk>	 +%sudo ALL=(ALL) ALL
[11:55:36] <petan|wk>	  #
[11:55:36] <petan|wk>	  #includedir /etc/sudoers.d
[11:55:43] <petan|wk>	 meh
[11:57:29] <petan|wk>	 hashar: we could create a local script with setuid
[11:57:46] <petan|wk>	 4755
[11:57:55] <petan|wk>	 but that may not work on gluster dunno
[11:58:11] <petan|wk>	 mutante: some idea?
[12:05:54] <j^>	 production uses $wgUseTidy = true?
[12:06:04] <petan|wk>	 mutante: yes
[12:06:07] <petan|wk>	 eh
[12:06:10] <petan|wk>	 j^: yes
[12:06:11] <petan|wk>	 mutante: nvm
[12:06:39] <j^>	 and that considers <video> and <audio> to be invalid html
[12:06:54] <j^>	 good its enabled on labs now
[12:06:58] <petan|wk>	 j^: did you verify that by changed it to false?
[12:07:00] <mutante>	 petan|wk: idea about what? should read backlog?
[12:07:05] <petan|wk>	 changing
[12:07:09] <petan|wk>	 mutante: sudo policies
[12:07:35] <mutante>	 what's the issue
[12:07:37] <petan|wk>	 mutante: [13:35:06] <petan|wk> hashar: there is no script so far because of weird sudo config
[12:07:38] <j^>	 petan|wk: yes, just checked setting it to false and the page works, setting it to true again it strips the videos
[12:07:43] <petan|wk>	 mutante: read scrollback from there
[12:07:55] <petan|wk>	 too many lines to paste
[12:08:07] <petan|wk>	 j^: ok that sounds like a bug of tidy
[12:09:19] <hashar>	 heading out to get some kind of lunch/snack
[12:09:20] <hashar>	 brb
[12:10:43] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[12:12:13] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[12:12:23] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[12:17:31] <mutante>	 petan|wk: you can use NOPASSWD in sudo policies. if there are no other ways that might be better for security
[12:21:22] <mutante>	 petan|wk: eh, i saw you tried it. but i meant via the puppet sudo stuff, because puppet writes that file..
[12:22:16] <mutante>	 i did not need NOPASSWD in labs yet though. most the time it should also be possible to change regular permissions to execute it without root
[12:33:39] <petan|wk>	 mutante:  service restart apache2
[12:33:43] <petan|wk>	 how do I do that without sudo
[12:34:06] <petan|wk>	 service apache2 restart
[12:34:07] <petan|wk>	 that one
[12:39:54] <mutante>	 ah ok, for that need to use sudo, unless apache would be on non-privileged port, unlike 80
[12:40:40] <mutante>	 then there is just NOPASSWD i guess
[12:40:43] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[12:42:02] <mutante>	 petan|wk: let puppet do the service restart?
[12:42:13] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[12:42:23] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[12:44:48] <mutante>	 like if you let somebody execute puppetd -tv already and puppet ensures services are running all the time..
[12:46:25] <petan|wk>	 mutante: ok so how would you reload all apaches
[12:46:48] <petan|wk>	 you change a config and you need to reload them, how to do that
[12:50:59] <mutante>	 petan|wk: in production like this:  http://wikitech.wikimedia.org/view/Apaches#All  we use a script "apache-graceful-all" on bastion host. the script itself uses "dsh"http://wikitech.wikimedia.org/view/Dsh
[12:51:29] <petan|wk>	 mutante: that script doesn't need to run as root?
[12:53:11] <mutante>	 petan|wk: apache-graceful-all uses apache-graceful, which uses sudo $APACHETCL graceful
[12:53:17] <mutante>	 so, yea , sudo
[12:54:22] <petan|wk>	 mutante: which doesn't require pw
[12:55:38] <mutante>	 petan|wk: correct, it uses NOPASSWD
[12:56:02] <mutante>	 for groups apache,mwdeploy and l10nupdate
[12:57:12] <petan|wk>	 ok
[13:08:14] <petan|wk>	 hashar: simple.wikipedia.deployment.wmflabs.org/wiki/Special:RecentChanges
[13:08:29] <petan|wk>	 hashar: can you fix this immediately?
[13:08:41] <petan|wk>	 it seems that all wikis are down except for main
[13:10:43] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[13:12:13] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[13:12:23] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[13:13:51] <hashar>	 petan|wk: hmm
[13:14:14] <hashar>	 was that the bug you talked about this morning?
[13:14:19] <petan|wk>	 I don't know
[13:14:20] <hashar>	 oh no that was wikiMedia
[13:14:22] <petan|wk>	 I noticed it now
[13:14:36] <petan|wk>	 maybe it was broken since last week
[13:14:59] <hashar>	 what is the real and official URL anyway?
[13:15:23] <petan|wk>	 real is simple.wikipedia.org and labs version simple.wikipedia.beta.wmflabs.org
[13:15:40] <hashar>	 ok so http://simple.wikipedia.beta.wmflabs.org/wiki/Main_Page
[13:15:42] <hashar>	 works for me
[13:15:58] <petan|wk>	 http://en.wikipedia.deployment.wmflabs.org/wiki/Special:RecentChanges
[13:16:03] <petan|wk>	 this doesn't work to me
[13:16:09] <hashar>	 so it is all about having   beta -> deployment :-]
[13:16:22] <hashar>	 probably missing a ServerAlias in the apache configuration
[13:16:26] <petan|wk>	 oh you are right
[13:16:31] <petan|wk>	 it used to redirect there
[13:16:37] <petan|wk>	 my browser has old links
[13:16:44] <petan|wk>	 in past it always redirected to beta
[13:18:02] <petan|wk>	 basically *.deployment -> *.beta
[13:28:10] <gerrit-wm>	 New patchset: Mark Bergsma; "Add precise to the default list of pbuilder distributions" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6386
[13:28:24] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6386
[13:29:09] <gerrit-wm>	 New review: Mark Bergsma; "(no comment)" [operations/puppet] (test); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/6386
[13:29:12] <gerrit-wm>	 Change merged: Mark Bergsma; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6386
[13:40:43] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[13:40:52] <petan|wk>	 mutante: can you give me access to etherpad project
[13:41:00] <petan|wk>	 or who is maintaining that
[13:42:13] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[13:42:23] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[13:49:30] <j^>	 can i get sudo on deployment-transcoding again?
[13:51:01] <petan|wk>	 sure
[13:51:17] <petan|wk>	 I am pretty sure you have it
[13:51:21] <petan|wk>	 unless you aren't J
[13:51:29] <petan|wk>	 you always had it
[13:52:27] <petan|wk>	 ok, it seems that my "fix" actually broke it
[13:52:35] <petan|wk>	 no one has sudo atm
[13:52:37] <petan|wk>	 I will fix it
[13:53:08] <petan|wk>	 we just need to wait for puppet to run
[13:53:38] <petan|wk>	 mutante: you know the default root pw / can you enforce puppet
[13:54:51] <mutante>	 !log deployment-transcoding runnng puppet
[13:54:51] <labs-morebots>	 deployment-transcoding is not a valid project.
[13:55:01] <mutante>	 petan|wk: busy, but running it ...
[13:55:21] <mutante>	 finished catalog run
[13:55:23] <petan|wk>	 thanks
[13:56:26] <petan|wk>	 j^: fixed
[13:56:47] <labs-nagios-wm>	 PROBLEM Disk Space is now: WARNING on nagios 127.0.0.1 output: DISK WARNING - free space: /home/petrb 3294 MB (19% inode=81%):
[14:00:50] <j^>	 ok, works again
[14:01:47] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on nagios 127.0.0.1 output: DISK OK
[14:02:04] <j^>	 is /mnt/data on gluster? somehow i get seek errors
[14:02:25] <j^>	 deployment-nfs-memc:/mnt/export/data on /mnt/data type nfs (rw,addr=10.4.0.58)
[14:02:34] <j^>	 always something new :)
[14:05:59] <petan|wk>	 j^: everything is on gluster, but /mnt/data should be virtual hdd which lives on gluster
[14:06:26] <petan|wk>	 in fact it's a nfs mount point
[14:06:45] <j^>	 ok, so should not have changed really, will look into it
[14:10:47] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[14:12:27] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[14:12:37] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[14:41:23] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[14:42:43] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[14:42:43] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[14:46:23] <labs-nagios-wm>	 PROBLEM HTTP is now: CRITICAL on mailman-01 i-00000235 output: CRITICAL - Socket timeout after 10 seconds
[15:01:26] <mutante>	 !log mailman - upping IP quota to 1
[15:01:27] <labs-morebots>	 Logged the message, Master
[15:02:28] <mutante>	 !log mailman - allocated public IP
[15:02:29] <labs-morebots>	 Logged the message, Master
[15:05:39] <Thehelpfulone>	 !log mailman - added apache2 class
[15:05:40] <labs-morebots>	 Logged the message, Master
[15:06:02] <Thehelpfulone>	 !log mailman - ran puppetmaster to set up apache
[15:06:02] <labs-morebots>	 Logged the message, Master
[15:06:16] <petan|wk>	 puppet master? :P
[15:06:29] <Thehelpfulone>	 !log mailman - assigned public IP to mailman-01 instance and gave DNS name mailman.wmflabs.org
[15:06:29] <petan|wk>	 master of puppets
[15:06:30] <labs-morebots>	 Logged the message, Master
[15:06:39] <Thehelpfulone>	 petan|wk: shush you :P
[15:06:53] <petan|wk>	 this bot operates even in talking channel
[15:06:55] <petan|wk>	 :D
[15:08:26] <petan|wk>	 hashar: can you somehow make the error logs be logged to some file
[15:08:34] <petan|wk>	 I meant errors
[15:08:36] <petan|wk>	 not error logs
[15:08:40] <petan|wk>	 php
[15:08:51] <petan|wk>	 since you updated php.ini it doesn't print to screen
[15:09:02] <hashar>	 did I ? :D
[15:09:10] <petan|wk>	 yes by replacing apache class in puppet
[15:09:17] <hashar>	 ohhh that side effect
[15:09:19] <petan|wk>	 hehe
[15:09:39] <hashar>	 I think I got a bug opened for that
[15:09:44] <petan|wk>	 really?
[15:09:46] <hashar>	 IIRC Apaches sent their error logs to syslog
[15:09:50] <petan|wk>	 ok
[15:09:57] <hashar>	 aka something like  syslog.pmtpa.wmnet
[15:10:02] <hashar>	 which does not exist in labs :-]
[15:10:03] <petan|wk>	 let's make syslog shared then
[15:11:03] <hashar>	 here is the change https://gerrit.wikimedia.org/r/#change,5813
[15:11:12] <hashar>	 bug https://bugzilla.wikimedia.org/36246
[15:11:23] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[15:12:09] <petan|wk>	 syslog is fine
[15:12:43] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[15:12:43] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[15:13:29] <mutante>	 Thehelpfulone: to make it work also need to add a security group that allows port 80
[15:13:39] <Thehelpfulone>	 ah ok
[15:13:52] <hashar>	 petan|wk: I have asked mark, we will see :-]
[15:14:12] <hashar>	 I think I have another bug about setting up a syslog server
[15:14:18] <labs-logs-bottie>	 !log deployment-prep petrb: making syslog on apaches be /data/project/apaches_log
[15:14:19] <labs-morebots>	 Logged the message, Master
[15:14:23] <Thehelpfulone>	 should I tick from the other ones that are "web" mutante?
[15:14:35] <Thehelpfulone>	 chose one of those or just add port 80?
[15:14:52] <Thehelpfulone>	 probably will need SSL too
[15:15:01] <mutante>	 Thehelpfulone: example for security group "web" from deployment-prep project:
[15:15:23] <mutante>	 From 80 To 80 Protocl TCP , CIDR ranges 0.0.0.0/0
[15:15:36] <mutante>	 and the same with 443 if you want ssl
[15:16:07] <mutante>	 last time i checked adding to default group of the project was easier than making a new group
[15:16:14] <mutante>	 because you can't delete the new group
[15:16:29] <petan|wk>	 mutante Thehelpfulone you know that once you create instance you can't change the groups
[15:16:38] <petan|wk>	 !security | Thehelpfulone
[15:16:38] <wm-bot>	 Thehelpfulone: https://labsconsole.wikimedia.org/wiki/Help:Security_Groups
[15:16:52] <mutante>	 petan|wk: yea, thats what i meant. but you can add rules to default group and delete rules again, right
[15:16:57] <petan|wk>	 yes
[15:17:09] <petan|wk>	 althought it may cause a lot of troubles on big projects
[15:17:18] <Thehelpfulone>	 !log mailman - added security group rules for ports 80 and 443
[15:17:18] <petan|wk>	 with like 20 instances each for something else
[15:17:18] <labs-morebots>	 Logged the message, Master
[15:17:29] <mutante>	 i think for his case it is ok if we just adds 80 and 443 to the default of this project
[15:17:32] <Thehelpfulone>	 there we go "it works"
[15:17:35] <mutante>	 nice
[15:19:37] <petan|wk>	 ok
[15:19:41] <gerrit-wm>	 New review: Mark Bergsma; "comments inline." [operations/puppet] (test); V: 0 C: -1;  - https://gerrit.wikimedia.org/r/5813
[15:21:13] <labs-nagios-wm>	 RECOVERY HTTP is now: OK on mailman-01 i-00000235 output: HTTP OK: HTTP/1.1 200 OK - 453 bytes in 0.010 second response time
[15:31:11] <mutante>	 !log mailman added puppet group mailman
[15:31:12] <labs-morebots>	 Logged the message, Master
[15:31:23] <Thehelpfulone>	 :)
[15:31:44] <mutante>	 !log mailman added class mailman::base to puppet group
[15:31:45] <labs-morebots>	 Logged the message, Master
[15:31:54] <mutante>	 Thehelpfulone: this installs just the mailman package
[15:32:03] <mutante>	 but also requires webserver::static
[15:32:07] <Thehelpfulone>	 ok
[15:32:15] <Thehelpfulone>	 so where do I find webserver::static from?
[15:34:01] <mutante>	 !log mailman added puppet group webserver and class webserver::static (lighttpd)
[15:34:02] <labs-morebots>	 Logged the message, Master
[15:34:36] <mutante>	 Thehelpfulone: this installs the lighttpd package and adds Nagios HTTP monitoring
[15:34:57] <Thehelpfulone>	 ok
[15:35:12] <Thehelpfulone>	 that's just to make sure that the webserver is up right?
[15:35:17] <mutante>	 yes
[15:35:38] <mutante>	 you can see it in webserver.pp in the repo
[15:41:23] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[15:42:43] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[15:42:43] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[15:44:45] <gerrit-wm>	 New patchset: Hashar; "rsyslog: ability to send logs to a custom server" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5813
[15:44:59] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/5813
[15:45:08] <gerrit-wm>	 New review: Hashar; "(no comment)" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5813
[15:46:09] <gerrit-wm>	 New review: Hashar; "Patchset 2 use a parameterized class which requires a syslog server as argument." [operations/puppet] (test); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/5813
[15:59:08] <gerrit-wm>	 New patchset: Hashar; "rsyslog: ability to send logs to a custom server" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5813
[15:59:18] <gerrit-wm>	 New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (test); V: -1 - https://gerrit.wikimedia.org/r/5813
[16:00:05] <gerrit-wm>	 New review: Hashar; "Patchset 3 avoids variable reassignment by setting a default AFTER the selection. Also rename the cl..." [operations/puppet] (test); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/5813
[16:02:14] <gerrit-wm>	 New review: Mark Bergsma; "This won't work, that class invocation is not valid Puppet syntax." [operations/puppet] (test); V: 0 C: -1;  - https://gerrit.wikimedia.org/r/5813
[16:03:51] <gerrit-wm>	 New patchset: Hashar; "(bug 36246) rsyslog: send logs to a custom server" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/5813
[16:04:05] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/5813
[16:04:18] <gerrit-wm>	 New review: Hashar; "And now without the $:: trick in global namespace :-]" [operations/puppet] (test); V: 0 C: 0;  - https://gerrit.wikimedia.org/r/5813
[16:11:54] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[16:12:44] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[16:12:44] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[16:41:54] <labs-nagios-wm>	 PROBLEM host: analytics is DOWN address: i-000000e2 CRITICAL - Host Unreachable (i-000000e2)
[16:42:44] <labs-nagios-wm>	 PROBLEM host: ganglia-test is DOWN address: i-00000202 CRITICAL - Host Unreachable (i-00000202)
[16:42:44] <labs-nagios-wm>	 PROBLEM host: salt is DOWN address: i-000001c1 CRITICAL - Host Unreachable (i-000001c1)
[16:51:17] <Thehelpfulone>	 ok so I put the mailman class on my mailman instance and mu.tan.te told me that from http://www.gnu.org/software/mailman/mailman-install/index.html I can go straight to step 4 checking the installation - http://www.gnu.org/software/mailman/mailman-install/node9.html - my question is, how do I figure out where mailman is actually installed? 
[17:07:14] <labs-nagios-wm>	 PROBLEM Puppet freshness is now: CRITICAL on aggregator1 i-0000010c output: Puppet has not run in last 20 hours
[17:07:14] <labs-nagios-wm>	 PROBLEM Puppet freshness is now: CRITICAL on asher1 i-0000003a output: Puppet has not run in last 20 hours
[17:07:14] <labs-nagios-wm>	 PROBLEM Puppet freshness is now: CRITICAL on backport i-0000016f output: Puppet has not run in last 20 hours
[17:07:14] <labs-nagios-wm>	 PROBLEM Puppet freshness is now: CRITICAL on bastion-restricted1 i-0000019b output: Puppet has not run in last 20 hours
[17:07:14] <labs-nagios-wm>	 PROBLEM Puppet freshness is now: CRITICAL on bastion1 i-000000ba output: Puppet has not run in last 20 hours
[17:16:31] <Thehelpfulone>	 any ideas petan|wk?
[17:17:44] <Thehelpfulone>	 I didn't actually run "make install" but there are some mailman files in /etc/mailman mm_cfg.py apache.conf 
[17:18:22] <Reedy>	 which binaryname
[17:19:18] <Damianz>	 dpkg -L mailman?
[17:19:41] <Thehelpfulone>	 Damianz: ooh that did something
[17:19:56] <Thehelpfulone>	 loaded a list of files 
[17:19:56] <Damianz>	 It might be --contents according to google, meh I'm a rhel guy
[17:20:04] <Thehelpfulone>	 is that a list of where the things are installed?
[17:20:10] <Damianz>	 AFAIK -L is the list of files the package installs
[17:20:15] <Thehelpfulone>	 not sure what that is Reedy 
[17:20:34] <Damianz>	 which returns where something is in $PATH
[17:21:18] <Thehelpfulone>	 oh lovely, 85 problems when I ran check_perms
[17:22:09] <Thehelpfulone>	 ah
[17:22:14] <Thehelpfulone>	 /var/lib/mailman/mail bad group (has: root, expected list) (fixing)
[17:22:47] <Thehelpfulone>	 how do I make that list it says Re-run as list (or root) with -f flag to fix -- but I'm running it as root and it's not fixing it
[17:22:51] <Thehelpfulone>	 is list a user?
[17:23:15] <Damianz>	 grep list /etc/passwd -q; echo $?
[17:23:49] <Thehelpfulone>	 it gives me a 0
[17:24:03] <Damianz>	 yes then
[17:25:07] <Thehelpfulone>	 so as root what's the sudo command to become a user? sudo -u list ?
[17:25:13] <Damianz>	 su user -
[17:25:55] <Thehelpfulone>	 root@mailman-01:/usr/lib/mailman# su list -
[17:25:56] <Thehelpfulone>	 su: Permission denied
[17:26:58] <Damianz>	 o.0
[17:27:17] <Damianz>	 Do we have some crazy funky pam config?
[17:27:32] <Thehelpfulone>	 I should have permission - I'm a netadmin and I'm running it as root
[17:27:54] <Thehelpfulone>	 Damianz: heh hopefully not!
[17:27:58] * Damianz  looks at Ryan_Lane and wonders how he implimented the sudo ldap policy stuff

[17:28:19] <Ryan_Lane>	 Damianz: are you subscribed to the list?
[17:28:33] <Thehelpfulone>	 Ryan_Lane: I was trying to become the user "list"
[17:28:35] <Thehelpfulone>	 [18:25:55]	<Thehelpfulone>	root@mailman-01:/usr/lib/mailman# su list -
[17:28:36] <Thehelpfulone>	 [18:25:55]	<Thehelpfulone>	su: Permission denied
[17:28:44] <Damianz>	 labs? yes but I've got like 400 unread emails in my main folder
[17:28:45] <Damianz>	 :P
[17:28:51] <Ryan_Lane>	 heh
[17:28:54] <Damianz>	 And for some reason labs isn't going to the labs folder again ....
[17:28:57] <Ryan_Lane>	 well, it mentions how I implemented it
[17:29:54] <Ryan_Lane>	 hm
[17:30:06] <Ryan_Lane>	 pam is denying it
[17:30:17] <Damianz>	 List-Id: Wikimedia Labs <labs-l.lists.wikimedia.org>
[17:30:18] <Ryan_Lane>	 that said, it doesn't actually block me from becoming a user
[17:30:18] <Damianz>	 if allof (header :contains ["List-Id"] ["\"A list for announcements and discussion related to the Wikimedia Labs project.\" <labs-l@lists.wikimedia.org>"]) {
[17:30:21] <Damianz>	 ARGH
[17:30:29] * Damianz  stabs mailman people changing things AGAIN

[17:32:39] <Thehelpfulone>	 Damianz, Ryan_Lane - full error message http://pastebin.com/cRJze2VV
[17:32:54] <Thehelpfulone>	 it tells me that I should be able to run it as root to fix it...
[17:33:13] <Ryan_Lane>	 don't run it as root
[17:33:14] <Damianz>	 Did it install with the wrong perms?
[17:33:20] <Ryan_Lane>	 fix the ownership
[17:33:28] <Ryan_Lane>	 chown list <filename>
[17:33:29] <Damianz>	 The package should set the correct perms, try reconfiging it or just manually fix it.
[17:33:44] <Thehelpfulone>	 do I run that chown list as root?
[17:33:46] <Damianz>	 Dunno why su doesn't work though, that's a seperate pam thing.
[17:33:47] <Ryan_Lane>	 yes
[17:33:56] <Ryan_Lane>	 well, su does actually work
[17:34:03] <Ryan_Lane>	 even though it gives back a permission denied
[17:34:18] <Ryan_Lane>	 list probably doesn't have a shell
[17:34:42] <Ryan_Lane>	 so: su -s /bin/bash - list
[17:34:45] <Ryan_Lane>	 that would work
[17:34:56] <Damianz>	 Good point
[17:35:02] <Ryan_Lane>	 pam access is denying it
[17:35:11] <Damianz>	 Though as it says 'run as list' one would think it hs a shell.
[17:35:20] <Ryan_Lane>	 I need to change pam to only use pam_access for ssh
[17:36:00] <Thehelpfulone>	 http://pastebin.com/s0fLu0jL
[17:36:11] <Thehelpfulone>	 looks like something is broken python wise
[17:36:30] <Ryan_Lane>	 that'll solve two things. lemme make that change
[17:36:43] <Thehelpfulone>	 ok
[17:37:00] <Damianz>	 Does chown .mailmanuser /var/lib/mailman/logs work?
[17:37:22] * Damianz  gusses mailmanuser is probably 'list'

[17:37:35] <Damianz>	 Totally never got mailman to work nicely, such a hack of an app
[17:37:35] <Thehelpfulone>	 nope it told me that the directory doesn't exist Damianz 
[17:37:48] <Damianz>	 eh
[17:38:00] <Thehelpfulone>	 No such file or directory for a few of them
[17:38:00] <Damianz>	 Wtf would path.walk return a path that doesn't exist
[17:38:05] <gerrit-wm>	 New patchset: Ryan Lane; "Verbose needs a value in essex" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6416
[17:38:16] * Damianz  votes you just mkdir -p them all and chown

[17:38:19] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6416
[17:40:36] <Thehelpfulone>	 hmm
[17:40:44] <Thehelpfulone>	 I just did that an chowned them all
[17:40:51] <Thehelpfulone>	 but it still thinks they belong to root
[17:41:14] <Damianz>	 are they?
[17:41:30] <Thehelpfulone>	 do I have to put a / because they could be folders?
[17:41:33] <gerrit-wm>	 New patchset: Ryan Lane; "Change pam configuration to only require pam_access for sshd" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6417
[17:41:45] <Damianz>	 eh?
[17:41:47] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6417
[17:41:51] <Damianz>	 they probably are folders
[17:41:55] <Thehelpfulone>	 yes
[17:42:01] <Thehelpfulone>	 according to chown --help
[17:42:10] <Thehelpfulone>	 i type chown root /u 
[17:42:22] <Thehelpfulone>	 to change the owner of /u to "root"
[17:42:29] <Ryan_Lane>	 why /u?
[17:42:30] <Damianz>	 yes
[17:42:31] <Thehelpfulone>	 I was just typing chown list logs
[17:42:40] <Damianz>	 /u  is a path in that example
[17:42:41] <gerrit-wm>	 New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/6417
[17:42:43] <Thehelpfulone>	 without a /
[17:42:43] <gerrit-wm>	 Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6417
[17:42:45] <Ryan_Lane>	 ah
[17:43:11] <Damianz>	 mkdir -p /var/lib/mailman/logs; chown list.list /var/lib/mailman/logs or .list /var/lib/mailman/logs should work
[17:43:22] <Damianz>	 .list = group only, no idea on the 'correct' ownership tbh
[17:43:42] <Damianz>	 You could do it without a / if you happen to be in some parent path
[17:43:55] <Thehelpfulone>	 yes I am in the /lib/mailman folder
[17:44:28] <Thehelpfulone>	 still not being fixed. hmm so I'm in /var/lib/mailman/
[17:44:39] <Thehelpfulone>	 ls shows the new folders that I created
[17:47:47] <Thehelpfulone>	 ok so now a ls of /lib/mailman
[17:47:52] <Thehelpfulone>	 shows some folders in dark blue
[17:47:57] <Thehelpfulone>	 and some in light blue
[17:49:15] <gerrit-wm>	 New patchset: Ryan Lane; "Use the new pam configuration, when pam is enabled" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6421
[17:49:29] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6421
[17:51:34] <gerrit-wm>	 New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/6421
[17:51:37] <gerrit-wm>	 Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6421
[17:51:45] <Ryan_Lane>	 so, that should fix the su problem
[17:52:21] <Thehelpfulone>	 still doesn't work :S
[17:52:29] <Ryan_Lane>	 puppet needs to run
[17:52:35] <Thehelpfulone>	 ah of course
[17:52:59] <Damianz>	 light and dark blue means nothing :P
[17:53:08] <Thehelpfulone>	 do you know what's causing http://pastebin.com/s0fLu0jL Damianz?
[17:53:24] <Thehelpfulone>	 well I thought there would be a differentiation between them somehow :P
[17:54:01] <Damianz>	 Wait
[17:54:09] <Damianz>	 Are you running this as lists?
[17:54:12] <Damianz>	 And the path is owned by root?
[17:54:15] <Damianz>	 That would be why
[17:54:31] <Damianz>	 chown .list /var/lib/mailman/mail /var/lib/mailman/logs
[17:54:35] <Damianz>	 as root
[17:54:39] <Damianz>	 then re-run the check as list
[17:54:43] <Thehelpfulone>	 ok
[17:56:30] <Thehelpfulone>	 same error message
[17:56:55] <Damianz>	 /var/lib/mailman/logs bad group (has: root, expected list) error?
[17:57:02] <Damianz>	 what does ls -lra  /var/lib/mailman/logs  say?
[17:57:29] <Thehelpfulone>	 sorry by same error I mean http://pastebin.com/s0fLu0jL
[17:57:34] <Thehelpfulone>	 I'm currently as list
[17:57:42] <Thehelpfulone>	 do you want me to run that ls -lra as list?
[17:57:55] <Damianz>	 Yeah
[17:57:58] <Damianz>	 doesn't matter really
[17:58:01] <Thehelpfulone>	 lrwxrwxrwx 1 root root 16 2012-05-02 15:49 /var/lib/mailman/logs -> /var/log/mailman
[18:00:01] <Damianz>	 you did chown .list /var/lib/mailman/logs as r00t, right?
[18:00:17] <Thehelpfulone>	 yes
[18:00:29] <Damianz>	 Then that line should say root list
[18:00:35] <Damianz>	 Did the chown return any output?
[18:00:38] * Thehelpfulone  tries again

[18:00:38] <Thehelpfulone>	 nope
[18:04:11] <Thehelpfulone>	 should it return output Damianz?
[18:06:41] <Damianz>	 Nope, if it doesn't return anything it most likely worked
[18:07:01] <Thehelpfulone>	 I ran it again but it didn't seem to work
[18:07:10] <Thehelpfulone>	 would it be easier if you had access to run it yourself?
[18:07:11] <Damianz>	 Might need to set it on /var/log/mailman also as a sidenote
[18:07:54] <Damianz>	 Probably easier, who knows :D
[18:08:37] <labs-home-wm>	 05/02/2012 - 18:08:37 - Creating a home directory for damian at /export/home/mailman/damian
[18:08:49] <Thehelpfulone>	 go crazy ;)
[18:09:38] <labs-home-wm>	 05/02/2012 - 18:09:38 - Updating keys for damian
[18:10:45] * Damianz  has a look

[18:10:48] <Thehelpfulone>	 thanks
[18:11:14] <Thehelpfulone>	 I'm trying to get to a stage where I have something at http://mailman.wmflabs.org - a mailman web interface
[18:11:30] <Thehelpfulone>	 ideally it would be setup without the actual mail server itself
[18:11:48] <Damianz>	 The mailman class actually configures exim properly
[18:12:21] <Thehelpfulone>	 yeah thats what I thought...
[18:13:44] <Damianz>	 Actually hmm
[18:14:49] <Damianz>	 Nope it seems it doesn't using that class, there is a seperate option in the exim class to configure it from the look of it
[18:15:00] <Thehelpfulone>	 !log mailman added Damianz to the project
[18:15:01] <labs-morebots>	 Logged the message, Master
[18:15:46] <Thehelpfulone>	 okay do I need to do anything from labswiki to fix that?
[18:18:00] <Damianz>	 If you just want mailman without a mailserver this should be right, just checking st00fs
[18:23:08] <Thehelpfulone>	 ok thanks
[18:25:54] <Damianz>	 hmm, my lack of puppet knwoelege makes this painful slightly.
[18:26:41] <Damianz>	 Ryan_Lane: If you have an 'include' at the bottom of a class which includes a class within the class is that applied if you require a class of that class?
[18:26:52] <Thehelpfulone>	 heh all I know about puppet is to puppetd -tv
[18:27:02] <Damianz>	 Ie class mailman { class base { } } and you require mailman::base but the include is in the mailman not the base class
[18:27:42] <Damianz>	 The webserver config stuff isn't being applied, which the include ... well... includes so I think you should be using mailman rather than mailman::base but me+puppet = suckageslightly
[18:28:53] <Damianz>	 I fixed your ownership anyway :P It's just not got part of the config atm
[18:29:22] <Thehelpfulone>	 heh thanks for the ownership fix - any idea what I was doing wrong? ;)
[18:30:22] <Thehelpfulone>	 ah Damianz so when this config is fixed, all of the stuff at http://www.gnu.org/software/mailman/mailman-install/node10.html will already be done?
[18:31:33] <Damianz>	 Yes, mailman::web-ui does that bit, or should anyway
[18:32:04] <Damianz>	 Though hmmm
[18:32:10] <Damianz>	 That is trying to install the ssl cert
[18:32:18] <Damianz>	 So might be failing due to that as there is no labs clause
[18:32:27] * Damianz  notes this is where puppet partly sucks

[18:37:43] <Damianz>	 !log mailman installing git-core
[18:37:43] <labs-morebots>	 Logged the message, Master
[18:37:52] <Damianz>	 That should be a class but I can't edit the puppet classes
[18:38:58] <Thehelpfulone>	 can I give you the relevant access Damianz or do you need higher access?
[18:39:15] <Damianz>	 AFAIK just the 'sysadmin' role in the project
[18:39:28] <^demon>	 Ryan_Lane: Do you think you could look at 3 puppet changes I've put in for gerrit? I added you as a reviewer.
[18:39:46] <Thehelpfulone>	 done
[18:40:13] <Damianz>	 Cool
[18:41:36] <Damianz>	 !log mailman Added generic::packages::git-core for git
[18:41:37] <labs-morebots>	 Logged the message, Master
[19:00:14] <Damianz>	 ah
[19:00:17] <Damianz>	 Yeah I was right
[19:00:29] * Damianz  waits for this to run

[19:00:33] <Thehelpfulone>	 evil puppet?
[19:01:11] <Damianz>	 The correct class is mailman not mailman::base BUT that breaks because you cannot access the wildcard cert... but I can fix that, I think
[19:04:31] <Thehelpfulone>	 ok
[19:04:36] <gerrit-wm>	 New patchset: DamianZaremba; "Adding condition for the wildcard ssl cert - this cannot be installed in labs and thus breaks the class" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6430
[19:04:42] <Damianz>	 if someone would be so kind ^
[19:04:50] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/6430
[19:07:17] <petan|wk>	 Thehelpfulone: yes
[19:07:56] <petan|wk>	 Thehelpfulone: what did you need
[19:08:06] * Damianz  points petan|wk to gerrit

[19:08:19] <Thehelpfulone>	 petan|wk: hmm can't remember one sec
[19:08:41] <Thehelpfulone>	 could you check Damianz's gerrit request abovethough now you're here?
[19:09:21] <Damianz>	 I really don't like the new diff colours :( It makes me think of google
[19:09:38] <Damianz>	 Anyway, going to grab food but if that gets merged puppet should run on this vm and mailman should start to w0rk properly
[19:10:59] <Thehelpfulone>	 Ryan_Lane: https://gerrit.wikimedia.org/r/#change,6430 can you merge that please?
[19:11:00] <petan|wk>	 Thehelpfulone: what is it
[19:11:02] <Damianz>	 Actually tbf it installed the cert, but it can't dump out a not-password-protected version
[19:11:17] <petan|wk>	 how can I review something when I don't know why is it
[19:11:19] <petan|wk>	 :D
[19:11:20] <Damianz>	 And the passwords for labs won't unlock it so it fails the install :(
[19:11:24] <Damianz>	 petan|wk: Just say 'yes' :D
[19:11:43] <Thehelpfulone>	 petan|wk: mailman on labs doesn't have secure certificates from what I'm aware
[19:11:48] <Damianz>	 petan|wk: basically only try and install *.wiki cert on production as it breaks mailman in labs]
[19:12:01] <petan|wk>	 Damianz: check_s
[19:12:01] <petan|wk>	 sl_cert!*.wikimedia.org
[19:12:07] <Damianz>	 Well it breaks puppet... mailman hasn't worked yet thereforce can't be broken
[19:12:16] <petan|wk>	 should it be directly for domain?
[19:12:35] <petan|wk>	 Thehelpfulone: mailman?
[19:12:42] <petan|wk>	 we don't have that on labs
[19:12:53] <Thehelpfulone>	 we do as of yesterday ;)
[19:13:04] <petan|wk>	 ok so what is it about then
[19:13:29] <petan|wk>	 you are trying some changes which are supposed to be pushed or just try something else
[19:13:30] <Thehelpfulone>	 what do you mean? it's going to be a copy of mailman that is on production on labs
[19:13:35] <Thehelpfulone>	 yes
[19:13:42] <Thehelpfulone>	 I'm going to be changing all the list info pages
[19:13:58] <petan|wk>	 ok, and purpose is to test stuff?
[19:14:09] <Thehelpfulone>	 to something prettier, providing a) you review that as a "yes" and b) that it doesn't break anything for future updates
[19:14:10] <Thehelpfulone>	 yes
[19:14:11] <petan|wk>	 then move it to prod?
[19:14:43] <Thehelpfulone>	 I don't think we'll be moving the whole instance, just the few .html pages that I modify
[19:14:50] <Thehelpfulone>	 but this is to test to make sure it doesn't break anything
[19:14:55] <gerrit-wm>	 New review: Petrb; "I hope he won't break it :P" [operations/puppet] (test) C: 1;  - https://gerrit.wikimedia.org/r/6430
[19:15:24] <petan|wk>	 ok
[19:15:29] <Damianz>	 I think it's in my job description to break stuff :D
[19:15:53] <Thehelpfulone>	 is there a list of people that have +2 petan|wk?
[19:15:58] <Thehelpfulone>	 and you're still opped in here :)
[19:16:03] <Damianz>	 Though tbf if I do break it there is only 1 box using the class
[19:16:12] <petan|wk>	 yay
[19:16:24] <petan|wk>	 Thehelpfulone: yes there is a list
[19:16:29] <petan|wk>	 everyone from operation
[19:16:53] <petan|wk>	 but they are lazy to review stuff
[19:17:02] <Damianz>	 Ryan_Lane: Really need per project branches! /cc gsoc person
[19:17:06] <Thehelpfulone>	 I can be very persuasive :D
[19:17:21] <petan|wk>	 I am not going to use puppet until we have per project branches heh
[19:17:34] <petan|wk>	 waiting for ops is like waiting for goddot
[19:17:38] <Thehelpfulone>	 question, do 2x +1 make a +2 ?
[19:17:50] <RoanKattouw_away>	 Nope
[19:18:07] <Thehelpfulone>	 and can the rules be relaxed on the labs project so that we can push our own tests through without waiting for ops?
[19:18:13] <Thehelpfulone>	 or is that the "per project branches"
[19:18:20] <petan|wk>	 yes
[19:18:22] <petan|wk>	 that is
[19:18:27] <Thehelpfulone>	 RoanKattouw: then can you https://gerrit.wikimedia.org/r/6430 +2 please 
[19:18:52] <RoanKattouw>	 Thehelpfulone: I don't have +2 rights on that repo
[19:19:15] <Thehelpfulone>	 ah :( who does?
[19:19:16] <petan|wk>	 Thehelpfulone: I told you that you need ops
[19:19:21] <petan|wk>	 lazy people do
[19:19:23] <petan|wk>	 :D
[19:19:24] * Damianz  thinks we should just make test a free for all like production until we actually have per project branches

[19:19:35] <Thehelpfulone>	 I thought RoanKattouw is part of ops?
[19:19:41] <Damianz>	 Nah
[19:19:45] <petan|wk>	 I think we should make production editable by all :P
[19:19:49] <petan|wk>	 like wikipeida
[19:19:55] <Damianz>	 That would be far more fun
[19:19:59] <petan|wk>	 yes
[19:20:12] <Damianz>	 There's like working for MW/MWF and being ops
[19:20:15] <Damianz>	 The later is cooler
[19:20:17] <petan|wk>	 Thehelpfulone: Roan is not lazy and that makes him noop
[19:20:31] <Thehelpfulone>	 I figured :(
[19:20:57] <Damianz>	 Tea time!
[19:21:11] <Thehelpfulone>	 enjoy!
[19:28:07] <Thehelpfulone>	 Ryan_Lane: when you come back (or maplebed) please review and +2 https://gerrit.wikimedia.org/r/6430
[19:59:18] <hashar>	 ^demon: ping pong. I am somehow willing to change the deployment preparation (beta) project name to something shorter / easy to refers to 
[19:59:45] <hashar>	 ^demon: we thought about "staging"  as a playground for mediawiki people
[20:00:14] <hashar>	 ^demon: the evil bug is https://bugzilla.wikimedia.org/36414 :-]
[20:00:46] <^demon>	 hashar: I don't really care what you call it :)
[20:00:57] <hashar>	 ahhh
[20:01:09] <hashar>	 I have thought you were Sam
[20:01:10] <hashar>	 sorry
[20:01:15] * hashar  is tired

[20:03:50] <petan|wk>	 hashar: once you find the name you like I rename it but I need someone from ops to register dns
[20:04:37] <hashar>	 petan|wk: robla replied and said "betalabs"
[20:04:48] <hashar>	 would it be fine to you?
[20:04:58] <j^>	 what about head
[20:05:30] <robla>	 "head" is ambiguous
[20:05:35] <petan|wk>	 hashar: it's ok but Ryan will complain
[20:05:44] <robla>	 what's wrong with betalabs?
[20:05:49] <petan|wk>	 he hate to call it labs
[20:05:52] <j^>	 why not just beta in that case?
[20:05:56] <robla>	 it's on labs
[20:06:00] <petan|wk>	 it's already beta
[20:06:10] <j^>	 but not just beta
[20:06:12] <petan|wk>	 anyway I like that name as well as stage
[20:06:19] <petan|wk>	 j^: right now it's just beta
[20:06:25] <petan|wk>	 beta.wmflabs.org
[20:07:23] <petan|wk>	 I am pretty fine with anything what isn't going to be renamed in future again :)
[20:07:36] <petan|wk>	 so maybe we should poke Ryan_Lane
[20:07:45] <hashar>	 maybe :-]
[20:07:48] <petan|wk>	 so he approve it :D
[20:08:00] <hashar>	 I personally don't care of the name as long as we find something easy to refers to
[20:08:44] <petan|wk>	 short too
[20:08:53] <petan|wk>	 I like short urls :P heh
[20:09:12] <gerrit-wm>	 New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/6430
[20:09:15] <gerrit-wm>	 Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6430
[20:10:26] <petan|wk>	 yay
[20:10:45] <Damianz>	 Thanks Ryan.
[20:10:49] <petan|wk>	 Thehelpfulone: done :P
[20:11:19] <hashar>	 Ryan_Lane: wanna bike shed with us about the name of beta labs / deployement preps project ?
[20:11:28] <Thehelpfulone>	 :D
[20:11:46] <Thehelpfulone>	 Damianz: what needs to be done to make it show up on http://mailman.wmflabs.org/?
[20:11:54] <Damianz>	 2min, just eating
[20:12:05] <Ryan_Lane>	 yw
[20:12:58] <Thehelpfulone>	 ah someone is already running puppetd -tv
[20:13:56] <Damianz>	 err: /Stage[main]/Mailman::Web-ui/File[/etc/lighttpd/htdigest]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///private/lighttpd/htdigest at /etc/puppet/manifests/mail.pp:310 ah ffs - I assume that file is the production admin users?
[20:14:24] <Damianz>	 oh private lists hmmm
[20:14:38] <Damianz>	 Ryan_Lane: Does labs have a private folder atm?
[20:15:44] <Damianz>	 Hmm ignroing it for labs doesn't make sense for replicating production, creating it on the puppet server doesn't make sense for making it accessible to labs.
[20:16:09] <Damianz>	 Widly known logins is probably a bad idea....
[20:24:02] <Thehelpfulone>	 hmm
[20:24:16] <Thehelpfulone>	 my error is 
[20:24:17] <Thehelpfulone>	 err: /Stage[main]/Mailman::Web-ui/File[/etc/lighttpd/htdigest]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///private/lighttpd/htdigest at /etc/puppet/manifests/mail.pp:310
[20:24:24] <Thehelpfulone>	 oh yeah it's the same one
[20:24:28] <Thehelpfulone>	 I thought it was different
[20:24:56] <Thehelpfulone>	 do you need a password to get the other lists then Damianz?
[20:26:49] <Damianz>	 ?
[20:27:09] <Damianz>	 That file holds usernames/passwords for private lists, hence it's in the private production repo.
[20:28:04] <Damianz>	 I could change the manifest so it's only applied on production but then we're breaking away from production and introducing manual work.
[20:28:38] <Damianz>	 I still have no idea how we're going to manage passwords on production as lots of thigns (including bots) need them but they are semi-secret and making everything manual for config changes breaks the puppet way of things.
[20:28:48] <Damianz>	 Hence me asking Ryan_Lane his thoughts, but he seems busy pole dancing atm
[20:28:50] <Ryan_Lane>	 Damianz: private?
[20:29:06] <Damianz>	 ../private/ yes
[20:29:26] <Ryan_Lane>	 I don't think I understand what you mean
[20:29:43] <Damianz>	 ok
[20:29:57] <Damianz>	 In mailman for example there is "puppet:///private/lighttpd/htdigest", which holds the private lists info
[20:30:06] <Ryan_Lane>	 right
[20:30:10] <Damianz>	 For anything in the production private repo (including that) how are we manageing that on labs?
[20:30:22] <Ryan_Lane>	 private/labs repo
[20:30:22] <Damianz>	 Totally can't type :(
[20:30:40] <Ryan_Lane>	 I'm wondering if this is currently working for people other than ops right now, though
[20:31:01] <Damianz>	 I can't see private in any form
[20:31:03] <Ryan_Lane>	 I don't think it is
[20:31:29] <Thehelpfulone>	 Damianz: the usernames/passwords for public lists - are they also not private?
[20:31:54] <Damianz>	 Thehelpfulone: I don't understand what you're asking
[20:32:06] <Thehelpfulone>	 [21:27:10]	<Damianz>	That file holds usernames/passwords for private lists, hence it's in the private production repo.
[20:32:18] <Thehelpfulone>	 what about public lists - aren't they also in the private production repo?
[20:32:24] <Damianz>	 It says 'private list archives'
[20:32:39] <Ryan_Lane>	 labs has its own private repo
[20:33:13] <Damianz>	 Ryan_Lane: How is access to that going to be managed generally? As we don't really want random troll going trashing databases/logging in as bots etc? Or just though signup process, sudo policies and hope?
[20:33:14] <Thehelpfulone>	 9_9
[20:33:18] <Thehelpfulone>	 O_O *
[20:33:28] <Damianz>	 Ryan_Lane: How is access to that going to be managed generally? As we don't really want random troll going trashing databases/logging in as bots etc? Or just though signup process, sudo policies and hope?
[20:33:32] <Ryan_Lane>	 so
[20:33:35] <Ryan_Lane>	 I just modified it
[20:33:44] <Ryan_Lane>	 it was open to everyone before, when testlabs was the default project
[20:33:48] <Ryan_Lane>	 now its bastion
[20:34:58] <Ryan_Lane>	 so, if you're in the project-bastion group, you can pull from and push to labs/private project
[20:35:08] <hashar>	 Ryan_Lane: can you get a look at the name for the beta / deployment preparation project?    https://bugzilla.wikimedia.org/36414 
[20:35:31] <hashar>	 Ryan_Lane: need a simple / easy to mention name,  rob says  "betalabs"
[20:35:37] <hashar>	 peter and I don't care
[20:36:02] <hashar>	 as long as we have something shorter than deployment-preparation and better than just "beta" :-]
[20:36:15] <Damianz>	 Ryan_Lane: Is that a ldap group? (as I dont see it in gerrit)
[20:36:43] <Ryan_Lane>	 Damianz: oh. lemme make the group public
[20:37:27] <Ryan_Lane>	 it's viewable now
[20:37:32] <Ryan_Lane>	 but, yes, it's an ldap group
[20:37:45] <Ryan_Lane>	 and if you are in the bastion project, you are in the project-bastion group
[20:37:57] <Ryan_Lane>	 labsconsole has instructions on checking out the private repo
[20:38:05] <Damianz>	 kk
[20:38:14] <Ryan_Lane>	 it may be missing the .gitreview stuff
[20:38:21] <Damianz>	 I can't see it in the web interface but I might need to logout and back in.
[20:38:47] <gerrit-wm>	 New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/6416
[20:38:54] <gerrit-wm>	 Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/6416
[20:39:37] <Damianz>	 fatal: '/labs/private.git': not a Gerrit project
[20:39:47] <Ryan_Lane>	 hm
[20:40:00] <Ryan_Lane>	 wtf
[20:40:03] <Ryan_Lane>	 the permissions didn't save
[20:40:39] <Ryan_Lane>	 internal server error....
[20:40:41] <Ryan_Lane>	 wtf
[20:41:58] <Ryan_Lane>	 Damianz: log out of gerrit web interface, then back in
[20:41:59] <Ryan_Lane>	 then tey
[20:42:02] <Ryan_Lane>	 *try
[20:42:40] <Damianz>	 Ok, I can see the group... not the project or clone the repo.
[20:45:06] <Thehelpfulone>	 what's the group called?
[20:45:10] <Ryan_Lane>	 it doesn't see me in the group...
[20:45:22] <Thehelpfulone>	 project-bastion found it
[20:45:30] <Ryan_Lane>	 I wonder if searches aren't implemented in a way that works for this.
[20:45:37] <Damianz>	 According to this there is no members lol
[20:45:51] <Ryan_Lane>	 it doesn't list members for ldap groups
[20:45:57] <Damianz>	 Ahh true
[20:48:13] <Ryan_Lane>	 but, it should find me in the group
[20:48:26] <Ryan_Lane>	 I wonder how it searches groups
[20:48:44] <Ryan_Lane>	 by groupOfNames
[20:48:52] <Ryan_Lane>	 member=dn
[20:49:23] <Damianz>	 I've not decide the most optimum way of ldap groups as everything seems to have its prefered way
[20:50:15] <Ryan_Lane>	 damn. that search doesn't work for virtual groups, it seems
[20:54:56] <Damianz>	 That sounds annoying.
[20:55:09] * Damianz  waits for someone to figure something out so he can create one file and magic can happen

[20:57:11] <Thehelpfulone>	 one file left for the magic to happen Damianz? damn.
[20:57:37] <Damianz>	 Allways
[20:58:07] <Thehelpfulone>	 so what's causing the block exactly?
[20:58:33] <Damianz>	 Fails copying the htdigest file so the lighty config fails
[20:58:42] <Damianz>	 Mailman is running but the webserver isn't yet.
[20:59:06] <Thehelpfulone>	 and to get the htdigest file you need access to the private repo?
[21:01:18] <Damianz>	 Yes.
[21:01:50] <Thehelpfulone>	 and how do you get access to that - is Ryan_Lane trying to give you access?
[21:02:53] <Damianz>	 Dunno, doing some work for work for a few min. Doubt he'll 'fix' it anytime very soon
[21:03:32] <Thehelpfulone>	 sure
[21:03:37] * Thehelpfulone  awaits Ryan_Lane's response

[21:03:47] <^demon>	 Ryan_Lane: I'm done, everything looks fine. Could you go and review+push 6439 for me?
[21:04:06] * Damianz  sits on paravoid and whispers labs related things in his ear

[21:06:57] <paravoid>	 ?
[21:07:16] <Damianz>	 hi
[21:07:35] <paravoid>	 hi, what's up?
[21:07:43] <Ryan_Lane>	 !gerrit 6439
[21:07:44] <wm-bot>	 https://gerrit.wikimedia.org/
[21:07:51] <Ryan_Lane>	 !change 6439
[21:07:51] <wm-bot>	 https://gerrit.wikimedia.org/r/6439
[21:08:02] <Ryan_Lane>	 (I am indeed this lazy)
[21:09:10] <Ryan_Lane>	 ^demon: merged
[21:09:14] <Ryan_Lane>	 wait
[21:09:19] <Ryan_Lane>	 is this a deployment window?
[21:09:31] <^demon>	 Blahh, again?
[21:09:34] <Ryan_Lane>	 ah
[21:09:34] <Ryan_Lane>	 nope
[21:09:35] <^demon>	 Did I seriously do that again?
[21:09:39] <^demon>	 :)
[21:09:39] <Ryan_Lane>	 it was 11-1
[21:10:40] <Damianz>	 Maybe he's on summertime.
[21:11:33] <^demon>	 Ryan_Lane: I'm crossing my fingers this will fix our utf8 issue once and for all. And gerrit's error log will become useful again :p
[21:11:41] <Ryan_Lane>	 heh
[21:12:16] <^demon>	 That error log gets spammed every time "Szymon ?wierkosz" != "Szymon Świerkosz"
[21:12:23] <Ryan_Lane>	 yeah
[21:12:37] <Ryan_Lane>	 so, utf8 usernames are unsupported in nova right now too
[21:12:44] <Ryan_Lane>	 I think andrewbogott fixed this for essex?
[21:12:49] <Ryan_Lane>	 or is it going to be folsom?
[21:12:57] <andrewbogott>	 Should be fixed in essex.
[21:13:02] <Ryan_Lane>	 \o/
[21:13:05] <andrewbogott>	 I fixed /some/ of the cases; hopefully all of them.
[21:13:57] <^demon>	 At least all the tables are now InnoDB so we can be less afraid of our data just deciding to go away.
[21:14:28] <Damianz>	 I totally havn't got essex working on 12.04 yet either :( Apparently keystone is rather more complex to configure than I imagined it to be.
[21:15:19] <Ryan_Lane>	 I'm at that stage right now too
[21:16:07] <Damianz>	 The officially ubuntu docs is VERY much tailored to jira or w/e their custom config management thing I hate is.
[21:18:07] <paravoid>	 what does jira have to do with config management?
[21:21:30] <Damianz>	 Because it's called Juju? And jira reminds me of ugly software.
[21:35:32] <Thehelpfulone>	 Damianz: tracking the issue, https://bugzilla.wikimedia.org/show_bug.cgi?id=36459 - please do chip in anything more you can add
[21:38:23] <Damianz>	 It's a known issue so I don't hugely see the point but w/e
[21:51:59] <Thehelpfulone>	 Damianz: can you temporarily make it so that we don't need access to the private repo then when it's fixed put it back into the process?
[21:52:10] <Damianz>	 not really
[21:53:15] <Damianz>	 I could annoy Ryan_Lane or one of the ops into making a blank file there but editing it when you progressd a little would be rather annoying and going outside our review process
[21:53:24] <Thehelpfulone>	 I think Ryan_Lane suggested earlier that we can make it not required in labs in the public repo - how does that work?
[21:54:27] <Damianz>	 I could submit a patch the same way as I did for the cert but that's also not a long term solution
[21:54:40] <Damianz>	 And it would need an op to merge it eitherway
[21:55:57] <Thehelpfulone>	 we have an op ( Ryan_Lane ) but as he's mentioned that he's unsure how long this could take to fix - a temporary fix would be appreciated :)
[21:57:15] <Damianz>	 He did?
[21:57:24] * Damianz  swears talking to Thehelpfulone the he is missing half the convo

[21:58:09] <Thehelpfulone>	 oh sorry, that was in a PM poke - not sure if he did in this channel
[21:58:22] <Thehelpfulone>	 heh, I take up that much of your time? :P
[21:59:36] <Ryan_Lane>	 ...
[21:59:51] <Ryan_Lane>	 can I ignore this for a while and go back to fixing things? :(
[22:00:24] <Damianz>	 Ryan_Lane: Yeah, just login to Thehelpfulone and sleep 108000 :D
[22:00:46] <Thehelpfulone>	 nooo :O
[22:01:09] * Thehelpfulone  sets up a sudo group policy to restrict access

[22:01:27] <Thehelpfulone>	 btw Ryan_Lane, the sudo group policy doesn't let netadmins access the instances by default - is that intentional?
[22:01:38] <Ryan_Lane>	 eh?
[22:01:46] <Ryan_Lane>	 I don't understand what you mean
[22:02:02] <Ryan_Lane>	 the default sudo policy lets every project member have full root
[22:02:07] <Ryan_Lane>	 on every instance
[22:02:22] <Ryan_Lane>	 the sudo stuff has absolutely nothing to do with the sysadmin or netadmin roles
[22:02:32] <Ryan_Lane>	 except that sysadmins can modify it
[22:02:47] <Thehelpfulone>	 I tried to access mailman-01 but it didn't let me
[22:02:48] <Thehelpfulone>	 I had to create a sudo group policy to allow me to access it
[22:02:48] <Thehelpfulone>	 tried to become sudo on mailman-01*
[22:02:48] <Thehelpfulone>	 bah what am I saying
[22:02:59] <Thehelpfulone>	 I could access the instance fine, I couldn't sudo to it
[22:04:36] <Thehelpfulone>	 Ryan_Lane: look at the logs for mailman-01 for failed sudo access
[22:04:41] <labs-home-wm>	 05/02/2012 - 22:04:40 - Creating a home directory for laner at /export/home/mailman/laner
[22:05:05] <Ryan_Lane>	 it was missing a default sudo poiicy?
[22:05:09] <Thehelpfulone>	 at 16:40 UTC
[22:05:12] <Thehelpfulone>	 yeah I think so
[22:05:16] <Ryan_Lane>	 it didn't have one named "default"?
[22:05:18] <Ryan_Lane>	 that's a bug.
[22:05:22] <Thehelpfulone>	 nope I had to make one
[22:05:41] <labs-home-wm>	 05/02/2012 - 22:05:41 - Updating keys for laner
[22:06:05] <Ryan_Lane>	 I just deleted yours and added the default
[22:06:33] <Thehelpfulone>	 ok
[22:44:23] <Sven_Manguard>	 Hey, where are the servers located?
[22:45:04] <Damianz>	 In my bathroom.
[22:46:09] <^demon|away>	 Sven_Manguard: Florida, Virginia, Netherlands.
[22:46:48] <Sven_Manguard>	 sry, the ones for Labs in specifc demon
[22:47:54] <^demon|away>	 Not sure. I want to say Virginia, but please don't quote me anywhere.
[22:48:37] <Damianz>	 I think the answer is Both florida and Virginia but only one site actually has 'produciton' labs stuff in and the other is for the next bit
[22:48:38] <Ryan_Lane>	 florida
[22:48:49] <Ryan_Lane>	 we haven't brought up the zone in virginia yet
[23:00:23] <Sven_Manguard>	 who maintains them? WMF staff?
[23:00:29] <Ryan_Lane>	 yes
[23:00:42] <Ryan_Lane>	 myself, paravoid, andrewbogott, and sarasmollett
[23:02:30] <Sven_Manguard>	 okay, thanks
[23:02:56] <Sven_Manguard>	 So if I have flagged bots, can they run off of labs?
[23:03:11] <Ryan_Lane>	 yes
[23:03:18] <Ryan_Lane>	 you just need access to the bots project
[23:04:50] * Damianz  prefers unflagged bots

[23:07:35] <Sven_Manguard>	 Ryan_Lane: Explain that please?
[23:07:51] <Sven_Manguard>	 I'm a Wikipedia editor with bot programs that run on Wikipedia
[23:08:37] <Sven_Manguard>	 Also if any of you are in the WMF offices, could you ask Jalexander to come online please?
[23:10:10] <Ryan_Lane>	 he isn't on our floor
[23:10:23] <Ryan_Lane>	 Sven_Manguard: so, if you run a bot, you can move the bot to labs
[23:10:27] <Ryan_Lane>	 labs is a set of projects
[23:10:34] <Ryan_Lane>	 those projects are security separations, for the most part
[23:10:40] <Ryan_Lane>	 bots is one of those projects
[23:10:46] <Sven_Manguard>	 Ah, I see.
[23:10:53] <Ryan_Lane>	 you need a bots project admin to add you as a member
[23:11:02] <Ryan_Lane>	 petan_, and Damianz are admins, I think
[23:11:14] <Sven_Manguard>	 So how would I go about making that happen, and how would I go about getting the bots set up to run daily?
[23:11:19] <Sven_Manguard>	 without my poking
[23:11:22] <Damianz>	 You have floors? I allways imagined you guys all sat around random cheap desks on cardboard boxes and be all like heairy and unix like :D
[23:11:27] <Ryan_Lane>	 heh
[23:11:33] <Ryan_Lane>	 we have two floors of a building
[23:11:56] <tjf>	 it's in san fransisco, right?
[23:11:56] <Damianz>	 Sven_Manguard: Do you have a labs account?
[23:12:03] <Ryan_Lane>	 I have a beard, but it isn't super unix-y
[23:12:10] <Sven_Manguard>	 Damianz: no, I'm flying by the seat of my pants here
[23:12:16] <Damianz>	 Ok
[23:12:28] <Sven_Manguard>	 I'm not even the bot programmer, I'm the operator, the programmer retired :(
[23:12:30] <Ryan_Lane>	 !accountreq
[23:12:30] <wm-bot>	 case you want to have an account on labs please read here: https://labsconsole.wikimedia.org/wiki/Help:Access#Access_FAQ
[23:12:32] <Damianz>	 So first you need a general labs account, that lets you login, gives you access to git etc.
[23:12:34] <Ryan_Lane>	 hm
[23:12:42] <Sven_Manguard>	 hm?
[23:12:44] <Damianz>	 Then after that you can get access to projects, such as bots to run things off.
[23:13:11] <Ryan_Lane>	 Sven_Manguard: labs is a shared cloud environment, like ec2 or rackspace cloud
[23:13:37] <Damianz>	 I wish I had a beard... currently at the scratchy horrible stage after being stupid and shaving for once.... meh saves loosing lunch and wondering around looking like you need a shower
[23:13:39] <Ryan_Lane>	 you get access to "instances", which are virtual machines
[23:13:41] <Ryan_Lane>	 !terms
[23:13:42] <wm-bot>	 https://labsconsole.wikimedia.org/wiki/Terminology
[23:14:04] <Sven_Manguard>	 I sorta understand that, but not entirely. I'm not a technical person, I wound up with the bots because I'm one of the few people familiar enough with the File namespace to know what the bots should be doing
[23:14:05] <Ryan_Lane>	 in some projects you may have full root access, in others you may just have shell access.
[23:15:11] <Ryan_Lane>	 you'll probably need to work with the project admins to help you out with your bots,then
[23:15:53] <Damianz>	 "project admins" should have a description like "we fix random shit"
[23:16:24] <Ryan_Lane>	 heh
[23:16:27] <Sven_Manguard>	 Ryan_Lane: what's "svn"
[23:16:49] <Ryan_Lane>	 svn is subversion, a source control system
[23:17:28] <Damianz>	 !log mailman remove mailman::base and added mailman to the puppet classes
[23:17:29] <labs-morebots>	 Logged the message, Master
[23:17:32] <Damianz>	 !log mailman Puppet run is blocked by private:///lighttpd/htdigest. Can't create in labs/private due to gerrit issue.
[23:17:33] <labs-morebots>	 Logged the message, Master
[23:17:35] <Damianz>	 !log mailman Once private:///lighttpd/htdigest exists run puppetd -tv and mailman should work. Mailman is currently running but lighttpd isn't configured.
[23:17:36] <labs-morebots>	 Logged the message, Master
[23:17:42] <Damianz>	 Morebot is lagged somewhat o.0
[23:17:49] <Sven_Manguard>	 Since I don't know what it is, I'm guessing that I don't have it?
[23:18:49] <tjf>	 Sven_Manguard: Basically, it lets you "check-in" a file to a repository of files, and shows the differences between the versions
[23:19:33] <tjf>	 Almost like FTP and Wikis mixed, except not everybody can "edit"
[23:19:42] <Sven_Manguard>	 FTP?
[23:19:50] <Ryan_Lane>	 Sven_Manguard: don't worry much about which things you need access to
[23:19:55] <Ryan_Lane>	 for running bots you need access to labs
[23:20:09] <Damianz>	 Wish I had access to svn but screw doing the retarded application process as we're moving to git
[23:20:13] <Ryan_Lane>	 you'll also happen to get gerrit/git access at the same time, but you can mostly ignore it
[23:20:15] <Sven_Manguard>	 Okay, so all of the instructions are too technical for me. In a perfect world, I wouldn't be doing any of this at all, but Fastily got chased off of Wikipedia, and these bots are kinda important.
[23:20:44] <Ryan_Lane>	 work with the bots people to get your bots running. they can help you
[23:20:51] <Ryan_Lane>	 it's a community after all :)
[23:21:03] <Sven_Manguard>	 That's Damianz or is that the BRFA people?
[23:21:19] <Ryan_Lane>	 !project bots
[23:21:20] <wm-bot>	 https://labsconsole.wikimedia.org/wiki/Nova_Resource:bots
[23:21:23] <Ryan_Lane>	 the people mentioned as members
[23:21:26] <Sven_Manguard>	 Community? Bullshit. If Wikipedia had a functional, healthy community, I wouldn't be here because Fastily woudln't have quit.
[23:21:34] <Ryan_Lane>	 Sven_Manguard: I mean labs
[23:21:52] <Damianz>	 The BRFA and general communitry around the BFRA is horrid :P
[23:22:08] <Ryan_Lane>	 I really need to make that project member list include sysadmin and netadmin members
[23:23:04] <Damianz>	 I really need to work on cb stuff but I really want oauth stuff so I don't make the new bits twice :(
[23:23:16] <Sven_Manguard>	 Well I know Peterb, DQ, and THO, so yay!
[23:23:18] <Ryan_Lane>	 oauth is months away at best
[23:23:49] <Thehelpfulone>	 what did I do? :O
[23:24:08] <Sven_Manguard>	 Thehelpfulone: congrats, you are now going to help me save myself from certain insanity
[23:24:16] <Ryan_Lane>	 :D
[23:24:16] <Thehelpfulone>	 I am? :P
[23:24:35] <Damianz>	 Yeah, packaging is blocked on labs status, new portal is blocked on oauth, docs is blocked on me being an un-friendly ass etc :D
[23:24:41] <Sven_Manguard>	 Yes. Do you know Java BTW? Fastily's bots are in Java and he has a nasty habit of "forgetting" to use throttles
[23:24:46] <Ryan_Lane>	 Damianz: packaging?
[23:25:21] <Damianz>	 Making the bot parts into debs and puppetizing
[23:25:29] <Ryan_Lane>	 well, you can package
[23:25:35] <Ryan_Lane>	 puppetization is hard, for sure
[23:26:01] <Thehelpfulone>	 unfortunately I don't know Java nope
[23:26:03] <Damianz>	 Well we don't reallt have a community apt server... though jenkins built debs pushed to an apt mirror would be nice *cough*
[23:26:16] <Ryan_Lane>	 yeah, it would
[23:26:18] <Damianz>	 I could use landscape and the ppt stuff if I could figure out landscape.
[23:26:33] <Ryan_Lane>	 you mean launchpad?
[23:26:39] <Damianz>	 errr yes
[23:26:45] <Ryan_Lane>	 true, you could use launchpad and ppas
[23:26:46] <Damianz>	 landscape is the paid stats thing isn't it
[23:26:49] <Ryan_Lane>	 yeah
[23:26:58] <Ryan_Lane>	 stats and patch management
[23:27:09] <Damianz>	 Spacewalk for debian
[23:27:24] <Damianz>	 Well ubuntu.. same thing.
[23:27:35] <Ryan_Lane>	 I wish spacewalk worked with ubuntu properly
[23:27:59] <Sven_Manguard>	 Okay Damianz http://www.mediawiki.org/wiki/Developer_access#User:Sven_Manguard is up. What next?
[23:27:59] <Damianz>	 I wish spacewalk didn't require java and oracale and GIGS OF RAM AND DISK but it's getting there
[23:28:25] <Sven_Manguard>	 Damianz: Gigs of disc is cheap. Ram not so much but...
[23:28:49] <Damianz>	 Annoy Ryan_Lane or the other 3 people who can do it
[23:28:58] <Sven_Manguard>	 Oh. 
[23:29:01] <Ryan_Lane>	 already on top of it ;)
[23:29:02] * Damianz  can't add accounts cause he's not 1337 enough

[23:29:28] <Sven_Manguard>	 Damianz: leetspeak is a sign of weakness. Speak in binary if you want to impress me.
[23:29:35] <Ryan_Lane>	 Sven_Manguard: sent a password via email
[23:29:50] <Damianz>	 It would be nice if spacewalk+cobbler+puppet worked together by default... it's a bit of a clunge to get smoothly atm and my cobbler server is actually a custom mirror server with cobbler just for managing pxe stuff.
[23:29:57] <Damianz>	 Sven_Manguard: Length limited in irc for binary.
[23:30:36] <Sven_Manguard>	 Is labs unified login with Wikipedia?
[23:30:40] <Thehelpfulone>	 nope
[23:30:49] <Damianz>	 Sumanah has a rather obnoxus signature
[23:31:13] <Sven_Manguard>	 Okay, I now have an account. Now what?
[23:31:14] * Thehelpfulone  wonders who can create private:///lighttpd/htdigest

[23:31:32] <Thehelpfulone>	 what OS are you on Sven_Manguard?
[23:31:40] <Sven_Manguard>	 Windows Vista
[23:32:06] <Sven_Manguard>	 Because you can't game on anything else, or you couldn't when I was still young enough do develop brand loyalty
[23:32:27] <Damianz>	 You can't run Visa on anything under 8G of ram either
[23:32:33] <Sven_Manguard>	 Well I have one Vista PC and one Windows 7 PC
[23:33:47] <Sven_Manguard>	 Okay, so now what?
[23:34:06] <Sven_Manguard>	 Now I go get dinner? Now I invade Canada and kill Terrance and Phillip?
[23:34:37] <Damianz>	 Find my drugs so I don't have to walk around the freezing room?
[23:34:53] <Sven_Manguard>	 Damianz: your drugs are in your mind, man
[23:36:53] <Damianz>	 Nah that's just my really ouchy jaw, ankle, knee, lower back etc
[23:37:06] * Damianz  pokes puppet with something pointy

[23:37:56] <Damianz>	 You're not part of the bastion project atm anyway so even if puppet does wake up it won't make a difference
[23:57:03] <Thehelpfulone>	 Damianz: would simply creating the htdigest file fix the problem? don't we need the access to it to be fixed too?
[23:58:41] <Thehelpfulone>	 Sven_Manguard: you need to download and install putty
[23:58:59] <Sven_Manguard>	 Eh? What's putty?
[23:59:20] <Thehelpfulone>	 it's some software that is going to allow you to connect to the WMF labs server via "SSH" an encryption protocol
[23:59:26] <Thehelpfulone>	 all the stuff you will do will be through command line
[23:59:31] <Thehelpfulone>	 no nice GUI that we get on windows ;)
[23:59:34] <Thehelpfulone>	 http://the.earth.li/~sgtatham/putty/latest/x86/putty-0.62-installer.exe