[00:01:14] <petan>	 !log bots installed all standart libs on bots-4 + patched kernel
[00:01:15] <labs-morebots>	 Logged the message, Master
[00:01:23] <petan>	 !log bots rebooting bots-4
[00:01:24] <labs-morebots>	 Logged the message, Master
[00:02:19] <petan>	 Krinkle: feel free to install anything on bots-4 just log it :)
[00:02:47] <Krinkle>	 petan: yep, later. need to do mw/wm work now. cvn has lower priority.
[00:02:53] <petan>	 btw Krinkle there is a /mnt/public_html you can insert stuff there so that it's visible on bots.wmflabs.org
[00:02:55] <Krinkle>	 Thanks for setting it up, now I can do it whenever I want
[00:03:11] <petan>	 np
[00:03:31] <petan>	 cluebot is running there too
[00:03:53] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on bots-4 bots-4 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[00:04:18] <petan>	 btw Ryan_Lane I want to set up a shared fs later and put all sw used by application servers there, so that we don't need to eat so much space installing same sw on dozen of application servers
[00:04:38] <petan>	 like shared bin on gluster etc
[00:04:39] <Ryan_Lane>	 that's fine. we can do that when we get gluster
[00:04:41] <petan>	 sure
[00:04:43] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on bots-4 bots-4 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[00:05:43] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on bots-4 bots-4 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[00:06:03] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on bots-4 bots-4 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[00:08:53] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on bots-4 bots-4 output: OK - load average: 0.08, 0.05, 0.02
[00:09:43] <labs-nagios-wm>	 RECOVERY Current Users is now: OK on bots-4 bots-4 output: USERS OK - 0 users currently logged in
[00:10:43] <labs-nagios-wm>	 RECOVERY Disk Space is now: OK on bots-4 bots-4 output: DISK OK
[00:11:03] <labs-nagios-wm>	 RECOVERY Free ram is now: OK on bots-4 bots-4 output: OK: 94% free memory
[01:32:18] <diederik>	 hi ryan
[01:32:38] <Ryan_Lane>	 howdy
[01:33:04] <diederik>	 what would it take to be able to access locke/emery from within the labs env?
[01:50:48] <diederik>	 ryan: so is there a way to access data on emery /locke from within the labs env?
[01:51:00] <Ryan_Lane>	 not right now, no
[01:51:07] <Ryan_Lane>	 I can't talk right now, I'm fixing the blog
[01:52:04] <diederik>	 ok, thx
[07:03:54] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on nova-dev3 nova-dev3 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[07:04:54] <labs-nagios-wm>	 PROBLEM Current Users is now: CRITICAL on nova-dev3 nova-dev3 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[07:05:14] <labs-nagios-wm>	 PROBLEM Disk Space is now: CRITICAL on nova-dev3 nova-dev3 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[07:06:04] <labs-nagios-wm>	 PROBLEM Free ram is now: CRITICAL on nova-dev3 nova-dev3 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[07:07:34] <labs-nagios-wm>	 PROBLEM Total Processes is now: CRITICAL on nova-dev3 nova-dev3 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[07:08:14] <labs-nagios-wm>	 PROBLEM dpkg-check is now: CRITICAL on nova-dev3 nova-dev3 output: CHECK_NRPE: Error - Could not complete SSL handshake.
[12:11:27] <gerrit-wm>	 New patchset: Dzahn; "try out planet-venus package in labs" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1929
[12:11:42] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/1929
[12:12:58] <gerrit-wm>	 New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1929
[13:09:34] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on deployment-web deployment-web output: WARNING - load average: 17.35, 18.34, 10.61
[13:14:34] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on deployment-web deployment-web output: CRITICAL - load average: 35.15, 23.68, 14.43
[13:39:49] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on deployment-web deployment-web output: WARNING - load average: 6.08, 16.00, 19.96
[16:31:49] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on bots-cb bots-cb output: WARNING - load average: 5.60, 13.82, 7.32
[16:41:49] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on bots-cb bots-cb output: OK - load average: 0.54, 2.34, 4.08
[16:44:38] <Damianz>	 lol
[17:24:35] <petan>	 hexmode: I created the tool
[17:24:46] <petan>	 it imports mw space :o
[17:24:51] <hexmode>	 petan: :)
[17:24:58] <hexmode>	 I got too tired
[17:25:09] <hexmode>	 petan: instructions on labs?
[17:25:15] <petan>	 I tried it on some wikis, only thing non automated is export it can't be done without shell access on prod
[17:25:29] <petan>	 hexmode: I will upload it somewhere
[17:26:19] <hexmode>	 petan: I think I have a way to get the export w/o production access, so maybe I can add that
[17:33:05] <petan>	 I sent it
[17:33:08] <petan>	 check mail
[17:33:37] <gerrit-wm>	 New patchset: Dzahn; "try out planet-venus package in labs" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1929
[17:33:52] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/1929
[17:34:28] <gerrit-wm>	 New review: Dzahn; "(no comment)" [operations/puppet] (test); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1929
[17:34:29] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1929
[19:00:34] <Oren|DSK>	 petan: hi
[19:14:59] <Oren|DSK>	 jeremyb: ping
[19:24:49] <jeremyb>	 Oren|DSK: hi
[19:25:01] <Oren|DSK>	 hi
[19:25:09] <Oren|DSK>	 ma nishma ?
[19:25:19] <jeremyb>	 hakol beseder
[19:25:29] <jeremyb>	 gam ata?
[19:25:41] <Oren|DSK>	 sabbaba
[19:25:59] <Oren|DSK>	 what are you up to these days?
[19:30:21] <Oren|DSK>	 ping me when you are available - I could use a little help
[19:33:48] <Oren|DSK>	 hi !!
[19:34:38] <Oren|DSK>	 Ryan_Lane: I waneted to ask you to put the cite extention on labs 
[19:34:50] <Ryan_Lane>	 oh, on labsconsole?
[19:34:53] <Ryan_Lane>	 it isn't already there?
[19:34:54] <Oren|DSK>	 Ryan_Lane: yes please
[19:35:05] <Ryan_Lane>	 gimme a sec
[19:35:17] <jeremyb>	 Oren|DSK: i'm available for <20 mins. but i'll be back on later (around 22:40 and then again around midnight your time) 
[19:35:26] <Oren|DSK>	 I got a personal page on it with instance specific info
[19:35:38] <Oren|DSK>	 ok
[19:35:59] <Oren|DSK>	 I need you to run a dump to html on search-test
[19:36:07] <jeremyb>	 Oren|DSK: err, sorry that was israeli time. subtract an hour from both
[19:36:11] <Oren|DSK>	 it should take a few days to run
[19:36:32] <Oren|DSK>	 so it not urgen
[19:36:34] <Oren|DSK>	 so it not urgent
[19:36:43] <Oren|DSK>	 but I need the html pages
[19:36:56] <Ryan_Lane>	 added Cite
[19:37:01] <Oren|DSK>	 thanks!
[19:37:19] <Ryan_Lane>	 yw
[19:37:39] <Oren|DSK>	 p.s It would be cool if you could set up squid there as well
[19:39:25] <Ryan_Lane>	 on labsconsole?
[19:39:26] <Ryan_Lane>	 why?
[19:39:32] <Ryan_Lane>	 it wouldn't do anything
[19:39:45] <Ryan_Lane>	 labsconsole requires login for basically everything
[19:40:46] <andrewbogott>	 Ryan_Lane:  If you have any spare brain cells today: http://wiki.openstack.org/PuppetConfigForNova
[19:41:03] <andrewbogott>	 I'm hoping my having written it wrong will motivate you to do it right :)
[19:44:10] <Oren|DSK>	 Ryan_Lane: Is there a way to replicate an instance ?
[19:45:48] <Ryan_Lane>	 andrewbogott: :D
[19:45:59] <Ryan_Lane>	 Oren|DSK: not right now, no
[19:46:09] <Ryan_Lane>	 Oren|DSK: the recommended option is puppet
[19:46:23] <Ryan_Lane>	 you completely puppetize something, then you can just select the same classes and variables
[19:47:23] <Oren|DSK>	 how do I puppertize something ?
[19:49:26] <Oren|DSK>	 say I wrote a script that installed it and edited some config files 
[19:55:30] <Ryan_Lane>	 !puppet
[19:55:30] <wm-bot>	 http://docs.puppetlabs.com/learning/ read it :)
[19:55:38] <Ryan_Lane>	 !puppet del
[19:55:39] <wm-bot>	 Successfully removed puppet
[19:55:54] <Ryan_Lane>	 !puppet is http://docs.puppetlabs.com/learning/
[19:55:54] <wm-bot>	 Key was added!
[19:57:57] <Oren|DSK>	 ok
[19:58:03] <Oren|DSK>	 I'll read
[19:58:12] <Oren|DSK>	 is that your work too ?
[19:59:09] <jeremyb>	 Ryan_Lane: what did you change?
[19:59:39] <jeremyb>	 (re the puppet key)
[19:59:55] <Ryan_Lane>	 Oren|DSK: no. it's a tool
[20:00:05] <jeremyb>	 oh, you removed the read it ;) ?
[20:00:10] <Ryan_Lane>	 yeah
[20:00:17] <Ryan_Lane>	 it's like telling people RTFM
[20:00:26] <jeremyb>	 !rtfm
[20:00:31] <jeremyb>	 heh
[20:00:40] <Ryan_Lane>	 yeah, we don't have that keyword for a reason ;)
[20:00:43] <jeremyb>	 i wonder if there's case sensitivty
[20:01:12] <jeremyb>	 Ryan_Lane: i'm a little afraid that the cite extension and squid were not for labsconsole
[20:01:23] <jeremyb>	 Oren|DSK: where did you want that for? on an instance?
[20:02:26] <Oren|DSK>	 search-test -> squid + running a dump to html
[20:02:48] <Oren|DSK>	 labs -> cite (all done already)
[20:05:32] <jeremyb>	 well, in context "It would be cool if you could set up squid there as well" means "in the same place as that last thing you did 2 mins ago". at least that's how i read it
[20:05:43] <jeremyb>	 "labs" is not very specific
[20:06:14] <Oren|DSK>	 labs == labsconsole
[20:07:16] <Ryan_Lane>	 oh, squid on labsconsole wouldn't help
[20:08:32] <jeremyb>	 Ryan_Lane: all logged in as you said before. now he said squid was for search-test
[20:08:37] <jeremyb>	 (an instance0
[20:08:57] <jeremyb>	 s/)/0/
[20:09:03] <jeremyb>	 gah
[20:09:05] <jeremyb>	 i can't type
[20:09:24] <Oren|DSK>	 that a regex emo
[21:40:35] <petan>	 Ryan_Lane: hey
[21:40:42] <petan>	 could you give me some parts of production squid config?
[21:40:59] <petan>	 I just enabled squid on labs, but dunno if it couldn't be better
[21:46:53] <Ryan_Lane>	 petan: I need to finish sanitizing it
[21:46:58] <Ryan_Lane>	 and I'm busy on sopa stuff right now
[21:48:27] <petan>	 ok
[21:48:27] <petan>	 np
[21:48:38] <petan>	 today web server had load 40+
[21:48:41] <petan>	 on labs
[21:48:47] <petan>	 so I think squid is a good idea
[22:01:50] <Ryan_Lane>	 petan: squid isn't going to help
[22:01:56] <petan>	 really?
[22:02:06] <petan>	 why do you think
[22:02:08] <Ryan_Lane>	 basically everything is logged in usage
[22:02:15] <petan>	 uh?
[22:02:22] <petan>	 I am talking about beta.wmflabs
[22:02:43] <Ryan_Lane>	 ahhhh pok
[22:02:44] <Ryan_Lane>	 *ok
[22:02:51] <Ryan_Lane>	 I thought you meant labsconsole
[22:02:54] <petan>	 that virtual server :)
[22:02:56] <petan>	 was on fire
[22:02:58] <petan>	 :D
[22:03:03] <petan>	 40+ load
[22:03:11] <petan>	 wasn't responding to ping
[22:44:29] <gerrit-wm>	 New patchset: Lcarr; "Seeing if puppet will autogenerate differently named files" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1938
[22:44:40] <gerrit-wm>	 New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (test); V: -1 - https://gerrit.wikimedia.org/r/1938
[22:44:49] <hexmode>	 petan: Y no wikispecies?
[22:55:47] <gerrit-wm>	 New patchset: Lcarr; "Seeing if puppet will autogenerate differently named files" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1938
[22:55:59] <gerrit-wm>	 New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (test); V: -1 - https://gerrit.wikimedia.org/r/1938
[23:06:49] <labs-nagios-wm>	 PROBLEM Current Load is now: CRITICAL on bots-cb bots-cb output: CRITICAL - load average: 5.39, 28.32, 17.16
[23:07:15] <gerrit-wm>	 New review: Lcarr; "I believe this will work despite the not passing the check :) let's see" [operations/puppet] (test); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1938
[23:11:49] <labs-nagios-wm>	 PROBLEM Current Load is now: WARNING on bots-cb bots-cb output: WARNING - load average: 0.28, 10.59, 12.52
[23:26:49] <labs-nagios-wm>	 RECOVERY Current Load is now: OK on bots-cb bots-cb output: OK - load average: 0.19, 0.81, 4.94
[23:31:07] <gerrit-wm>	 New review: Lcarr; "(no comment)" [operations/puppet] (test); V: 1 C: 2;  - https://gerrit.wikimedia.org/r/1938
[23:32:19] <gerrit-wm>	 Change merged: Lcarr; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1938
[23:35:44] <gerrit-wm>	 New patchset: Lcarr; "Revert "Seeing if puppet will autogenerate differently named files"" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1939
[23:36:08] <gerrit-wm>	 New review: Lcarr; "(no comment)" [operations/puppet] (test); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/1939
[23:36:08] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/1939
[23:36:14] <gerrit-wm>	 Change merged: Lcarr; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1939
[23:50:45] <gerrit-wm>	 New patchset: Lcarr; "testing puppet to make multiple files" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1940
[23:51:01] <gerrit-wm>	 New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/1940
[23:51:18] <gerrit-wm>	 New review: Lcarr; "(no comment)" [operations/puppet] (test); V: 0 C: 2;  - https://gerrit.wikimedia.org/r/1940
[23:51:18] <gerrit-wm>	 Change merged: Lcarr; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/1940